Road disease prediction method and device based on image generative adversarial network

Abstract

The present application relates to the field of data processing and artificial intelligence, and discloses a road disease prediction method and device based on an image generative adversarial network, comprising: obtaining a road disease mark image set and a to-be-detected road image set; generating a road disease training image set according to the road disease mark image set; determining a reference road disease discrimination result set according to the road disease mark image set and the road disease training image set; determining a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease mark image set; adjusting parameters of an initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model; and identifying road diseases of each to-be-detected road image in the to-be-detected road image set according to the target image generative adversarial network model to obtain more accurate target road disease discrimination results.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence, and in particular to a method and apparatus for predicting road defects based on image generative adversarial networks. Background Technology

[0002] In the maintenance and management of road infrastructure, timely and accurate detection of road defects is crucial. With technological advancements, automated road defect detection equipment and methods have emerged, generating a large amount of images and data during the detection process. However, current methods for efficient feature extraction and processing of these collected road defect images are lacking, making it difficult to extract key defect information. Furthermore, analyzing large volumes of data struggles to quickly and accurately identify the development trends and potential risks of defects, thus failing to provide strong support for road maintenance decisions. Therefore, improving the accuracy of the road defect identification process has become an urgent technical challenge. Summary of the Invention

[0003] This invention provides a method and apparatus for predicting road defects based on image generative adversarial networks, in order to solve the technical problem of how to improve the accuracy of the road defect identification process.

[0004] Firstly, a road defect prediction method based on image generative adversarial networks is provided, including:

[0005] Obtain a set of road defect labeled images with road defect category tags, and a set of road images to be detected;

[0006] Based on the generator in the initial image generative adversarial network model, a set of road defect training images is generated according to the set of road defect marked images;

[0007] Based on the discriminator in the initial image generative adversarial network model, a reference set of road defect discrimination results is determined according to the set of road defect marked images and the set of road defect training images.

[0008] Based on the reference set of road defect identification results, the set of road defect training images, and the set of road defect marked images, determine the set of road defect loss values;

[0009] Based on the set of road damage loss values, the parameters of the initial image generative adversarial network model are adjusted to obtain the target image generative adversarial network model.

[0010] Based on the target image generative adversarial network model, road defects are identified for each road image in the set of road images to be detected, and the target road defect discrimination result is obtained.

[0011] In a second aspect, a road disease prediction device based on an image generative adversarial network is provided, and includes:

[0012] An acquisition module is configured to acquire a road disease mark image set with road disease category labels and a to-be-detected road image set;

[0013] A generation module is configured to generate a road disease training image set based on a generator in an initial image generative adversarial network model according to the road disease mark image set;

[0014] A first determination module is configured to determine a reference road disease discrimination result set based on a discriminator in the initial image generative adversarial network model according to the road disease mark image set and the road disease training image set;

[0015] A second determination module is configured to determine a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease mark image set;

[0016] A first processing module is configured to adjust parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model;

[0017] A second processing module is configured to perform road disease recognition on each to-be-detected road image in the to-be-detected road image set according to the target image generative adversarial network model to obtain a target road disease discrimination result.

[0018] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the road disease prediction method based on the image generative adversarial network when executing the computer program.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the steps of the road disease prediction method based on the image generative adversarial network.

[0020] In the scheme implemented by the road disease prediction method and device based on the image generation type adversarial network, the road disease marked image set with road disease class labels and the to-be-detected road image set are acquired, the generator in the initial image generation type adversarial network model is used to generate a road disease training image set according to the road disease marked image set, the discriminator in the initial image generation type adversarial network model is used to determine a reference road disease discrimination result set according to the road disease marked image set and the road disease training image set, and the road disease loss value set is further determined according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set. Therefore, the initial image generation type adversarial network model can be adjusted in parameters according to the road disease loss value set to obtain a target image generation type adversarial network model, and then the road disease recognition of each to-be-detected road image in the to-be-detected road image set can be performed according to the target image generation type adversarial network model to obtain a target road disease discrimination result which is more accurate and meets the road disease recognition requirement, thereby achieving the purpose of more accurate and efficient road disease recognition. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0022] Figure 1 is an application environment schematic diagram of the road disease prediction method based on the image generation type adversarial network in an embodiment of the present application;

[0023] Figure 2 is a flow schematic diagram of the road disease prediction method based on the image generation type adversarial network in an embodiment of the present application;

[0024] Figure 3 is a structure schematic diagram of the road disease prediction device based on the image generation type adversarial network in an embodiment of the present application;

[0025] Figure 4 is a structure schematic diagram of the computer device in an embodiment of the present application;

[0026] Figure 5 is another structure schematic diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0028] The road disease prediction method based on the image generative adversarial network provided in the embodiments of the present application can be applied in the application environment such as Figure 1 The client can communicate with the server through the network. For example, in the scene of road disease identification for the to-be-detected road image, the target user (such as a developer or manager of the image generative adversarial network model) can upload a road disease marked image set with road disease category labels and a to-be-detected road image set through the client.

[0029] Correspondingly, the server can obtain the road disease marked image set with road disease category labels and the to-be-detected road image set through the client, generate a road disease training image set based on the generator in the initial image generative adversarial network model according to the road disease marked image set, determine a reference road disease discrimination result set based on the discriminator in the initial image generative adversarial network model according to the road disease marked image set and the road disease training image set, further determine a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, and thus can adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model, and then can identify the road disease of each to-be-detected road image in the to-be-detected road image set according to the target image generative adversarial network model to obtain a target road disease discrimination result more accurate and more in line with the requirements of road disease identification, and feed back the target road disease discrimination result to the client. Correspondingly, the client can receive the target road disease discrimination result from the server and display the target road disease discrimination result on the client for the target user to query or browse. By adopting the road disease prediction method based on the image generative adversarial network provided in the present application, the accuracy of road disease identification can be effectively improved, and the target road disease discrimination result more accurate and more in line with the requirements of road disease identification can be obtained.

[0030] The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0031] Referring to Figure 2 as shown, Figure 2 A flowchart of a road disease prediction method based on an image generative adversarial network provided by an embodiment of the present application is shown, including the following steps:

[0032] S10: Obtain a road disease marked image set with road disease category labels and a to-be-detected road image set.

[0033] The road disease marked image set can include one or more road disease marked images, which can be images marked with road disease categories. That is, the road disease marked image can be a real road disease marked image with road disease annotation. The road disease category label can be a label annotated by an artificial intelligence system and audited by an artificial, and the present application does not limit this.

[0034] The to-be-detected road image set can include one or more to-be-detected road images, which can be images that need to be detected for road diseases. That is, the server can determine whether each to-be-detected road image in the to-be-detected road image set has a road disease, and if there is a road disease, the category of the road disease needs to be determined, and the present application does not limit this.

[0035] S20: Based on the generator in the initial image generative adversarial network model, generate a road disease training image set according to the road disease marked image set.

[0036] The initial image generative adversarial network (GAN) model is a deep learning model architecture composed of a generator (Generator) and a discriminator (Discriminator). It can be understood that the initial image generative adversarial network model can be a model that has not been adjusted in parameters, and its performance is usually limited.

[0037] The generator can be used to generate realistic road disease training images to try to deceive the discriminator. The generator can be composed of a series of neural network layers, such as convolutional neural networks or fully connected neural networks, to take random noise or latent vectors as input and gradually convert them into road disease training images required by the present application, and the present application does not limit this. The road disease training image set can include one or more road disease training images, which can be used to indicate images generated based on the generator and according to the road disease marked image.

[0038] It needs to be understood that the generation of the road disease training image set based on the generator in the initial image generative adversarial network model according to the road disease marking image set refers to the process of how to generate the road disease training image. In step S20, that is, the generation of the road disease training image set based on the generator in the initial image generative adversarial network model according to the road disease marking image set can include the following steps:

[0039] S21: performing feature extraction on each road disease marking image in the road disease marking image set based on the generator in the initial image generative adversarial network model to obtain a road disease feature vector set;

[0040] S22: performing dimensionality increasing processing on each random noise vector in the random noise vector set according to the road disease feature vector set to obtain a random dimensionality increasing vector set;

[0041] S23: performing feature mapping processing on each random dimensionality increasing vector in the random dimensionality increasing vector set according to the road disease feature vector set to obtain a random feature map set;

[0042] S24: performing up-sampling processing on each random feature map in the random feature map set according to the road disease feature vector set to obtain a random high-resolution feature map set;

[0043] S25: performing road disease feature establishment processing on each random high-resolution feature map in the random high-resolution feature map set according to the road disease feature vector set to obtain a road disease training image set.

[0044] The road disease feature vector set can include one or more road disease feature vectors, which can be used to indicate the road disease feature vector obtained after the generator performs feature extraction on the road disease marking image. For example, the server can extract the shape of the crack, the size and depth of the pothole, and the texture of the road surface damage, and other key road disease features from the road disease marking image, to convert complex image data into a feature vector representation that is easier to process and analyze.

[0045] Optionally, the server can use a convolutional neural network (CNN) or other method to implement the feature extraction step. The CNN can automatically learn different levels of features in the image through the combination of multiple convolutional layers and pooling layers, for example, for a road disease marking image with a road crack disease marking, the CNN can extract edge features, direction, depth, and other features of the crack, which is not limited in the present application.

[0046] The random high-dimensional vector set can include one or more random high-dimensional vectors, which can be used to indicate vectors obtained after the random noise vector is processed by the high-dimensional processing according to the road disease feature vector. The server can implement the high-dimensional processing by using a fully connected layer or a convolutional layer, so that the random high-dimensional vector obtained after the high-dimensional processing can have more information and higher dimensions, so as to facilitate subsequent feature mapping processing.

[0047] It can be understood that in the process of a series of feature transformation operations on the random noise vector, such as the above-mentioned high-dimensional processing, and the subsequent feature mapping processing and convolution processing to be mentioned, the generator can learn feature patterns (road disease feature vectors) from the real road disease label image, and gradually apply these feature patterns to the shaping process of the random noise vector. Optionally, the random high-dimensional vector obtained after the high-dimensional processing can better interact with the road disease feature vector set in the subsequent steps, thereby generating more realistic road disease training images.

[0048] The random feature map set can include one or more random feature maps, which can be used to indicate feature maps obtained after the random high-dimensional vector is processed by the feature mapping according to the road disease feature vector. Optionally, the server can use a convolutional layer or a deconvolutional layer to perform feature mapping processing, for example, the server can fuse the random high-dimensional vector after the high-dimensional processing with the road disease feature vector through a convolutional layer to obtain a random feature map with road disease features.

[0049] The random high-resolution feature map set can include one or more random high-resolution feature maps, which can be used to indicate feature maps obtained after the random feature map is processed by the up-sampling according to the road disease feature vector. Optionally, the server can use a deconvolutional layer or interpolation to perform up-sampling processing, for example, the server can gradually up-sample the low-resolution random feature map through a deconvolutional layer to obtain a high-resolution random high-resolution feature map, so that it gradually approaches the resolution of the real road disease label image.

[0050] The server can further process the high-resolution random feature map according to the road disease feature vector set to establish more accurate road disease features, thereby obtaining a final road disease training image set, so that the generated feature map is more consistent with the real road disease situation, and the quality and authenticity of the generated image are improved. Optionally, the server can use a convolutional layer, an activation function, and the like to perform road disease feature establishment processing, for example, the server can use a combination of multiple convolutional layers and activation functions to further extract and enhance the features of the high-resolution random feature map, thereby establishing more accurate road disease features, and further obtaining the road disease training image set.

[0051] For example, the server can construct the following generator network (assuming the size of the road disease marking image is 64x64 pixels, and it is a color image (3 channels)):

[0052] 1. Input layer, used to receive a random noise vector, such as receiving a 100-dimensional random noise vector.

[0053] 2. Hidden layer 1, fully connected layer, used for dimensionality processing, such as mapping a 100-dimensional random noise vector to a 256-dimensional random dimensionality vector. It can be understood that in order to enable the generator to gradually construct more complex and rich feature representations from low-dimensional random noise in order to generate high-quality road disease images or other types of output, the server can help the generator explore and capture more potential features and variations in higher-dimensional space through dimensionality processing, thereby increasing the diversity and realism of the generated images. Optionally, the hidden layer 1 can use a leaky rectified linear unit (Leaky ReLU) for activation processing, which is not limited in the present application.

[0054] 3. Hidden layer 2, deconvolution layer, used for feature mapping processing, such as mapping a 256-dimensional random dimensionality vector to a random feature map with a size of 4x4x128. It can be understood that the server can further extract higher-level and more abstract feature representations from the original 256-dimensional random dimensionality vector. Since different dimensions and spatial structures can correspond to different feature attributes, the server can perform feature mapping to convert the original features into a more suitable feature form for the road disease recognition task. For example, for the road disease image generation task, the server can extract new features related to road disease shape, texture, color, etc. from the original features. Optionally, the hidden layer 2 can use a batch normalization function (BatchNormalization) for normalization, and a rectified linear unit (ReLU) for activation processing, which is not limited in the present application.

[0055] 4. Hidden layer 3, deconvolutional layer, used for up-sampling processing, such as expanding the feature map with the size of 4x4x128 into the feature map with the size of 8x8x64. It can be understood that the up-sampling is an operation of converting a low-resolution feature map into a high-resolution feature map, and in the process of expanding the feature map with the size of 4x4x128 into the feature map with the size of 8x8x64, the spatial size of the feature map can be increased (i.e., from 4x4 to 8x8), and the number of channels can be adjusted (from 128 to 64). Optionally, the server can use Batch Normalization for normalization, and can use ReLU for activation processing, which is not limited in the present application.

[0056] 5. Hidden layer 4, deconvolutional layer, used for up-sampling processing, such as expanding the feature map with the size of 8x8x64 into the feature map with the size of 16x16x32. Optionally, the server can use Batch Normalization for normalization, and can use ReLU for activation processing, which is not limited in the present application.

[0057] 6. Hidden layer 5, deconvolutional layer, used for up-sampling processing, such as expanding the feature map with the size of 16x16x32 into the feature map with the size of 32x32x16. Optionally, the server can use Batch Normalization for normalization, and can use ReLU for activation processing, which is not limited in the present application.

[0058] 7. Output layer, deconvolutional layer, used for road disease feature establishment processing, such as generating a feature map with the size of 64x64x3. Optionally, the server can use Hyperbolic Tangent (Tanh) for activation processing to map the output value to the range of [-1, 1], which is not limited in the present application.

[0059] For steps S21-S25, the server extracts features from each road disease marking image in the road disease marking image set by the generator in the initial image generative adversarial network model to obtain a road disease feature vector set, performs dimensionality increasing processing on each random noise vector in the random noise vector set according to the road disease feature vector set to obtain a random dimensionality increased vector set, performs feature mapping processing on each random dimensionality increased vector in the random dimensionality increased vector set according to the road disease feature vector set to obtain a random feature map set, and further performs up-sampling processing on each random feature map in the random feature map set according to the road disease feature vector set to obtain a random high-resolution feature map set. Thus, the road disease training image set can be obtained by performing road disease feature establishment processing on each random high-resolution feature map in the random high-resolution feature map set according to the road disease feature vector set, which is conducive to adjusting the parameters of the initial image generative adversarial network model based on the road disease training image set in subsequent steps.

[0060] S30: Determine a reference road disease discrimination result set based on the discriminator in the initial image generative adversarial network model according to the road disease marking image set and the road disease training image set.

[0061] The discriminator can be used to distinguish whether the input image is a real road disease marking image or a road disease training image generated by the generator. The discriminator usually has a similar structure to the generator, such as being composed of a neural network, but the purpose of the discriminator is different from that of the generator. The discriminator receives an image (such as a road disease marking image or a road disease training image) as input and outputs a scalar value representing the probability that the input image is a real road disease marking image. The goal of the discriminator can be to accurately judge the authenticity of the image as much as possible, thereby providing feedback to the generator to encourage the generator to generate more realistic images.

[0062] The reference road disease discrimination result set can include one or more reference road disease discrimination results, which can be used to indicate the discrimination result obtained by the discriminator in the initial image generative adversarial network model after discriminating the road disease marking image and the road disease training image. It can be understood that the reference road disease discrimination result is not necessarily accurate, and therefore the parameters of the initial image generative adversarial network model can be adjusted further according to the reference road disease discrimination result to obtain a target image generative adversarial network model that is more accurate and more consistent with the requirements of road disease discrimination, which is not limited in the present application.

[0063] It needs to be understood that, based on the discriminator in the initial image generative adversarial network model, the reference road disease discrimination result set is determined according to the road disease marked image set and the road disease training image set, which refers to the process of how the discriminator determines the reference road disease discrimination result. In step S30, that is, based on the discriminator in the initial image generative adversarial network model, the reference road disease discrimination result set is determined according to the road disease marked image set and the road disease training image set, which can include the following steps:

[0064] S31: performing road disease feature extraction on each road disease marked image in the road disease marked image set and each road disease training image in the road disease training image set to obtain a road disease feature map set;

[0065] S32: performing maximum pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease enhanced feature map set;

[0066] S33: performing average pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease smooth feature map set;

[0067] S34: respectively splicing each road disease local feature map in the road disease local feature map set with the corresponding road disease global feature map in the road disease global feature map set to obtain a road disease splicing feature map set;

[0068] S35: performing attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a road disease attention feature map set;

[0069] S36: performing full connection processing on each road disease attention feature map in the road disease attention feature map set to obtain a reference road disease discrimination result set.

[0070] The road disease feature map set can include one or more road disease feature maps, which can be used to indicate the feature maps obtained after performing road disease feature extraction on the road disease marked image and the road disease training image. The server can use convolutional neural network, principal component analysis, etc. to perform road disease feature extraction on the road disease training image, which is not limited in the present application.

[0071] The set of road disease enhancement feature maps can include one or more road disease enhancement feature maps, which can be used to indicate the feature map obtained after maximum pooling operation is performed on the road disease feature map. The server performing maximum pooling operation on the road disease feature map can highlight the local significant features in the image. For example, if there is an obvious crack edge or the deepest point of a pothole, the maximum pooling can capture these areas with higher pixel values; and it is helpful to strengthen the edge information. For example, the pixel value change at the edge is usually large, and by selecting the local maximum value through the maximum pooling operation, the change of edge information can be highlighted more obviously. Optionally, the maximum pooling operation has certain anti-interference ability, which can make the extracted features more robust, has certain tolerance to light changes, slight angle changes, etc., helps to reduce the number of parameters and calculation amount of the model, and improves the calculation efficiency, which is not limited in the present application.

[0072] The set of road disease smoothing feature maps can include one or more road disease smoothing feature maps, which can be used to indicate the feature map obtained after average pooling operation is performed on the road disease feature map. The server performing average pooling operation on the road disease feature map can reduce the sharp changes and noise in the road disease feature map, can capture the smooth changes and overall trend of the image, and to some extent, retain more background information, which is helpful to improve the stability and generalization ability of the model. For example, during the process of processing the road disease feature map, the average pooling operation can obtain the average intensity, color and other information of the road disease area, which is not limited in the present application.

[0073] The set of road disease splicing feature maps can include one or more road disease splicing feature maps, which can be used to indicate the feature map obtained after the road disease local feature map and the corresponding road disease global feature map are respectively spliced.

[0074] The set of road disease attention feature maps can include one or more road disease attention feature maps, which can be used to indicate the feature map obtained after attention calculation is performed on the road disease splicing feature map.

[0075] It needs to be understood that the attention calculation of each road disease splicing feature map in the road disease splicing feature map set to obtain the road disease attention feature map set refers to the process of how to perform attention calculation. In step S35, that is, the attention calculation of each road disease splicing feature map in the road disease splicing feature map set to obtain the road disease attention feature map set can include the following steps:

[0076] S351: Perform road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set;

[0077] S352: Perform road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set;

[0078] S353: Perform attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set, respectively, to obtain a fused attention feature map set;

[0079] S354: Perform deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set;

[0080] S355: Perform residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set;

[0081] S356: Perform adaptive activation processing on each residual attention feature map in the residual attention feature map set to obtain a road disease attention feature map set.

[0082] The local attention feature map set can include one or more local attention feature maps, which can be used to indicate the feature map obtained after performing road disease local attention calculation on the road disease splicing feature map. By introducing the local self-attention mechanism, the server can better focus on the local disease area of the road. For example, for a road image with a wide range, the server can first divide the road image into multiple local areas to perform local self-attention calculation in each local area, and then combine the results of global self-attention in subsequent steps, so that the model can focus on local disease areas while paying attention to global information.

[0083] The server can first determine the size and shape of the local disease area of the road according to specific circumstances. For example, the server can divide the image into non-overlapping rectangular regions, which is not limited in the present application. Secondly, for each local disease area of the road, the server can calculate the corresponding local attention weight respectively. Optionally, the server can use a traditional self-attention mechanism, such as the server can use a multi-head attention mechanism (MHA) to calculate the local attention weight. Further, the server can weight and sum the local attention weight and the value of the corresponding local area to obtain the local attention feature map set.

[0084] Optionally, the server can perform road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain the local attention feature map set, which can be referred to the following formula:

[0085]

[0086] wherein LocalAttention(T local_t ) can be used to represent the t-th local attention feature map in the local attention feature map set, T local_t can be used to represent the t-th road disease splicing feature map, t can be used to represent the index in the road disease splicing feature map set; n x n can be used to represent the number of local disease areas of the road divided from the first road disease splicing feature map; m can be used to represent the index of the number of local disease areas of the road; Attention m () can be used to represent the operation of local attention calculation on the m-th local disease area of the road; Q m can be used to represent the first feature matrix extracted from the m-th local disease area of the road, i.e. the Query matrix of the m-th local disease area of the road; K m can be used to represent the second feature matrix extracted from the m-th local disease area of the road, i.e. the Key matrix of the m-th local disease area of the road; V m can be used to represent the third feature matrix extracted from the m-th local disease area of the road, i.e. the Value matrix of the m-th local disease area of the road.

[0087] The global attention feature map set can include one or more global attention feature maps, which can be used to indicate the feature map obtained after performing road disease global attention calculation on the road disease splicing feature map. Optionally, the server can use the same or different way to perform global attention calculation as local attention calculation, which is not limited in the present application.

[0088] Optionally, the server can directly use the self-attention mechanism on the entire input feature map (i.e., the road disease splicing feature map) to calculate the global attention weight of the input feature map. The server performs road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set. The step can be referred to the following formula:

[0089]

[0090] wherein, GlobalAttention(T global_t ) can be used to represent the t-th global attention feature map in the global attention feature map set, T global_t can be used to represent the t-th road disease splicing feature map, t can be used to represent the index in the road disease splicing feature map set; softmax() can be used to represent the normalization operation; Q can be used to represent the first overall feature matrix extracted from the t-th road disease splicing feature map, i.e., the Query matrix of the t-th road disease splicing feature map; K can be used to represent the second overall feature matrix extracted from the t-th road disease splicing feature map, i.e., the Key matrix of the t-th road disease splicing feature map, K T can be used to represent the transpose of the second overall feature matrix; d k can be used to represent the dimension of the second overall feature matrix; V can be used to represent the third feature matrix extracted from the t-th road disease splicing feature map, i.e., the Value matrix of the t-th road disease splicing feature map.

[0091] The fusion attention feature map set can include one or more fusion attention feature maps, which can be used to indicate the feature map obtained by respectively performing attention fusion on each local attention feature map and the corresponding global attention feature map. Optionally, the server can use the weighted fusion method to fuse the local attention feature map and the global attention feature map, which is not limited in the present application.

[0092] Optionally, the step of respectively performing attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set to obtain a fusion attention feature map set can be referred to the following formula:

[0093] FusedAttention(T t )

[0094] = δ × GlobalAttention(T global_t ) + ε × LocalAttention(T local_t )

[0095] wherein, FusedAttention(T t ) can be used to represent the t-th global attention feature map in the fused attention feature map set, T t can be used to represent the t-th road disease splicing feature map, and t can be used to represent the index in the road disease splicing feature map set; δ can be used to represent the weight coefficient of the global attention feature map; GlobalAttention(T global_t ) can be used to represent the t-th global attention feature map in the global attention feature map set, T g1obal_t can be used to represent the t-th road disease splicing feature map; ε can be used to represent the weight coefficient of the local attention feature map; LocalAttention(T local_t ) can be used to represent the t-th local attention feature map in the local attention feature map set, T local_t can be used to represent the t-th road disease splicing feature map.

[0096] The depth attention feature map set can include one or more depth attention feature maps, which can be used to represent the feature map obtained after performing depth convolution operation on the fused attention feature map. After the server performs the self-attention mechanism operation on the road disease splicing feature map, the server can further pass through the feed forward network (FFN) to increase the nonlinear expression ability of the model, so as to extract more complex features.

[0097] Optionally, the server can use depth separable convolution in the FFN to reduce the number of parameters while maintaining good feature extraction ability, and introduce residual connection to enhance the information transmission and feature reuse ability of the FFN, so that the model can better learn the complex changes of road disease features, and further use an adaptive activation function to automatically adjust the activation degree according to the input features, thereby improving the nonlinear expression ability of the model.

[0098] Optionally, the server can perform depth convolution operation on each fused attention feature map in the fused attention feature map set to obtain the depth attention feature map set, which can be referred to as the following formula:

[0099]

[0100] wherein, S DW,cmay be used to represent the depth attention feature map of the c-th channel in the set of depth attention feature maps; c may be used to represent the index of the channel; H may be used to represent the size of the convolution kernel, and each channel uses an HxH convolution kernel to perform convolution operation independently; p may be used to represent the first position of the convolution kernel to be traversed; q may be used to represent the second position of the convolution kernel to be traversed; W DW,c (p, q) may be used to represent the convolution kernel position of the c-th channel to be traversed; FusedAttention(m+p, n+q, c) may be used to represent the value of the c-th channel of the input feature map (i.e., the fused attention feature map) at position (m+p, n+q); (m, n) may be used to represent the position of each convolution operation of the input feature map (i.e., the fused attention feature map).

[0101] The set of residual attention feature maps may include one or more residual attention feature maps, which may be used to represent the feature map obtained after the residual connection of the depth attention feature map. By performing residual connection, the server can make the FFN more easily learn the residual between the input and the output, avoid the problem of gradient disappearance, thereby accelerating the training, which is not limited by the present application.

[0102] The server may further perform adaptive activation processing on the residual attention feature map through an adaptive activation function, such as a self-gating activation function (Sigmoid-weighted Linear Unit, SiLU, also known as Swish function), Mish activation function (A Self Regularized Non-Monotonic Neural Activation Function, Mish), etc., so that the activation degree can be automatically adjusted according to the input feature, and then a more needed set of road disease attention feature maps can be obtained, which is not limited by the present application.

[0103] For steps S351-S356, the server performs road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set, performs road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set, and respectively performs attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set to obtain a fusion attention feature map set. Further, each fusion attention feature map in the fusion attention feature map set is subjected to deep convolution operation to obtain a deep attention feature map set, and each deep attention feature map in the deep attention feature map set is subjected to residual connection to obtain a residual attention feature map set. Thus, each residual attention feature map in the residual attention feature map set can be subjected to adaptive activation processing to obtain a road disease attention feature map set, which is conducive to capturing more accurate and comprehensive road disease features.

[0104] Further, after obtaining the road disease attention feature map set, the server can perform full connection processing on each road disease attention feature map to convert the road disease attention feature map into an output form suitable for a road disease discrimination task, i.e., to obtain a reference road disease discrimination result set. Optionally, the server can perform the above full connection processing through a full connection layer, such as a full connection layer to integrate and fuse features, further extract high-level abstract features, learn complex nonlinear relationships, and output results of specific tasks, such as probability distribution of different disease types or severity level of the disease, etc. The present application does not limit this.

[0105] Illustratively, the server can construct a discriminator network as follows:

[0106] 1. Input layer, such as receiving a road disease training image with a size of 64x64x3;

[0107] 2. Hidden layer 1, convolution layer, used for extracting road disease features, such as outputting a road disease feature map with a size of 32x32x64;

[0108] 3. Hidden layer 2, convolution layer, used for further extracting features, such as performing maximum pooling operation or average pooling operation, such as outputting a road disease enhanced feature map with a size of 16x16x64 and a road disease smoothed feature map with a size of 16x16x64, and further performing splicing processing, such as outputting a road disease splicing feature map with a size of 16x16x128;

[0109] 4. Hidden layer 3, hidden layer with attention mechanism, used for attention calculation, such as outputting a road disease attention feature map with a size of 16x16x128;

[0110] 5. an output layer, a full connection layer, configured to perform full connection processing, and output a value for indicating a probability that the input road disease training image is a real road disease label image.

[0111] For steps S31-S36, the server performs road disease feature extraction on each road disease training image in the road disease training image set to obtain a road disease feature map set, performs max-pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease enhanced feature map set, and performs average-pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease smooth feature map set, so that each road disease local feature map in the road disease local feature map set and the corresponding road disease global feature map in the road disease global feature map set can be respectively spliced to obtain a road disease splicing feature map set, each road disease splicing feature map in the road disease splicing feature map set can be subjected to attention calculation to obtain a road disease attention feature map set, and each road disease attention feature map in the road disease attention feature map set can be subjected to full connection processing to obtain a reference road disease discrimination result set, which is conducive to training a more desirable target image generative adversarial network model based on the reference road disease discrimination result set in subsequent steps.

[0112] S40: determining a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set, and the road disease label image set.

[0113] The road disease loss value set can include one or more road disease loss values, which can be used to indicate a loss value calculated based on a constructed loss function according to a reference road disease discrimination result, a road disease training image, and a road disease label image.

[0114] The road disease loss value set can measure the performance of the initial image generative adversarial network model in the road disease identification task, and can reflect the difference between the road disease training image generated by the generator and the real road disease label image, and can reflect the discrimination ability of the discriminator for the generated image and the real image. Optionally, the road disease loss value can include a generator loss and a discriminator loss, the generator loss can be obtained based on the judgment result of the discriminator for the generated image and the difference between the generated image and the real image; the discriminator loss can be obtained based on its correct classification ability for the real image and the generated image, which is not limited in the present application.

[0115] It needs to be understood that determining the road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set refers to the process of how to calculate the road disease loss value. In step S40, that is, determining the road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set can include the following steps:

[0116] S41: performing adversarial loss calculation according to each reference road disease discrimination result in the reference road disease discrimination result set to obtain an adversarial loss value set of the generator;

[0117] S42: performing content loss calculation according to each road disease training image in the road disease training image set to obtain a content loss value set of the generator;

[0118] S43: performing content loss calculation according to each road disease training image in the road disease training image set and each road disease marked image in the road disease marked image set to obtain a content loss value set of the generator;

[0119] S44: performing real loss calculation according to each road disease marked image in the road disease marked image set to obtain a real loss value set of the discriminator;

[0120] S45: performing fake loss calculation according to each road disease training image in the road disease training image set to obtain a fake loss value set of the discriminator;

[0121] S46: determining a discrimination loss value set of the discriminator according to the real loss value set and the fake loss value set;

[0122] S47: determining the road disease loss value set according to the generation loss value set and the discrimination loss value set.

[0123] The adversarial loss value set can include one or more adversarial loss values, which can be used to indicate the loss value of the generator obtained after performing adversarial loss calculation on the reference road disease discrimination result. The generator introduces the adversarial loss value to represent the probability that the discriminator judges the image generated by the generator as real in order to make the discriminator judge the image generated by the generator as real, which is not limited in the present application.

[0124] Optionally, the server performs adversarial loss calculation according to each reference road disease discrimination result in the reference road disease discrimination result set to obtain the adversarial loss value set of the generator, which can be seen from the following formula:

[0125] L gen_adv_i= -log(D(G(X i ))

[0126] wherein, L gen_adv_i may be used to represent the i-th adversarial loss value in the adversarial loss value set of the generator; log() may be used to represent the logarithm operation; D(G(X i )) may be used to represent the output of the discriminator on the i-th image generated by the generator, i.e., the i-th reference road disease discrimination result, which is the output value of the discriminator; G(X i ) may be used to represent the i-th image generated by the generator, i.e., the output value of the generator, which is the input value of the discriminator; X i may be used to represent the i-th random noise vector, which is the input value of the generator.

[0127] The content loss value set can include one or more content loss values, which can be used to indicate the loss value of the generator obtained after content loss calculation is performed on the road disease training image. The generator can calculate the content loss between the road disease training image generated by the generator and the road disease mark image with a road disease class label, to ensure that the generated road disease training image is similar in content to the real road disease mark image, and the application does not limit this. Optionally, the server can use methods such as calculating the mean squared error (MSE) or calculating the structural similarity index (SSIM) to calculate the content loss value, and the application does not limit this.

[0128] Optionally, taking the case where the server uses the MSE calculation method as an example, the server performs content loss calculation on each road disease training image in the road disease training image set and each road disease mark image in the road disease mark image set to obtain the content loss value set of the generator, which can be seen from the following formula:

[0129]

[0130] wherein, L gen_content_i may be used to represent the i-th content loss value in the content loss value set of the generator, i may be used to represent the loss value index in the content loss value set, which can correspond to the loss value index in the adversarial loss value set; N i may be used to represent the number of pixels in the i-th road disease training image, i may be used to represent the image index in the road disease training image set, which can correspond to the loss value index in the content loss value set; j may be used to represent the pixel index in the i-th road disease training image; G(X i) can be used to represent the i-th road disease training image generated by the generator, i can be used to represent the image index in the road disease training image set; G(X i ) j Yi,j can be used to represent the j-th pixel value of the i-th road disease training image generated by the generator. i i ) can be used to represent the i-th road disease label image corresponding to the i-th road disease training image, (Y j gen ) can be used to represent the j-th pixel value of the i-th road disease label image corresponding to the i-th road disease training image, j can be used to represent the pixel index in the i-th road disease label image, which corresponds to the pixel index in the i-th road disease training image.

[0131] The generation loss value set can include one or more generation loss values, which can be used to indicate the total loss value of the generator after fusing the adversarial loss value and the content loss value. The server can perform weighted summation operation on the adversarial loss value and the content loss value to obtain the total loss of the generator, and then the total loss of the generator can be minimized and the parameters of the generator can be adjusted to make the generator generate more realistic road disease images, which is not limited in the present application.

[0132] Optionally, the server determines the generation loss value set of the generator according to the adversarial loss value set and the content loss value set, which can refer to the following formula:

[0133] L gen = L gen_adv + aL gen_content

[0134] Wherein, L gen can be used to represent the generation loss value set of the generator; L gen_adv can be used to represent the adversarial loss value set; a can be used to represent the weight coefficient of the generator loss value, which can be used to balance the contribution of the adversarial loss and the content loss; L gen_content can be used to represent the content loss value set.

[0135] The real loss value set can include one or more real loss values, which can be used to indicate the loss value of the discriminator after real loss calculation is performed on the road disease mark image. The goal of the discriminator is to distinguish between real road disease mark images and road disease training images generated by the generator, so the server can calculate the loss value of the discriminator for real road disease mark images, i.e., the real loss value set described above, and calculate the loss value of the discriminator for road disease training images generated by the generator, i.e., the fake loss value set to be mentioned below, to further improve the recognition ability of the discriminator.

[0136] Optionally, for real road disease mark images, the output of the discriminator should be as close to 1 as possible to indicate that the image is a real image. The server performs real loss calculation on each road disease mark image in the road disease mark image set to obtain the real loss value set of the discriminator, which can be seen in the following steps:

[0137] L dis_real_i i i )

[0138] Wherein, L dis_real_i i may be used to represent the i-th real loss value in the real loss value set of the discriminator; log() can be used to represent the logarithmic operation; D(Y i i ) can be used to represent the output of the discriminator for the i-th road disease mark image, i.e., the output value of the discriminator; Y i i may be used to represent the i-th road disease mark image, i.e., the input value of the discriminator.

[0139] The fake loss value set can include one or more fake loss values, which can be used to indicate the loss value of the discriminator after fake loss calculation is performed on the road disease training image. Optionally, for road disease training images generated by the generator, the output of the discriminator should be as close to 0 as possible to indicate that the image is a fake image.

[0140] Optionally, the server performs fake loss calculation on each road disease training image in the road disease training image set to obtain the fake loss value set of the discriminator, which can be seen in the following steps:

[0141] L dis_fake_i i i )

[0142] Wherein, L dis_fake_i i may be used to represent the i-th real loss value in the real loss value set of the discriminator; log() can be used to represent the logarithmic operation; D(G(X i) can be used to represent the output of the discriminator on the i-th image generated by the generator, i.e., the i-th reference road disease discrimination result, that is, the output value of the discriminator; G(X i ) can be used to represent the i-th image generated by the generator, i.e., the output value of the generator, that is, the input value of the discriminator; X i ) can be used to represent the i-th random noise vector, i.e., the input value of the generator.

[0143] The set of discrimination loss values can include one or more discrimination loss values, which can be used to indicate the total loss value of the discriminator obtained by fusing the real loss value and the false loss value. The server can perform weighted summation operation on the real loss value and the false loss value to obtain the total loss of the discriminator, and then can minimize the total loss of the discriminator and continuously adjust the parameters of the discriminator, so that the discriminator can better distinguish the real road disease mark image and the generated road disease training image, which is not limited by the present application.

[0144] Optionally, the server determines the set of discrimination loss values of the discriminator according to the set of real loss values and the set of false loss values, which can be referred to as the following formula:

[0145] L dis = L dis_real + βL dis_fake

[0146] Wherein, L dis can be used to represent the set of discrimination loss values of the discriminator; L dis_real can be used to represent the set of real loss values; β can be used to represent the weight coefficient of the discriminator loss value, which can be used to balance the contribution of the real road disease mark image and the generated road disease training image; L dis_fake can be used to represent the false loss value.

[0147] The server can further combine the set of generation loss values and the set of discrimination loss values to determine the set of road disease loss values corresponding to the image generative adversarial network model, so as to minimize the loss value of the image generative adversarial network model while adjusting the parameters of the generator and the discriminator, so that the generator can generate more realistic road disease training images, and the discriminator can better distinguish the real road disease mark image and the generated road disease training image, which is not limited by the present application.

[0148] Optionally, the server determines the set of road disease loss values according to the set of generation loss values and the set of discrimination loss values, which can be referred to as the following formula:

[0149] L street = L gen + γL dis

[0150] wherein, L street may be used to represent a set of road disease loss values of the image generation type adversarial network model; L gen may be used to represent a set of generation loss values of the generator; γ may be used to represent a weight coefficient of the discriminator loss value, and may be used to balance the contribution of the loss of the generator and the discriminator; L dis may be used to represent a set of discrimination loss values of the discriminator.

[0151] For steps S41-S47, the server performs adversarial loss calculation according to each reference road disease discrimination result in the set of reference road disease discrimination results, obtains a set of adversarial loss values of the generator, and performs content loss calculation according to each road disease training image in the set of road disease training images, obtains a set of content loss values of the generator, to further perform content loss calculation according to each road disease training image in the set of road disease training images and each road disease label image in the set of road disease label images, obtains a set of content loss values of the generator, and performs real loss calculation according to each road disease label image in the set of road disease label images, obtains a set of real loss values of the discriminator, to perform false loss calculation according to each road disease training image in the set of road disease training images, obtains a set of false loss values of the discriminator, so as to further determine a set of discrimination loss values of the discriminator according to the set of real loss values and the set of false loss values, and then determine a set of road disease loss values according to the set of generation loss values and the set of discrimination loss values, which is conducive to determining a more accurate target image generation type adversarial network model based on the set of road disease loss values in subsequent steps.

[0152] S50: According to the set of road disease loss values, the initial image generation type adversarial network model is adjusted in parameters to obtain a target image generation type adversarial network model.

[0153] The target image generation type adversarial network model may be an image generation type adversarial network model obtained by adjusting the initial image generation type adversarial network model in parameters according to the set of road disease loss values. It can be understood that the target image generation type adversarial network model may be a model with more clear target orientation, higher accuracy and more stability.

[0154] It can be understood that the aforementioned set of road disease loss values can provide direction and basis for parameter adjustment of the model. The server can determine which parameters in the model need to be adjusted and how to adjust these parameters to reduce the loss value and improve the performance of the model by analyzing the changes of the road disease loss value. For example, if the loss value of the generator is high, the network structure, model learning rate, regularization parameter, etc. of the generator may need to be adjusted to improve the quality and realism of the generated images; if the loss value of the discriminator is high, the network structure of the discriminator may need to be adjusted, the parameters may be initialized, etc. to improve the discrimination ability of the discriminator, which is not limited in the present application.

[0155] After the server adjusts the parameters and iteratively trains for multiple times, when the loss value of the model is within an acceptable range and the performance on the validation set is good, the target image generative adversarial network model can be considered to be obtained. In the task of processing road disease images, the generator of the target image generative adversarial network model can generate more realistic road disease training images, and the discriminator can more accurately judge the authenticity of the input images, greatly improving the performance and reliability of the model.

[0156] S60: According to the target image generative adversarial network model, road disease identification is performed on each to-be-detected road image in the to-be-detected road image set to obtain a target road disease discrimination result.

[0157] The target road disease discrimination result can be a discrimination result obtained by performing road disease identification on a to-be-detected road image based on the target image generative adversarial network model. The target road disease discrimination result can include, but is not limited to, whether the to-be-detected road image identified has road disease, and if the to-be-detected road image has road disease, the type of road disease, the severity of road disease, the location of road disease, etc. The present application is not limited in this regard.

[0158] The target road disease discrimination result can provide an important decision basis for road maintenance and management departments. For example, according to the identified disease type and severity, the management department can develop a corresponding maintenance plan and arrange maintenance personnel to repair in a timely manner. The target road disease discrimination result can be used for data analysis and prediction. For example, by statistically analyzing a large number of road disease identification results, the distribution and development trend of road diseases can be understood. The target road disease discrimination result can be combined with an intelligent transportation system to provide real-time road condition information for vehicles. For example, a vehicle navigation system can provide the best driving route for the driver according to the identified road disease conditions to avoid passing through a road section with serious disease, which is not limited in the present application.

[0159] It can be seen that in the above scheme, the server can generate a road disease training image set according to the road disease marked image set based on the generator in the initial image generative adversarial network model, and determine a reference road disease discrimination result set according to the road disease marked image set and the road disease training image set based on the discriminator in the initial image generative adversarial network model, so as to determine a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, so as to further adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model, and then perform road disease identification on each road image to be detected in the road image to be detected set according to the target image generative adversarial network model, to obtain a target road disease discrimination result more accurate and more in line with the demand of road disease identification.

[0160] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0161] In an embodiment, a road disease prediction device based on an image generative adversarial network is provided, which corresponds one-to-one to the road disease prediction method based on an image generative adversarial network in the above embodiment. As shown in the figure, the road disease prediction device based on an image generative adversarial network includes an acquisition module 101, a generation module 102, a first determination module 103, a second determination module 104, a first processing module 105 and a second processing module 106. The functions of each functional module are described in detail as follows: Figure 3

[0162] The acquisition module 101 is configured to acquire a road disease marked image set with road disease category labels and a road image set to be detected.

[0163] The generation module 102 is configured to generate a road disease training image set according to the road disease marked image set based on the generator in the initial image generative adversarial network model.

[0164] The first determination module 103 is configured to determine a reference road disease discrimination result set according to the road disease marked image set and the road disease training image set based on the discriminator in the initial image generative adversarial network model.

[0165] ​The second determining module 104 is configured to determine a set of road disease loss values according to the set of reference road disease discrimination results, the set of road disease training images, and the set of road disease marked images.

[0166] The first processing module 105 is configured to perform parameter adjustment on the initial image generative adversarial network model according to the set of road disease loss values, to obtain a target image generative adversarial network model.

[0167] The second processing module 106 is configured to perform road disease recognition on each of the set of to-be-detected road images according to the target image generative adversarial network model, to obtain target road disease discrimination results.

[0168] In an embodiment, the generating module 102 is configured to generate the set of road disease training images according to the set of road disease marked images based on a generator in the initial image generative adversarial network model, and specifically configured to:

[0169] extract features from each of the set of road disease marked images based on the generator in the initial image generative adversarial network model, to obtain a set of road disease feature vectors;

[0170] perform dimension increasing processing on each of the set of random noise vectors according to the set of road disease feature vectors, to obtain a set of random dimension-increased vectors;

[0171] perform feature mapping processing on each of the set of random dimension-increased vectors according to the set of road disease feature vectors, to obtain a set of random feature maps;

[0172] perform up-sampling processing on each of the set of random feature maps according to the set of road disease feature vectors, to obtain a set of random high-resolution feature maps;

[0173] perform road disease feature establishment processing on each of the set of random high-resolution feature maps according to the set of road disease feature vectors, to obtain the set of road disease training images.

[0174] In an embodiment, the first determining module 103 is configured to determine the set of reference road disease discrimination results according to the set of road disease marked images and the set of road disease training images based on a discriminator in the initial image generative adversarial network model, and specifically configured to:

[0175] extract road disease features from each of the set of road disease marked images and each of the set of road disease training images, to obtain a set of road disease feature maps;

[0176] perform a max-pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease enhanced feature map set;

[0177] perform an average-pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease smooth feature map set;

[0178] perform splicing processing on each road disease local feature map in the road disease local feature map set and the corresponding road disease global feature map in the road disease global feature map set respectively to obtain a road disease splicing feature map set;

[0179] perform attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a road disease attention feature map set;

[0180] perform full connection processing on each road disease attention feature map in the road disease attention feature map set to obtain a reference road disease discrimination result set.

[0181] In an embodiment, the first determination module 103 is configured to perform attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a road disease attention feature map set, and specifically configured to:

[0182] perform road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set;

[0183] perform road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set;

[0184] perform attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set respectively to obtain a fusion attention feature map set;

[0185] perform deep convolution operation on each fusion attention feature map in the fusion attention feature map set to obtain a deep attention feature map set;

[0186] perform residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set;

[0187] perform self-adaptive activation processing on each residual attention feature map in the residual attention feature map set to obtain a road disease attention feature map set.

[0188] In an embodiment, the second determining module 104 is configured to determine a set of road disease loss values according to the set of reference road disease discrimination results, the set of road disease training images and the set of road disease marked images, and specifically configured to:

[0189] According to each reference road disease discrimination result in the set of reference road disease discrimination results, an adversarial loss calculation is performed to obtain a set of adversarial loss values of the generator;

[0190] According to each road disease training image in the set of road disease training images and each road disease marked image in the set of road disease marked images, a content loss calculation is performed to obtain a set of content loss values of the generator;

[0191] According to the set of adversarial loss values and the set of content loss values, a set of generation loss values of the generator is determined;

[0192] According to each road disease marked image in the set of road disease marked images, a real loss calculation is performed to obtain a set of real loss values of the discriminator;

[0193] According to each road disease training image in the set of road disease training images, a fake loss calculation is performed to obtain a set of fake loss values of the discriminator;

[0194] According to the set of real loss values and the set of fake loss values, a set of discrimination loss values of the discriminator is determined;

[0195] According to the set of generation loss values and the set of discrimination loss values, the set of road disease loss values is determined.

[0196] The present application provides a road disease prediction device based on an image generative adversarial network. By obtaining a set of road disease marked images with road disease category labels and a set of to-be-detected road images, a generator in an initial image generative adversarial network model can be used to generate a set of road disease training images based on the set of road disease marked images, and a discriminator in the initial image generative adversarial network model can be used to determine a set of reference road disease discrimination results based on the set of road disease marked images and the set of road disease training images. Thus, a set of road disease loss values can be determined based on the set of reference road disease discrimination results, the set of road disease training images and the set of road disease marked images, so that the initial image generative adversarial network model can be further adjusted in parameters based on the set of road disease loss values to obtain a target image generative adversarial network model, and then each to-be-detected road image in the set of to-be-detected road images can be identified for road disease based on the target image generative adversarial network model to obtain a target road disease discrimination result that is more accurate and more in line with the requirements of road disease identification.

[0197] Specific limitations regarding the road defect prediction device based on image-generative adversarial networks can be found in the limitations of the road defect prediction method based on image-generative adversarial networks mentioned above, and will not be repeated here. Each module in the aforementioned road defect prediction device based on image-generative adversarial networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0198] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a road defect prediction method based on image generative adversarial networks on the server side.

[0199] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a road defect prediction method based on image generative adversarial networks.

[0200] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0201] Obtain a set of road defect labeled images with road defect category tags, and a set of road images to be detected;

[0202] The generator in the initial image generative adversarial network model generates a road disease training image set according to the road disease marked image set;

[0203] The discriminator in the initial image generative adversarial network model determines a reference road disease discrimination result set according to the road disease marked image set and the road disease training image set;

[0204] The road disease loss value set is determined according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set;

[0205] The initial image generative adversarial network model is adjusted in parameters according to the road disease loss value set, and a target image generative adversarial network model is obtained;

[0206] According to the target image generative adversarial network model, road disease recognition is performed on each road image to be detected in the road image set to be detected, and a target road disease discrimination result is obtained.

[0207] The present application provides a kind of computer equipment, by obtaining road disease marked image set with road disease class label, and road image set to be detected, can be based on the generator in the initial image generative adversarial network model, according to road disease marked image set, generate road disease training image set, and based on the discriminator in the initial image generative adversarial network model, according to road disease marked image set and road disease training image set, determine reference road disease discrimination result set, so that it can be according to reference road disease discrimination result set, road disease training image set and road disease marked image set, determine road disease loss value set, to further according to road disease loss value set, the initial image generative adversarial network model is adjusted in parameters, and target image generative adversarial network model is obtained, and then according to target image generative adversarial network model, road disease recognition is performed on each road image to be detected in the road image set to be detected, and more accurate, more target road disease discrimination result that meets the demand of road disease recognition is obtained.

[0208] In one embodiment, a computer-readable storage medium having a computer program stored thereon is provided, and the computer program is executed by a processor to implement the following steps:

[0209] Obtain road disease marked image set with road disease class label, and road image set to be detected;

[0210] The generator in the initial image generative adversarial network model generates a road disease training image set according to the road disease marked image set;

[0211] The discriminator in the initial image generative adversarial network model is based on the road disease marked image set and the road disease training image set to determine a reference road disease discrimination result set;

[0212] According to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, a road disease loss value set is determined;

[0213] According to the road disease loss value set, the initial image generative adversarial network model is adjusted in parameters to obtain a target image generative adversarial network model;

[0214] According to the target image generative adversarial network model, road disease recognition is performed on each road image to be detected in the road image set to be detected to obtain a target road disease discrimination result.

[0215] The present application provides a computer readable storage medium, by acquiring a road disease marked image set with road disease category label and a road image set to be detected, a generator in an initial image generative adversarial network model can be used to generate a road disease training image set based on the road disease marked image set, and a discriminator in the initial image generative adversarial network model can be used to determine a reference road disease discrimination result set based on the road disease marked image set and the road disease training image set, so that a road disease loss value set can be determined based on the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, and the initial image generative adversarial network model can be further adjusted in parameters based on the road disease loss value set to obtain a target image generative adversarial network model, and then the target image generative adversarial network model can be used to perform road disease recognition on each road image to be detected in the road image set to be detected to obtain a target road disease discrimination result which is more accurate and more in line with the requirements of road disease recognition.

[0216] It should be noted that the functions or steps that the computer readable storage medium or the computer device can achieve can be referred to the foregoing method embodiments, and the related descriptions of the server side and the client side are not repeated here.

[0217] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0218] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0219] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for road disease prediction based on image generative adversarial network, characterized in that, The method comprises: acquiring a road disease mark image set with road disease category labels and a to-be-detected road image set; generating a road disease training image set according to the road disease mark image set based on a generator in an initial image generative adversarial network model; determining a reference road disease discrimination result set according to the road disease mark image set and the road disease training image set based on a discriminator in the initial image generative adversarial network model; determining a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease mark image set; adjusting parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model; performing road disease identification on each to-be-detected road image in the to-be-detected road image set according to the target image generative adversarial network model to obtain a target road disease discrimination result; the generating of the road disease training image set according to the road disease mark image set based on the generator in the initial image generative adversarial network model comprises: performing feature extraction on each road disease mark image in the road disease mark image set based on the generator in the initial image generative adversarial network model to obtain a road disease feature vector set; performing dimension increasing processing on each random noise vector in a random noise vector set according to the road disease feature vector set to obtain a random dimension-increased vector set; performing feature mapping processing on each random dimension-increased vector in the random dimension-increased vector set according to the road disease feature vector set to obtain a random feature map set; performing up-sampling processing on each random feature map in the random feature map set according to the road disease feature vector set to obtain a random high-resolution feature map set; performing road disease feature establishment processing on each random high-resolution feature map in the random high-resolution feature map set according to the road disease feature vector set to obtain the road disease training image set; the determining of the reference road disease discrimination result set according to the road disease mark image set and the road disease training image set based on the discriminator in the initial image generative adversarial network model comprises: performing road disease feature extraction on each road disease mark image in the road disease mark image set and each road disease training image in the road disease training image set to obtain a road disease feature map set; performing maximum pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease enhanced feature map set; performing average pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease smoothed feature map set; respectively performing splicing processing on each road disease local feature map in the road disease local feature map set and a corresponding road disease global feature map in the road disease global feature map set to obtain a road disease spliced feature map set; The attention calculation is performed on each road disease splicing feature map in the road disease splicing feature map set, and a road disease attention feature map set is obtained. The full connection processing is performed on each road disease attention feature map in the road disease attention feature map set, and a reference road disease discrimination result set is obtained. 2.The image-based generative adversarial network-based road disease prediction method of claim 1, wherein, The attention calculation is performed on each road disease splicing feature map in the road disease splicing feature map set, and a road disease attention feature map set is obtained. The road disease local attention calculation is performed on each road disease splicing feature map in the road disease splicing feature map set, and a local attention feature map set is obtained. The road disease global attention calculation is performed on each road disease splicing feature map in the road disease splicing feature map set, and a global attention feature map set is obtained. The attention fusion is respectively performed on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set, and a fusion attention feature map set is obtained. The deep convolution operation is performed on each fusion attention feature map in the fusion attention feature map set, and a deep attention feature map set is obtained. The residual connection is performed on each deep attention feature map in the deep attention feature map set, and a residual attention feature map set is obtained. The adaptive activation processing is performed on each residual attention feature map in the residual attention feature map set, and a road disease attention feature map set is obtained. 3.The image-based generative adversarial network-based road disease prediction method of claim 2, wherein, The adversarial loss calculation is performed according to each reference road disease discrimination result in the reference road disease discrimination result set, and an adversarial loss value set of the generator is obtained. The content loss calculation is performed according to each road disease training image in the road disease training image set and each road disease labeled image in the road disease labeled image set, and a content loss value set of the generator is obtained. The generation loss value set of the generator is determined according to the adversarial loss value set and the content loss value set. The real loss calculation is performed according to each road disease labeled image in the road disease labeled image set, and a real loss value set of the discriminator is obtained. The fake loss calculation is performed according to each road disease training image in the road disease training image set, and a fake loss value set of the discriminator is obtained. The discrimination loss value set of the discriminator is determined according to the real loss value set and the fake loss value set. The road disease loss value set is determined according to the generation loss value set and the discrimination loss value set. The road disease prediction device based on the image generative adversarial network comprises: 4.A road disease prediction device based on an image generative adversarial network, characterized by An acquisition module is configured to acquire a road disease labeled image set with road disease category labels and a to-be-detected road image set. ​ The generating module is configured to generate a generator in an initial image generative adversarial network model, and generate a road disease training image set according to the road disease labeled image set. The first determining module is configured to determine a reference road disease discrimination result set according to the road disease labeled image set and the road disease training image set based on a discriminator in the initial image generative adversarial network model. The second determining module is configured to determine a road disease loss value set according to the reference road disease discrimination result set, the road disease training image set and the road disease labeled image set. The first processing module is configured to adjust parameters of the initial image generative adversarial network model according to the road disease loss value set, and obtain a target image generative adversarial network model. The second processing module is configured to perform road disease identification on each of the to-be-detected road image set according to the target image generative adversarial network model, and obtain a target road disease discrimination result. The generating module is configured to generate a generator in an initial image generative adversarial network model, and generate a road disease training image set according to the road disease labeled image set. The generating module is configured to generate a generator in an initial image generative adversarial network model, and generate a road disease training image set according to the road disease labeled image set. The generating module is configured to generate a generator in an initial image generative adversarial network model, and generate a road disease training image set according to the road disease labeled image set. The generating module is configured to generate a generator in an initial image generative adversarial network model, and generate a road disease training image set according to the road disease labeled image set. The first determining module is configured to determine a reference road disease discrimination result set according to the road disease labeled image set and the road disease training image set based on a discriminator in the initial image generative adversarial network model. The first determining module is configured to determine a reference road disease discrimination result set according to the road disease labeled image set and the road disease training image set based on a discriminator in the initial image generative adversarial network model. The first determining module is configured to determine a reference road disease discrimination result set according to the road disease labeled image set and the road disease training image set based on a discriminator in the initial image generative adversarial network model. ​ ​ ​ each road disease local feature map in the road disease local feature map set is spliced with a corresponding road disease global feature map in the road disease global feature map set respectively to obtain a road disease splicing feature map set; each road disease splicing feature map in the road disease splicing feature map set is subjected to attention calculation to obtain a road disease attention feature map set; each road disease attention feature map in the road disease attention feature map set is subjected to full connection processing to obtain a reference road disease discrimination result set.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the road disease prediction method based on the image generative adversarial network in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to realize the road disease prediction method based on the image generative adversarial network in any one of claims 1 to 3.

Images