Rainy day image generation model training method and device, equipment, medium and product

Abstract

The invention provides a training method and device for a rainy day image generation model, equipment, a medium and a product. The training method comprises the following steps: acquiring a training data set; the training data set comprises a real conventional image, a real rainy day image, a real conventional segmented image and a real rainy day segmented image; determining a first loss between the real conventional image and a first rainy day image; the first rainy day image is generated by using the real conventional image; determining a second loss between the real conventional image, the real rainy day image and the first rainy day image; determining a third loss among the real conventional image, the first rainy day image, the real conventional segmented image and the real rainy day segmented image; and training a rainy day image generation model by using the first loss, the second loss and the third loss until the rainy day image generation model converges.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a training method, device, equipment, medium and product for a rainy day image generation model. Background Art

[0002] In recent years, image visual perception technology has been widely used in the field of intelligent driving of automobiles. At present, it mainly uses on-board cameras to collect images of the driving environment of the car, and further perceives driving-related targets in the image through algorithm networks. In related technologies, for different rainy weather scenes, the original data of different degrees of rainy weather (normal, light rain, heavy rain) scenes are collected and labeled on the training data side to meet the needs of rich rainy scene scenes, but it requires a lot of human resources for labeling. At the same time, weather scenes with different rainfall levels are sporadic, which makes it impossible to generate high-quality rainy road scene images. Summary of the invention

[0003] The purpose of this application is to provide a training method, device, equipment, medium and product for a rainy day image generation model to solve the problem that the existing technology cannot generate high-quality rainy day road scene images.

[0004] The technical solutions adopted in this application are as follows:

[0005] A training method for a rainy day image generation model, the training method comprising: obtaining a training data set; the training data set comprising a real regular image, a real rainy day image, a real regular segmented image and a real rainy day segmented image; determining a first loss between the real regular image and a first rainy day image; the first rainy day image is generated using the real regular image; determining a second loss between the real regular image, the real rainy day image and the first rainy day image; determining a third loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image; and using the first loss, the second loss and the third loss to train a rainy day image generation model until the rainy day image generation model converges.

[0006] According to the above technical means, on the one hand, by obtaining a comprehensive training data set including real regular images, real rainy day images, real regular segmented images and real rainy day segmented images, it is helpful to improve the adaptability of the rainy day image generation model to different weather conditions and image contents, thereby generating more realistic and natural rainy day images; on the other hand, by determining the loss between the real regular image and the first rainy day image, the loss between the real regular image, the real rainy day image and the first rainy day image, and the loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image, it is helpful for the rainy day image generation model to maintain the content, structure and semantic information of the original image while generating rainy day images, thereby improving the quality and credibility of the generated rainy day images.

[0007] Further, a training data set is obtained, including: collecting an initial real conventional image and an initial real rainy day image; segmenting the initial real conventional image and the initial real rainy day image respectively, and correspondingly obtaining an initial real conventional segmented image and an initial real rainy day segmented image; scaling and converting the initial real conventional image, the initial real rainy day image, the initial real conventional segmented image, and the initial real rainy day segmented image in sequence respectively, and correspondingly obtaining the real conventional image, the real rainy day image, the real conventional segmented image, and the real rainy day segmented image; based on the real conventional image, the real rainy day image, the real conventional segmented image, and the real rainy day segmented image, the training data set is determined.

[0008] According to the above technical means, by obtaining a comprehensive training data set including real regular images, real rainy day images, real regular segmented images and real rainy day segmented images, it helps to improve the adaptability of the rainy day image generation model to different weather conditions and image contents, thereby generating more realistic and natural rainy day images.

[0009] Further, the initial real conventional image is scaled and converted in sequence to obtain the real conventional image, including: scaling the initial real conventional image to obtain a scaled real conventional image; performing color space conversion on the scaled real conventional image to obtain a first real conventional image; converting the format of the first real conventional image into a first format to obtain the real conventional image.

[0010] According to the above technical means, on the one hand, the high-resolution initial real regular image is converted into a lower-resolution image through scaling processing, which significantly reduces the amount of calculation and memory usage required for the rainy day image generation model to process the image; on the other hand, the color space conversion of the scaled real regular image helps the rainy day image generation model to better separate and utilize the brightness information and chromaticity information in the image.

[0011] Furthermore, the first rainy day image is generated by the generator module using the real regular image, and the generator module includes four layers of upsampling modules with different multiples and four layers of downsampling modules with different multiples. The real regular image is input into the generator module, and the first rainy day image is output, including: downsampling the real regular image in different multiples in sequence through the four layers of downsampling modules with different multiples to obtain corresponding downsampling features; upsampling the downsampling features output by the fourth downsampling module through the first upsampling module to obtain the first upsampling features; starting from the second upsampling module, the upsampling features output by the previous upsampling module are spliced ​​with the downsampling features output by the downsampling module with the corresponding multiples to obtain corresponding upsampling features; splicing the upsampling features output by the third upsampling module with the downsampling features output by the downsampling module with the corresponding multiples through the fourth upsampling module to obtain the first rainy day image.

[0012] According to the above technical means, on the one hand, the generator module extracts multi-scale features of real regular images through four layers of downsampling modules with different multiples, which helps to capture the details of different levels in real regular images and provides a rich feature basis for the subsequent generation of rainy day images; on the other hand, in the upsampling stage, the generator module gradually restores the features extracted in the downsampling stage to the resolution of real regular images through the upsampling module. At the same time, through the feature splicing operation, the features of different scales are effectively fused, which helps to generate more realistic and accurate rainy day images.

[0013] Further, determining the second loss between the real regular image, the real rainy day image and the first rainy day image includes: for the same pixel point in the image, subtracting the pixel value of the first rainy day image from the pixel value of the real regular image to obtain a first pixel difference; subtracting the pixel value of the first rainy day image from the pixel value of the real rainy day image to obtain a second pixel difference; adding the absolute value of the first pixel difference to the absolute value of the second pixel difference to obtain the loss of the same pixel; summing the losses of all pixels in the image and dividing the sum by the area of ​​the image to obtain the second loss.

[0014] According to the above technical means, on the one hand, the differences between the first rainy day image, the real regular image and the real rainy day image are taken into account at the same time. By calculating the difference pixel by pixel, the change of each pixel in the image can be accurately captured, thereby ensuring that the loss is more accurate; on the other hand, the loss of all pixels is summed and divided by the area of ​​the image to obtain a standardized second loss. This standardized processing makes the loss more comparable and robust, and can be applied to images of different sizes and resolutions.

[0015] Further, determining a third loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image includes: determining a target similarity score between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image; determining an image block loss between an image block in the real regular image and an image block in the first rainy day image; determining a weight coefficient based on the target similarity score and a preset weighted scaling factor; and determining the third loss based on the target similarity score and the weight coefficient.

[0016] According to the above technical means, on the one hand, by determining the target similarity scores between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image, the overall similarity between the images is taken into account; on the other hand, by determining the image block loss between the image blocks in the real regular image and the image blocks in the first rainy day image, the local loss between the image blocks is taken into account, so that the differences between the images can be more comprehensively evaluated, thereby more accurately determining the third loss.

[0017] Further, a target similarity score is determined between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image, including: determining a first similarity between the real regular image and the first rainy day image; determining a second similarity between the real regular segmented image and the real rainy day segmented image; subtracting 1 from the second similarity and then multiplying the result by the first similarity to obtain a first similarity score; subtracting 1 from the first similarity and then multiplying the result by the second similarity to obtain a second similarity score; and using the first similarity score to divide the sum of the second similarity score and a preset value to obtain a target similarity score.

[0018] According to the above technical means, by determining the first similarity between the real regular image and the first rainy day image, and the second similarity between the real regular segmented image and the real rainy day segmented image, not only the similarity between the real regular image and the first rainy day image, but also the similarity between the two segmented images is considered. By combining these two similarities, a more accurate target similarity score can be obtained, which can more comprehensively evaluate the overall similarity and structural similarity between images.

[0019] Further, the rainy day image generation model is trained using the first loss, the second loss and the third loss until the rainy day image generation model converges, including: respectively obtaining a first weight value corresponding to the first loss, a second weight value corresponding to the second loss and a third weight value corresponding to the third loss; respectively multiplying the first loss by the first weight value, the second loss by the second weight value, and the third loss by the third weight value to obtain a first target loss, a second target loss and a third target loss respectively; adding the first target loss, the second target loss and the third target loss to obtain a target loss; and using the target loss to train the rainy day image generation model until the rainy day image generation model converges.

[0020] According to the above technical means, on the one hand, by introducing multiple losses (first loss, second loss, third loss), the performance of the rainy day image generation model in different aspects can be comprehensively evaluated, thereby ensuring that the generated rainy day images are close to the real rainy day images in multiple dimensions; on the other hand, by setting corresponding weight values ​​for each loss, users can flexibly adjust the contribution of different losses to the total loss according to specific needs, thereby obtaining better training effects.

[0021] A training device for a rainy day image generation model, the training device comprising:

[0022] An acquisition module is used to acquire a training data set; the training data set includes real conventional images, real rainy day images, real conventional segmented images and real rainy day segmented images;

[0023] a discriminator module for determining a first loss between the real regular image and a first rainy image; the first rainy image is generated using the real regular image;

[0024] a discriminator module for determining a second loss between the real regular image, the real rainy image, and the first rainy image;

[0025] a segmentation contrast learning module, configured to determine a third loss between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image;

[0026] A training module is used to train the rainy day image generation model by using the first loss, the second loss and the third loss until the rainy day image generation model converges.

[0027] An electronic device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0028] A computer-readable storage medium stores a computer program, which implements the steps in the above method when executed by a processor.

[0029] A computer program product, comprising a computer program or instructions, wherein the computer program or instructions implement the steps in the above method when executed by a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the implementation process of a training method for a rainy day image generation model proposed in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of a rainy day image generation model construction method and an image generation method implementation flow diagram proposed in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of training data proposed in an embodiment of the present application;

[0033] Figure 4 This is a general schematic diagram of a rainy day image generation model proposed in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a generator module proposed in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of an identifier module proposed in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of a maximum probability constraint proposed in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a segmentation contrast learning module proposed in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the composition structure of a training device for a rainy day image generation model proposed in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of a hardware entity of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0041] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0042] The present application embodiment proposes a training method for a rainy day image generation model, such as Figure 1 As shown, the training method of the rainy day image generation model includes the following steps S101 to S105, wherein:

[0043] Step S101, obtaining a training data set; the training data set includes real regular images, real rainy day images, real regular segmented images and real rainy day segmented images;

[0044] Here, the real regular image may refer to an image that does not contain rainy scenes or images with high noise such as night scenes, the real rainy day image may refer to an image containing rain (for example, a light rain image, a moderate rain image, a heavy rain image, etc.), the real regular segmented image may be a segmented image corresponding to the real regular image, and the real rainy day segmented image may be a segmented image corresponding to the real rainy day image.

[0045] In some embodiments, a real regular image can be obtained by shooting a natural scene under clear or normal weather conditions by a camera on the vehicle. The real regular image can contain a variety of backgrounds, objects, and lighting conditions to ensure that the trained rainy day image generation model can adapt to different environments.

[0046] In some embodiments, a natural scene under rainy conditions may be photographed by a camera on the vehicle to obtain a real rainy day image, which may truly reflect the visual effects of a rainy day, including raindrops, rain lines, rain fog, and the like.

[0047] In some embodiments, the real regular segmented image and the real rainy day segmented image can be generated by using a semantic segmentation technique from a real regular image and a real rainy day image, respectively. Semantic segmentation is a computer vision task that aims to classify each pixel in an image into a specific category (e.g., sky, ground, building, tree, etc.).

[0048] Step S102, determining a first loss between the real regular image and a first rainy day image; the first rainy day image is generated using the real regular image;

[0049] Here, the first loss is a quantitative indicator to measure the difference between the real regular image and the first rainy image.

[0050] In some embodiments, the first rainy day image is generated using a real regular image, and the loss between the real regular image and the first rainy day image can be calculated using some loss function (e.g., L1 loss, L2 loss, perceptual loss, etc.), and this loss can reflect the difference between the first rainy day image and the real regular image.

[0051] Step S103, determining a second loss between the real regular image, the real rainy day image and the first rainy day image;

[0052] Here, the second loss refers to the loss between the real regular image, the real rainy day image and the first rainy day image, and the second loss can be a maximum probability constrained loss.

[0053] In some embodiments, the loss between the real regular image, the real rainy day image and the first rainy day image can be calculated by a maximum probability constrained loss function to obtain a maximum probability constrained loss. The maximum probability constrained loss can reflect the similarity between the generated rainy day image and the real rainy day image in terms of global characteristics or probability distribution. By introducing the real rainy day image as a reference, the generated rainy day image can be further constrained to make it closer to the real rainy day effect.

[0054] Step S104, determining a third loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image;

[0055] Here, the third loss refers to the loss between the true regular image, the first rainy day image, the true regular segmented image, and the true rainy day segmented image.

[0056] In some embodiments, the loss between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image can be calculated by a segmentation contrast learning loss function to obtain a segmentation contrast learning loss. The segmentation contrast learning loss can reflect the semantic consistency of the generated rainy day image with the real rainy day image. By comparing the real regular segmented image with the real rainy day segmented image, it can be ensured that the generated rainy day image is not only visually close to the real image, but also semantically consistent.

[0057] Step S105 , training the rainy day image generation model by using the first loss, the second loss and the third loss until the rainy day image generation model converges.

[0058] In some implementations, during the training of the rainy day image generation model, the first loss, the second loss, and the third loss calculated above are used as optimization targets, and the parameters of the rainy day image generation model are adjusted in reverse order to gradually approach the optimal solution. This process is continued until the rainy day image generation model converges.

[0059] In the embodiments of the present application, on the one hand, by acquiring a comprehensive training data set including real regular images, real rainy day images, real regular segmented images and real rainy day segmented images, it helps to improve the adaptability of the rainy day image generation model to different weather conditions and image contents, thereby generating more realistic and natural rainy day images; on the other hand, by determining the loss between the real regular image and the first rainy day image, the loss between the real regular image, the real rainy day image and the first rainy day image, and the loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image, it helps the rainy day image generation model to maintain the content, structure and semantic information of the original image while generating the rainy day image, thereby improving the quality and credibility of the generated rainy day images.

[0060] In the embodiment of the present application, the implementation of step S101 "obtaining a training data set" may include the following steps S111 to S114, wherein:

[0061] Step S111, collecting an initial real conventional image and an initial real rainy day image;

[0062] In some embodiments, a total of 10w frames of initial real regular images are collected by the camera on the vehicle, and the initial real regular images do not contain rainy scenes and scenes with large noise such as night; a total of 10w frames of initial real rainy images with different rain levels are collected by the camera on the vehicle, and the initial real rainy images contain three scene data of light rain, moderate rain, and heavy rain, which are evenly distributed. In the heavy rain scene images, there are obvious raindrops blocking the lens, water accumulation on the road surface, and road wetting.

[0063] Step S112, segmenting the initial real conventional image and the initial real rainy day image respectively, and obtaining an initial real conventional segmented image and an initial real rainy day segmented image correspondingly;

[0064] In some embodiments, segmented images can be generated by using an open source segmentation model, which can mainly segment common driving scene elements such as road surface and sky. For example, the initial real conventional image is segmented to obtain an initial real conventional segmented image, and the initial real rainy day image is segmented to obtain an initial real rainy day segmented image. The segmentation process usually involves dividing the image into different semantic regions (such as sky, ground, buildings, etc.), which helps the rainy day image generation model to better understand the image content when generating rainy day images.

[0065] Step S113, respectively scaling and converting the initial real conventional image, the initial real rainy day image, the initial real conventional segmented image, and the initial real rainy day segmented image in sequence, to obtain the real conventional image, the real rainy day image, the real conventional segmented image, and the real rainy day segmented image, respectively;

[0066] In some embodiments, the initial real regular image, the initial real rainy day image, the initial real regular segmented image, and the initial real rainy day segmented image are scaled respectively. The scaling is to ensure that all images have the same size, which is convenient for subsequent rainy day image generation model training and processing.

[0067] In some implementations, the scaled image is converted, which may include operations such as color correction, brightness adjustment, and contrast enhancement to ensure consistency and diversity of training data, and help the rainy day image generation model better adapt to different image conditions.

[0068] Step S114, determining the training data set based on the real regular image, the real rainy day image, the real regular segmented image and the real rainy day segmented image.

[0069] In some embodiments, after the above steps, the obtained real regular image, real rainy day image, real regular segmented image and real rainy day segmented image together constitute a training data set. This training data set contains all the information required for training the rainy day image generation model, including the original image content, rainy day effect and corresponding semantic segmentation information.

[0070] In an embodiment of the present application, by acquiring a comprehensive training data set including real regular images, real rainy day images, real regular segmented images and real rainy day segmented images, it helps to improve the adaptability of the rainy day image generation model to different weather conditions and image contents, thereby generating more realistic and natural rainy day images.

[0071] In the embodiment of the present application, the implementation of "scaling and converting the initial real conventional image in sequence to obtain the real conventional image" in step S113 may include the following steps S121 to S123, wherein:

[0072] Step S121, scaling the initial real conventional image to obtain a scaled real conventional image;

[0073] In some embodiments, since the resolution of the initial real conventional image is very high (e.g., 3840x2160), such a high-resolution image contains a large number of details, but when processed in the rainy image generation model, a large amount of computing resources are consumed. Therefore, in order to reduce the computing cost and improve the processing speed, the initial real conventional image is scaled to reduce its resolution from 3840x2160 to 960x512. This scaling operation reduces the number of pixels in the image, thereby reducing the amount of computing required when the rainy image generation model processes the image.

[0074] Step S122, performing color space conversion processing on the scaled real regular image to obtain a first real regular image;

[0075] In some embodiments, the scaled image is converted from the Red Green Blue (RGB) color space to the Luminance and Chroma (YUV, Luminance (Y) and Chroma (U and V)) color space. Since the RGB color space represents colors based on the combination of red, green, and blue, and the YUV color space is divided into two parts: luminance (Y) and chroma (U and V). When dealing with tasks on rainy roads, luminance information is often more susceptible to noise, while chroma information is relatively stable. Therefore, converting the scaled real regular image to the YUV color space can help the rainy day image generation model better separate and utilize this information, thereby improving the robustness of the rainy day image generation model.

[0076] Step S123: convert the format of the first real regular image into a first format to obtain the real regular image.

[0077] Here, the first format may be a tensor format, such as a tensor format. A tensor is a data structure used to store and operate multidimensional arrays in a deep learning framework (such as PyTorch).

[0078] In some embodiments, since the rainy day image generation model training adopts the PyTorch structure, it is necessary to convert the image data of the first real regular image into a tensor format so as to serve as the input of the rainy day image generation model. This step is the key to ensure that the real regular image can be correctly received and processed by the rainy day image generation model.

[0079] In the embodiments of the present application, on the one hand, the high-resolution initial real conventional image is converted into a lower-resolution image through scaling processing, which significantly reduces the amount of calculation and memory usage required for the rainy day image generation model to process the image; on the other hand, the scaled real conventional image is converted into a color space, which helps the rainy day image generation model to better separate and utilize the brightness information and chromaticity information in the image.

[0080] In the embodiment of the present application, the first rainy day image is generated by the generator module using the real regular image, the generator module includes four layers of upsampling modules with different multiples and four layers of downsampling modules with different multiples, and the implementation of "inputting the real regular image into the generator module and outputting the first rainy day image" may include the following steps S131 to S134, wherein:

[0081] Step S131, downsampling the real regular image in different multiples in sequence through the four layers of downsampling modules with different multiples, and obtaining corresponding downsampling features;

[0082] Here, the generator module is a deep learning model based on the U-Net network structure, including four layers of upsampling modules with different multiples and four layers of downsampling modules with different multiples. These downsampling modules downsample the input image by 2, 4, 8, and 16 times respectively to extract features of different scales. These upsampling modules are used to gradually restore the features extracted in the downsampling stage to the resolution of the original image.

[0083] In some embodiments, a real regular image (e.g., a road image in clear weather) is input into the generator module. The real regular image first passes through a 2x downsampling module for preliminary feature extraction. Then, it passes through 4x, 8x, and 16x downsampling modules in sequence, and each downsampling module outputs corresponding downsampling features, which contain information about the image at different scales.

[0084] Step S132, upsampling the downsampled features output by the fourth downsampled module through the first upsampled module to obtain a first upsampled feature;

[0085] In some embodiments, in the upsampling stage, the features output by the 16-fold downsampling module are first upsampled by a 16-fold upsampling module to obtain a first upsampled feature.

[0086] Step S133, starting from the second upsampling module, concatenate the upsampling features output by the previous upsampling module with the downsampling features output by the downsampling module of the corresponding multiple, and obtain the upsampling features accordingly;

[0087] In some embodiments, at the 8x, 4x, and 2x upsampling modules, the upsampling features output by the previous upsampling module are spliced ​​with the downsampling features output by the downsampling module of the corresponding multiple. This splicing operation helps to fuse feature information of different scales and improve the quality of the generated image.

[0088] Step S134, using a fourth upsampling module to splice the upsampling features output by the third upsampling module and the downsampling features output by a downsampling module of a corresponding multiple, to obtain the first rainy day image.

[0089] In some embodiments, at the 2x upsampling module, the upsampled features output by the 4x upsampling module are concatenated with the downsampled features output by the 2x downsampling module to obtain a first rainy day image. The first rainy day image is visually similar to a real rainy day road image.

[0090] In the embodiment of the present application, on the one hand, the generator module extracts multi-scale features of real regular images through four layers of downsampling modules with different multiples, which helps to capture the details of different levels in the real regular images and provides a rich feature basis for the subsequent generation of rainy day images; on the other hand, in the upsampling stage, the generator module gradually restores the features extracted in the downsampling stage to the resolution of the real regular images through the upsampling modules. At the same time, through the feature splicing operation, the features of different scales are effectively fused, which helps to generate more realistic and accurate rainy day images.

[0091] In the embodiment of the present application, the implementation of “determining the second loss between the real regular image, the real rainy day image and the first rainy day image” in step S103 may include the following steps S141 to S144, wherein:

[0092] Step S141, for a same pixel point in the image, subtract the pixel value of the first rainy day image from the pixel value of the real regular image to obtain a first pixel difference value;

[0093] Here, in the process of calculating the second loss between the real regular image, the real rainy day image and the first rainy day image, it is necessary to select pixels at the same position from the first rainy day image, the real regular image and the real rainy day image for comparison, and use the comparison results of all pixels as the second loss.

[0094] In some embodiments, the pixel values ​​at the same pixel point of the first rainy day image and the real regular image are subtracted to obtain a first pixel difference value, which reflects the difference between the first rainy day image and the real regular image at the same pixel point.

[0095] Step S142, subtracting the pixel value of the first rainy day image from the pixel value of the real rainy day image to obtain a second pixel difference value;

[0096] In some embodiments, the pixel values ​​at the same pixel point of the first rainy day image and the real rainy day image are subtracted to obtain a second pixel difference value, and the second pixel difference value reflects the difference at the same pixel point of the first rainy day image and the real rainy day image.

[0097] Step S143, adding the absolute value of the first pixel difference and the absolute value of the second pixel difference to obtain the loss of the same pixel;

[0098] In some implementations, taking the absolute values ​​of the first pixel difference and the second pixel difference respectively can ensure that the difference is positive regardless of whether it is positive or negative, and the positive value indicates the difference between the images. Then, the two positive values ​​are added to obtain the loss of the same pixel point, which reflects the comprehensive difference between the first rainy day image, the real regular image, and the real rainy day image at the same pixel point.

[0099] Step S144, summing the losses of all pixels in the image and dividing the sum by the area of ​​the image to obtain the second loss.

[0100] In some embodiments, the above steps are repeated for all pixels in the image to obtain the loss of each pixel; the loss values ​​of all pixels are added together to obtain the total loss of the entire image; the total loss of the image is divided by the area of ​​the image (i.e., the total number of pixels) to obtain a standardized loss value of the overall difference between the three images, which is called the second loss.

[0101] In some embodiments, the second loss is determined by the following formula (1):

[0102]

[0103] Among them, L max (X,Y,Z) represents the second loss; H represents the height of the image; W represents the width of the image; D(X) i,j represents the pixel value of the real regular image at the pixel point (i, j); D(Y) i,j represents the pixel value of the real rainy day image at the pixel point (i, j); D(Z) i,j Represents the pixel value of the first rainy day image at pixel point (i, j).

[0104] In the embodiment of the present application, on the one hand, the differences between the first rainy day image, the real regular image and the real rainy day image are taken into account at the same time, and the difference is calculated pixel by pixel, so that the change of each pixel in the image can be accurately captured, thereby ensuring that the loss is more accurate; on the other hand, the loss of all pixels is summed and divided by the area of ​​the image to obtain a standardized second loss. This standardized processing makes the loss more comparable and robust, and can be applied to images of different sizes and resolutions.

[0105] In the embodiment of the present application, the implementation of “determining the third loss between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image” in step S104 may include the following steps S151 to S154, wherein:

[0106] Step S151, determining target similarity scores among the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image;

[0107] Here, the target similarity score is used to measure the similarity between the true regular image, the first rainy day image, the true regular segmented image, and the true rainy day segmented image.

[0108] In some embodiments, the target similarity scores among the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image may be calculated by a similarity function. If the target similarity score is large, the similarity between the above four images is high; if the target similarity score is small, the similarity between the above four images is low.

[0109] In some implementations, the target similarity score may be determined by the following formula (2):

[0110]

[0111] Among them, F(i,j) represents the target similarity score; Sim(X,Z) represents the first similarity between the real regular image and the first rainy day image; mIoU(X,Y) represents the second similarity between the real regular segmented image and the real rainy day segmented image.

[0112] Step S152, determining the image block loss between the image block in the real regular image and the image block in the first rainy day image;

[0113] Here, patch loss is used to capture the differences in local regions in the image.

[0114] In some embodiments, the image block loss between the image block in the real regular image and the image block in the first rainy day image is calculated by an image block loss function, and the image block loss can reflect the difference between the local area in the real regular image and the first rainy day image.

[0115] In some embodiments, the image block loss between the image block in the real regular image and the image block in the first rainy day image is determined by the following formula (3):

[0116]

[0117] Among them, L PatchNCE (X, Z) represents the image patch loss between the image patch in the real regular image and the image patch in the first rainy day image; N represents the number of image patches; x i～N are the image block features from i to N in the real regular image; z i～N are the features of the i-th to N-th image blocks in the first rainy day image, and τ is a temperature parameter used to control the smoothness of the decision boundary.

[0118] Step S153, determining a weight coefficient based on the target similarity score and a preset weighted scaling factor;

[0119] In some embodiments, the weight coefficient is determined by the following formula (4):

[0120]

[0121] Among them, w (i,j) represents the weight coefficient; e represents the base of the natural logarithm; α and β are preset weighted scaling factors.

[0122] Step S154: determining the third loss based on the target similarity score and the weight coefficient.

[0123] In some embodiments, the third loss is determined by the following formula (5):

[0124]

[0125] Among them, L compare (X,Y,Z) represents the third loss.

[0126] In the embodiment of the present application, on the one hand, by determining the target similarity scores between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image, the overall similarity between the images is taken into account; on the other hand, by determining the image block loss between the image blocks in the real regular image and the image blocks in the first rainy day image, the local loss between the image blocks is taken into account, so that the differences between the images can be more comprehensively evaluated, thereby more accurately determining the third loss.

[0127] In the embodiment of the present application, the implementation of “determining the target similarity scores between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image” in step S151 may include the following steps S161 to S165, wherein:

[0128] Step S161, determining a first similarity between the real regular image and the first rainy day image;

[0129] In some embodiments, the similarity between the real normal image and the first rainy day image is calculated by a similarity function (eg, Sim) to obtain a first similarity. The first similarity may reflect the degree of similarity between the image under normal weather conditions and the generated rainy day image.

[0130] Step S162, determining a second similarity between the real conventional segmented image and the real rainy day segmented image;

[0131] In some embodiments, the similarity between the real conventional segmented image and the real rainy day segmented image is calculated by a similarity function (eg, mIoU) to obtain the second similarity. The first similarity may reflect the similarity between the segmented images under the two weather conditions.

[0132] Step S163, subtracting 1 from the second similarity and then multiplying the result by the first similarity to obtain a first similarity score;

[0133] In some embodiments, subtracting the second similarity from 1 and then multiplying the difference by the first similarity can combine the two similarity information and emphasize the importance of the first similarity (the similarity between the real regular image and the first rainy day image) when the similarity between the two segmented images is low (i.e., the two segmented images are very different).

[0134] Step S164, subtracting 1 from the first similarity and then multiplying the result by the second similarity to obtain a second similarity score;

[0135] In some embodiments, the first similarity is subtracted from 1, and then the difference is multiplied by the second similarity. The two similarity information can be combined, and the importance of the second similarity (i.e., the similarity between the real regular image segmented image and the real rainy image segmented image) can be emphasized when the similarity between the real regular image and the first rainy image is low (i.e., the real regular image and the first rainy image are very different).

[0136] Step S165: Divide the sum of the second similarity score and a preset value by the first similarity score to obtain a target similarity score.

[0137] In some implementations, the first similarity score is used to divide the sum of the second similarity score and a preset value to obtain a target similarity score.

[0138] In an embodiment of the present application, by determining a first similarity between the real regular image and the first rainy day image, and a second similarity between the real regular segmented image and the real rainy day segmented image, not only the similarity between the real regular image and the first rainy day image is considered, but also the similarity between the two segmented images is considered. By combining these two similarities, a more accurate target similarity score can be obtained, which can more comprehensively evaluate the overall similarity and structural similarity between images.

[0139] In the embodiment of the present application, the implementation of “using the first loss, the second loss and the third loss to train the rainy day image generation model until the rainy day image generation model converges” in step S105 may include the following steps S171 to S174, wherein:

[0140] Step S171, respectively obtaining a first weight value corresponding to the first loss, a second weight value corresponding to the second loss, and a third weight value corresponding to the third loss;

[0141] Here, the first weight value, the second weight value, and the third weight value can be specific numerical values ​​set artificially, for example, 0.3, 0.4, 0.5, etc. For each loss (first loss, second loss, third loss), there will be a corresponding weight value (first weight value, second weight value, third weight value). These weight values ​​are used to adjust the contribution of each loss to the total loss. The larger the weight value, the more important the loss is in the training process.

[0142] Step S172, respectively multiplying the first loss by the first weight value, the second loss by the second weight value, and the third loss by the third weight value to obtain a first target loss, a second target loss, and a third target loss;

[0143] In some implementations, each loss is multiplied by its corresponding weight value to obtain weighted losses, namely, the first target loss, the second target loss, and the third target loss.

[0144] Step S173, adding the first target loss, the second target loss and the third target loss to obtain a target loss;

[0145] In some embodiments, the target loss is determined by the following formula (6):

[0146] L(X,Y,Z) = δ1L gan (X,Z)+δ2L max (X,Y,Z)+δ3L compare (X,Y,Z) (6);

[0147] Among them, L(X,Y,Z) represents the target loss; L gan (X, Z) represents the first loss; δ1 represents the first weight value; δ2 represents the second weight value; δ3 represents the third weight value.

[0148] Step S174, training the rainy day image generation model using the target loss until the rainy day image generation model converges.

[0149] In some embodiments, the target loss is used to train the rainy day image generation model. The purpose of the training is to make the rainy day images generated by the rainy day image generation model as close as possible to the real rainy day images. The training process will continue until the rainy day image generation model converges. Convergence usually means that the target loss no longer decreases significantly, or a preset number of training rounds or iterations is reached.

[0150] In the embodiment of the present application, on the one hand, by introducing multiple losses (first loss, second loss, third loss), the performance of the rainy day image generation model in different aspects can be comprehensively evaluated, thereby ensuring that the generated rainy day images are close to the real rainy day images in multiple dimensions; on the other hand, by setting a corresponding weight value for each loss, the user can flexibly adjust the contribution of different losses to the total loss according to specific needs, thereby obtaining better training effects.

[0151] Image visual perception technology is widely used in the field of intelligent driving of automobiles. At present, it mainly uses on-board cameras to collect images of the driving environment of the car, and further perceives driving-related targets in the image through the algorithm network. The current visual perception models for rainy scenes are mostly data-driven. The diversity of rainy scene data trained with original annotations determines the generalization performance of the model in rainy scenes. At the same time, the detection elements of the model related to intelligent driving scenes are mostly distributed on the driving road. In rainy weather, on the one hand, the lens is blocked by raindrops, which affects the imaging quality of the image, such as distortion and blurred objects; on the other hand, during heavy rain, the details of the road scene and the image texture are further weakened, which will increase the difficulty of model detection in rainy scenes. Therefore, it is particularly important to optimize the model detection capability of complex rainy days in intelligent driving.

[0152] In the related art, for different rainy weather scenes, one way to optimize on the data side is to collect and label the original data of rainy weather scenes of different degrees (normal, light rain, heavy rain) on the training data side to meet the needs of rich rainy scene. The disadvantage of this method is that it requires a lot of manpower resources to collect and label the weather scenes. At the same time, weather scenes with different rainfall levels are sporadic, which will increase the difficulty of data collection and make it difficult to accumulate a large amount of data. In addition, a commonly used optimization method is to perform data enhancement on the model training side, such as data expansion based on a small amount of real complex rainy weather data sets to balance the scene distribution and data volume distribution of different rainy scenes. The disadvantage of this method is that it focuses more on expanding the amount of training data for different rainy weather scenes, and does not solve the imbalance problem of different rainfall levels by increasing the diversity of different rainy weather scenes.

[0153] With the development of generative adversarial neural network algorithms, their applications in data image conversion and image style transfer have become increasingly widespread. Therefore, based on the development of generative image technology, the generation of scene data for different weather conditions has extended more possibilities. For example, converting a regular weather scene into another complex rainy scene can solve the problem of large accumulation of data and category imbalance in different rainy weather scenes with lower labor costs and more diverse scene distribution. Although the existing generative adversarial network technology has made some progress, there is a lack of targeted data synthesis for intelligent driving rainy road scenes. The current image generative adversarial networks are mostly used in image style transfer, super resolution, image restoration and other scenarios. The general difficulty of this technology is that it is difficult to obtain a large amount of original paired data for network training. For example, paired images of the same road scene at the same location on different sunny / rainy days are difficult to obtain in intelligent driving road scenes.

[0154] A method for training an image generation model in the related art uses the original image to be generated as the first and second images of the original input, and through the network training of the secondary generator and the discriminator, the first image generated by the first image generator has the same style as the first sample image, and the second image generated by the second image generator has the same style as the second sample image. This method can solve the problem of paired images and ensure a high generation efficiency of paired images, but the method can only be used as a single element scene such as a portrait for good pairing generation. The intelligent driving road rainy day scene is complex and has great limitations.

[0155] A method for constructing a power image generation model in the related art adopts an image adversarial generation network, replaces the original generator structure in the Star Generative Adversarial Network (StarGAN) model with an improved generator, designs a color consistency loss function, and improves the power image generation model; at the original training set data level, a real outdoor scene image data set under various weather conditions is used as a training data set, and by calculating the adversarial loss, domain classification loss, reconstruction loss, and color consistency loss and continuously reducing the loss, the improved generator and discriminator in the improved power image generation model are optimized until the loss function converges, and a trained power image generation model is obtained. The advantages of this method: no paired original training data of different weather conditions are required; mutual conversion of multiple weather fields can be realized; and image elements related to power equipment are added to generate targeted data of different weather conditions, which solves the problem of imbalanced distribution of weather categories in power equipment image data sets. However, when this method is applied to intelligent driving road scenes, although there are many reference points, such as unpaired image training, color consistency loss, and multi-weather scene conversion, it also has some shortcomings. Its image generation model is divided into a two-stage model. In the second stage, the rainy day physical model and two-dimensional noise methods are used to enhance the data generation effect. This physical process involves too many adjustments to the image quality parameters, relies heavily on a priori, and is not fully achieved through algorithm learning. At the same time, compared with the power weather scene image, the intelligent driving road scene image has greater motion blur, artifacts and distortion, and the scene reusability is poor.

[0156] The embodiment of the present application proposes a method for constructing a rainy day image generation model and an image generation method, which are implemented under Ubuntu 22.04 and Pytorch deep learning framework. The computer configuration uses an Intel (R) Core (TM) i5-13600K processor with a main frequency of 4.0 GHz and a memory of 128 gigabytes (GB). The graphics card is an NVIDIA RTX 4090 with a video memory of 24GB. The above method constructs a rainy day road scene image generation model based on the discriminator improved by the maximum probability constraint loss function and the segmentation semantic contrast learning strategy; by training the model, high-quality rainy day road scene images can be generated using intelligent driving road scene images in normal weather, which is used to improve the generalization ability of the intelligent driving visual perception model for rainy day scenes.

[0157] The present application embodiment proposes a rainy day image generation model construction method and an image generation method, such as Figure 2 As shown, the following steps are included:

[0158] Step S201, prepare data;

[0159] In the present application, the data used include 4 types, such as Figure 3 As shown, they are respectively a real regular image 301, a real rainy day image 302, a real regular segmented image 303, and a real rainy day segmented image 304. Among them, the real regular image is a real clear image data totaling 10w frames collected by the vehicle camera, and the image data does not contain rainy day scenes and scenes with large noise such as night. The real rainy day image is a real rainy day image data of different raining degrees collected by the vehicle camera, totaling 10w frames. The image data set contains three scene data of light rain, moderate rain, and heavy rain, which are evenly distributed. In the specific heavy rain scene data image, there are obvious raindrops blocking the lens, road surface water and road wetting. The real regular segmented image is the segmented image corresponding to the real regular image, and the real rainy day segmented image is the segmented image corresponding to the real rainy day image. The two segmented images can be generated by segmenting the image through the open source segmentation model, which can mainly segment common driving scene elements such as the road surface and the sky.

[0160] Step S202, constructing a rainy day image generation model;

[0161] The present application embodiment provides a rainy day image generation model, such as Figure 4 As shown, it includes a data preprocessing module ( Figure 4), generator module 401, discriminator module 402 and segmentation contrast learning module 403. The data preprocessing module converts the training data into the original input of each module by reading the training data and performing conventional data operations; the real regular image 301 outputs the rainy day generated image 411 through the generator module 401, and the network parameters of the generator module are updated by the image generation loss between the real regular image and the rainy day generated image; the realism of the generated image is evaluated by the discriminator module 402, which is used to output the maximum probability constraint loss based on the rainy day generated image 411, the real regular image 301 and the real rainy day image 302, and the network parameters of the discriminator module are updated by the maximum probability constraint loss; the segmentation contrast learning module 403 (essentially an encoder) learns the local features of the image through a multi-layer perceptron, which is used to output the segmentation contrast loss based on the rainy day generated image 411, the real regular image 301, the real regular segmented image 303 and the real rainy day segmented image 304, and the network parameters of the segmentation contrast learning module are updated by the segmentation contrast loss. The above loss can ensure that the rainy day image generated by the rainy day image generation model has less distortion, and can not only focus on the overall appearance of the image, but also pay attention to the local features and semantic information inside the image, so that the generated rainy day image is more realistic and credible. Finally, the target loss after comprehensive calculation of the image generation loss, the maximum probability constraint loss and the segmentation contrast loss guides the training of the rainy day image generation model, and the rainy day image generation model is obtained after the training is completed.

[0162] The data preprocessing module prepares four types of original vehicle-side training data with a resolution of 3840x2160. Since the resolution of the original image is large, the number of network operation parameters on the resolution of the original image is large. Therefore, the original image is first scaled to reduce the image to a resolution of 960x512. Secondly, the RGB channels of the reduced image are converted to YUV. Converting the image to the YUV color space can help the model better separate the brightness and chromaticity information. In the task of processing rainy roads, the original image noise usually affects the brightness information, but has little effect on the chromaticity information (U and V), which can improve the robustness of the rainy image generation model. Finally, since the rainy image generation model training adopts the pytorch structure, the four types of image data need to be converted into tensor format as the final input format of the rainy image generation model.

[0163] The present application embodiment provides a generator module, such as Figure 5As shown, the overall generator network structure is a U-Net network structure, which is composed of four layers of downsampling modules and four layers of upsampling modules of 2, 4, 8, and 16 times. The basic module includes an input downsampling module 501, a conventional downsampling module 502, a conventional upsampling module 503, and an output upsampling module 504. The input downsampling module 501 is composed of a nonlinear activation function layer (Leaky Relu) 5011 and a conventional 4×4 basic convolution layer 5012; the conventional downsampling module 502 is composed of a nonlinear activation function layer 5011, a 4×4 basic convolution layer 5012 and a normalization layer (norm layer) 5021; the conventional upsampling module 503 is composed of an activation function layer (Relu) 5031, a 2d basic deconvolution layer 5032 and a normalization layer 5021; the output upsampling module 504 is composed of a Tanh activation layer 5041, a 2d basic deconvolution layer 5032 and a normalization layer 5021. The real conventional image (960x512x3) tensor format after processing by the data preprocessing module. First, the input downsampling module is performed at 2 times, and then the output of the conventional downsampling module at 4, 8, and 16 times downsampling is connected in sequence. At the same time, each downsampling outputs the corresponding downsampling features separately; in the upsampling stage, it is first connected in sequence through the conventional upsampling module at 16, 8, and 4 times upsampling, and then connected to the output upsampling module at 2 times. At the same time, the output features of the same layer of downsampling are connected in sequence at 8, 4, and 2 times upsampling for feature splicing, and finally a 960x512x3 rainy day image is output. In the basic module design, LeakyRelu is used in the downsampling module, and Relu is used in the upsampling module. This choice helps to generate higher quality images while maintaining gradient transfer. In the output layer upsampling module, a Tahn activation function is connected to standardize the output results of the generator network to the range of [-1, 1]. It has good symmetry and can better handle the negative values ​​of the image data itself. At the same time, it ensures that the generated image and the real image are consistent when input to the discriminator, which helps to improve the performance of the discriminator.

[0164] The present application embodiment provides a discriminator module, such as Figure 6As shown, the discriminator module includes a basic module 601 and an output module 602. The structure of the discriminator module is the common NLayerDiscriminator discriminator architecture. The core is to gradually extract image features for each image through multiple convolutional layers, and finally output a 960x512x1 feature map. These feature maps are used to calculate the loss and help the generator generate more realistic rainy day images. The basic module 601 is composed of a 4×4 basic convolution layer 6011, a normalization layer 6012, and a nonlinear activation function layer 6013. The output module 602 (similar to full connection) is a 4×4 basic convolution layer 6011, which is used for dimensionality reduction processing and full connection, and outputs the features after dimensionality reduction. First, the rainy day generated image output by the generator module with an input resolution of 916x512x3 is downsampled through 3 layers of basic modules to obtain 120x56x128 downsampled features. Then, it is upscaled through a basic module to obtain 916x512x128 upscaled features. Finally, it is output through an output module with a feature map of 916x512x1, where 1 represents the probability of the rainy day generated image being close to the real regular image. The discriminator network here uses the idea of ​​local pixel judgment, and further uses the pixel probability features (1 in 916x512x1) of the feature map of the HxW class of the global range class to calculate the maximum probability constraint loss. In the loss module, the image generation loss uses the mean squared error (MSE) loss L gan (X, Z) (i.e., the loss between the first rainy day image and the real regular image); at the same time, combined with the expected effect of generating rainy day images, a maximum probability constraint loss based on real regular images, rainy day generated images, and real rainy day images is designed. Because the road rainy day images are ambiguous when generated, and the Generative Adversarial Network (GAN) is unstable during the training process, the generated road rainy day images will be affected by artifacts and distortion. Therefore, in order to optimize this item, X is a real regular image, Y is a real rainy day image, and Z is a rainy day generated image. The maximum range probability loss is based on the output representation of the discriminator to ensure that the generated road rainy day images follow a distribution similar to that of clear images and real rainy images. At the same time, it is agreed that the maximum rainy day effect is distributed within the range of real road rainy days to avoid the generated road rainy day images deviating too far from the clear image and rainy day image.

[0165] The present application embodiment provides a maximum probability constraint loss, such as Figure 7As shown in the figure, assume that the real rainy day image is D(Y), the real regular image is D(X), and the rainy day generated image is D(Z). With D(X) as the center of the circle and the distance from D(X) to D(Y) as the radius, the range from the real regular image to the rainy day generated image is obtained. D(Z) may be too far away from D(X) and D(Y), resulting in artifacts and distortion that do not belong to or appear in the generated image. Then, the circumscribed square of the circle is calculated based on this range, and the circumscribed circle of this square is used as the radius of the maximum generated road rainfall range D(Z). max , The maximum probability constraint loss defined based on the distance between D(Z) to D(X) and D(Z) to D(Y) is shown in the following formula (1):

[0166]

[0167] The constraint condition of D(Z) is as shown in the following formula (7):

[0168]

[0169] The reason for adding the segmentation contrast learning module is that in the process of generating rainy days on roads, the background of the ideal rainy day image on roads is a real regular image, while the rainfall and feature distribution are real rainy day images. The image contrast between the generated rainy day image and the real regular image and the semantic segmentation contrast information between the real regular segmented image and the real rainy day segmented image are added to guide the image to be generated in a direction closer to the real image and reduce image distortion. At the same time, considering that the application scenario of the generated rainy day image is the intelligent driving road scene, the road scene has obvious features and high repetition in the image, and the real effect of the rainfall on the road can be enhanced by segmentation. The designed segmentation contrast learning is divided into two groups, one is the segmentation contrast learning mIoU(X, Y) of the real regular segmented image and the real rainy day segmented image, and the contrast learning Sim(X, Z) of the real regular image and the generated rainy day image, so that the input data is not paired rainy day image contrast learning.

[0170] The present application embodiment provides a segmentation contrast learning module, such as Figure 8As shown, the segmentation contrast learning module consists of a feature flattening operation module (flatten) 801, a patch basic module 802 and a multilayer perceptron (MLP) module 803. The multilayer perceptron module 803 includes two linear layers (linear) 8031 ​​and an activation layer (relu) 8032. The patch basic module 802 includes a random crop feature layer (rand crop feature) 8021. The multilayer perceptron network module is often used in classification contrast learning. Its essence is to approximate the function of complex tasks through nonlinear transformation through a multi-layer fully connected network. It is very suitable for application in the image contrast learning of the road rainy day. The patch module is mainly used to realize local sampling of the output features. Since the semantic attributes of the image cannot be fully aligned globally, operating the image in a global range may cause artifacts in the generated image; another reason is that the semantic distribution position of the road and its rainy day on the road in the road rainy day scene image is relatively fixed. In the contrast learning process, simply operating at the global image level, the learned features are relatively simple, which will lose the diversity of the generated road rainy day image. First, a full connection (convolution) is performed on the real regular image, the real rainy day image, the real regular segmentation image, and the real rainy day segmentation image for feature extraction, and then the output rainy day generated image (960x520) is input into the segmentation contrast learning module. Then, the feature flattening operation module is used to obtain the output feature of 491520x3, and then the local features are randomly sampled. The total number of samples is 128 (equivalent to cutting a picture into 128 small pictures), and the minimum local positive feature size is 3840x3 (the dimension of each small picture feature is 3840x3). The sampling method is to randomly generate a sampling id (label) in the range of 0-128 (randomly take a small picture to calculate the feature each time), and then index the corresponding Patch range feature in the global feature under the corresponding id and input it into the MLP module for learning to obtain the corresponding output feature. After sampling 128 times, the output features under all sampling ids and the corresponding sampling ids are saved. In the loss of segmentation contrastive learning, based on the pairwise contrastive learning of the two groups of images mentioned above, for the loss of segmentation contrastive learning of the real conventional segmented images and the real rainy day segmented images, mIoU is used to evaluate the similarity between the two segmented images. The interpretability lies in that the two are unpaired rainy day images. The semantic information of the images in the rainy day scene is similar. The common mIoU loss in the segmentation field can be used to evaluate the similarity of the two images, which is specifically expressed as mIoU(X,Y)=X i ·Y j ; For the comparative learning loss of real regular images and rainy day generated images, SSIM image similarity is used. The interpretability lies in that the two are homologous paired images with completely aligned pixel positions, which is specifically expressed as Sim(X,Z)=Xi ·Z j (i, j range is 0-128, calculated 128 times) Based on this definition, the target similarity scores between the real regular image, the rainy day generated image, the real regular segmented image and the real rainy day segmented image are shown in the following formula (2):

[0171]

[0172] The overall segmentation contrast learning loss is a common loss for reference image-to-image translation tasks. PatchNCE (X, Z), and based on it, we designed a weighted guidance method to compare the learning results, guiding the generated image to be close to the real rainy day in terms of semantic features and to retain the original clear image in terms of image authenticity. At the same time, considering that the goal is to generate high-quality images in local areas of the image rather than globally on the entire image, L PatchNCE The (X,Z) loss focuses on the common parts between the two domains, such as shape, and ignores the differences between the two domains, such as texture. PatchNCE The (X, Z) loss is shown in the following formula (3):

[0173]

[0174] Taking the sum of the negative numbers of all patches in the entire image (4 images) can encourage the discriminator to assign patches of the same image to high probabilities as much as possible, and assign patches of different images to low probabilities (if the patches of two images are highly similar, it proves that the closer the generated image is to the patch in the original image, the higher the probability). In this design, when all negative samples are treated equally, easy negative samples (those that can be easily distinguished, if a rainy day picture is generated from the original image, then the negative sample means that the generated image output by the generator module is not a rainy day picture) may dominate, thereby reducing the contribution of more challenging negative samples to model training, which may cause the model to perform poorly when dealing with more complex scenes because the model has not fully learned how to distinguish those difficult-to-distinguish negative samples. Based on this, we make a weighted improvement on this loss and define a segmentation contrast learning loss as shown in the following formula (5):

[0175]

[0176] The segmentation contrast learning loss not only considers the similarity between the local feature and itself, but also the similarity with other local features. At the same time, this difference helps the model better learn negative samples of different difficulty levels, weight negative samples and limit the number of negative samples to improve the performance of the model when processing complex data. The purpose of this design is explained in conjunction with the reverse similarity F(i, j):

[0177] When the difference between the real regular image and the real rainy day image is large (i.e., mIoU(X, Y) is small), the generated rainy day image has a high similarity with the real regular image (i.e., Sim(X, Z) is large), and the F(i, j) score is large, which is recorded as scene 1; the numerator is particularly large and the denominator is particularly small;

[0178] When the difference between the real regular image and the real rainy day image is small, the generated rainy day image has a high similarity with the real regular image, and F(i, j) is close to 1, which is recorded as scene 2;

[0179] When the real regular image is very different from the real rainy day image, the generated rainy day image has a low similarity with the real regular image, and F(i, j) is close to 1, which is recorded as scene 3;

[0180] When the difference between the real regular image and the real rainy day image is small, the generated rainy day image has low similarity with the real regular image, and F(i, j) is large, which is recorded as scene 4.

[0181] After each iterative update, F(i, j) will change, and F(i, j) will be used to adjust w(i, j).

[0182] Scene 1 is explained as generating a better rainfall picture, F(i, j) is larger, w(i, j) weight coefficient is smaller, and loss function update is smaller; Scene 2 is explained as the difference between the real regular image and the real rainy day image is small, and the generated rainy day image has a good effect, F(i, j) is close to 1, because the real rainy day image is close to the generated rainy day image, and conventional loss update is sufficient; Scene 3 is explained as the difference between the real regular image and the generated rainy day image is large, and the generated rainy day image is not good, which is reasonable, and conventional loss update is sufficient; Scene 4 is explained as the real regular image is close to the rainy day, and there is no need to over-generate this pair of images, so F(i, j) is larger, w(i, j) weight coefficient is smaller, and loss function update is smaller. Comprehensively for other intermediate scenes, the loss function is designed to ensure that the generated rainy day image has both the authenticity of the original image and a good rainfall effect.

[0183] The image generation loss L is integrated with the road rain scene discriminator module gan (X,Z), maximum probability constraint loss L max (X,Y,Z), and segmentation contrastive learning loss L compare (X, Y, Z), the final training target loss is shown in the following formula (6):

[0184] L(X,Y,Z) = δ1L gan (X,Z)+δ2L max (X,Y,Z)+δ3L compare (X,Y,Z) (6);

[0185] Among them, δ1 and δ3 are both set to 1. Since the maximum probability constraint loss is an auxiliary loss, δ2 is set to 0.1. Combining the network design and loss design of the above three modules, the final rainy day image generation model for intelligent driving roads is constructed.

[0186] Step S203, training the rainy day image generation model using target loss;

[0187] Using the constructed rainy day image generation model of intelligent driving roads, four types of image training data are loaded to train the rainy day image generation model, and the network weight parameters of the generator module, discriminator module, and segmentation contrast learning module are obtained respectively. At the same time, the network is updated in combination with the target loss. After training the final set number of model training iterations, the final model of the three modules is obtained.

[0188] Step S204: Generate a rainy road scene image using a rainy day image generation model.

[0189] During the usage phase, only the generator module is needed. Given the input of the generator module as a real regular image, the generator module generates the corresponding rainy day image.

[0190] The embodiment of the present application proposes a method for generating rainy day images of intelligent driving road scenes. The method designs an improved discriminator based on the maximum probability constraint of real regular images, real rainy day images, and generated rainy day images. By constraining the maximum probability range of rainy day effects, the quality problem of the generated image caused by excessive rainfall is avoided; a segmentation semantic comparison method is designed, and the rainy day semantic information in the segmentation semantics of real regular images and real rainy day images is learned through the network, which optimizes the artifact influence of the generated rainy day image and improves the authenticity of the generated image; a rainy day image generation model based on unpaired original rainy day data training is designed, and the difficulty of training data collection in the image generation of unpaired real regular images and rainy day images is solved by the method of comparing and learning two groups of images. The generalization ability of the intelligent driving visual perception model for rainy day scenes is further improved by generating rainy day scene images.

[0191] In an embodiment of the present application, a maximum probability constrained generation method based on real regular images, real rainy day images, and rainy day generated images is designed to improve the discriminator, and image generation is better guided by the maximum probability range; a method for adding segmentation semantic comparison is designed, and the rainy day semantic information in the segmentation semantics of real regular images and real rainy day images is learned through a network to optimize the authenticity of the generated rainy day images; a rainy day image generation model based on unpaired original rainy day data training is designed, and the image generation problem of unpaired real regular images and rainy day images is solved by a method of comparing and learning two groups of images.

[0192] The present application embodiment proposes a training device for a rainy day image generation model, such as Fig. 9 As shown, the training device of the rainy day image generation model includes:

[0193] An acquisition module 910 is used to acquire a training data set; the training data set includes a real conventional image, a real rainy day image, a real conventional segmented image, and a real rainy day segmented image;

[0194] a discriminator module 920 for determining a first loss between the real regular image and a first rainy image; the first rainy image is generated using the real regular image;

[0195] a discriminator module 920 for determining a second loss between the real regular image, the real rainy day image and the first rainy day image;

[0196] a segmentation contrast learning module 930, configured to determine a third loss between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image;

[0197] The training module 940 is used to train the rainy day image generation model using the first loss, the second loss and the third loss until the rainy day image generation model converges.

[0198] In the embodiment of the present application, the acquisition module 910 includes: an acquisition unit, which is used to acquire an initial real conventional image and an initial real rainy day image; a segmentation unit, which is used to segment the initial real conventional image and the initial real rainy day image respectively, and correspondingly obtain an initial real conventional segmented image and an initial real rainy day segmented image; a processing unit, which is used to scale and convert the initial real conventional image, the initial real rainy day image, the initial real conventional segmented image and the initial real rainy day segmented image in sequence, and correspondingly obtain the real conventional image, the real rainy day image, the real conventional segmented image and the real rainy day segmented image; a first determination unit, which is used to determine the training data set based on the real conventional image, the real rainy day image, the real conventional segmented image and the real rainy day segmented image.

[0199] In an embodiment of the present application, the processing unit includes: a first processing subunit, used to scale the initial real conventional image to obtain a scaled real conventional image; a second processing subunit, used to perform color space conversion on the scaled real conventional image to obtain a first real conventional image; and a conversion subunit, used to convert the format of the first real conventional image into a first format to obtain the real conventional image.

[0200] In an embodiment of the present application, the first rainy day image is generated by a generator module using the real regular image, and the generator module includes four layers of upsampling modules with different multiples and four layers of downsampling modules with different multiples. The generator module is used to downsample the real regular image in different multiples in sequence through the four layers of downsampling modules with different multiples to obtain corresponding downsampling features; upsample the downsampling features output by the fourth downsampling module through the first upsampling module to obtain the first upsampling features; starting from the second upsampling module, the upsampling features output by the previous upsampling module are spliced ​​with the downsampling features output by the downsampling module with the corresponding multiples to obtain corresponding upsampling features; the upsampling features output by the third upsampling module are spliced ​​with the downsampling features output by the downsampling module with the corresponding multiples through the fourth upsampling module to obtain the first rainy day image.

[0201] In the embodiment of the present application, the discriminator module 920 includes: a first difference unit, which is used to subtract the pixel value of the first rainy day image from the pixel value of the real regular image for the same pixel point in the image to obtain a first pixel difference; a second difference unit, which is used to subtract the pixel value of the first rainy day image from the pixel value of the real rainy day image to obtain a second pixel difference; a first addition unit, which is used to add the value of the first pixel difference after taking the absolute value and the value of the second pixel difference after removing the absolute value to obtain the loss of the same pixel point; a division unit, which is used to sum the losses of all pixel points in the image and divide the sum by the area of ​​the image to obtain the second loss.

[0202] In the embodiment of the present application, the segmentation contrast learning module 930 includes: a second determination unit, used to determine the target similarity scores between the real regular image, the first rainy day image, the real regular segmented image and the real rainy day segmented image; a third determination unit, used to determine the image block loss between the image blocks in the real regular image and the image blocks in the first rainy day image; a fourth determination unit, used to determine the weight coefficient based on the target similarity score and a preset weighted scaling factor; a fifth determination unit, used to determine the third loss based on the target similarity score and the weight coefficient.

[0203] In an embodiment of the present application, the second determination unit includes: a first determination subunit, used to determine the first similarity between the real regular image and the first rainy day image; a second determination subunit, used to determine the second similarity between the real regular segmented image and the real rainy day segmented image; a first multiplication subunit, used to subtract 1 from the second similarity and then multiply the result by the first similarity to obtain a first similarity score; a second multiplication subunit, used to subtract 1 from the first similarity and then multiply the result by the second similarity to obtain a second similarity score; a division subunit, used to use the first similarity score to divide the sum of the second similarity score and a preset value to obtain a target similarity score.

[0204] In an embodiment of the present application, the training module 940 includes: an acquisition unit, used to respectively acquire a first weight value corresponding to the first loss, a second weight value corresponding to the second loss, and a third weight value corresponding to the third loss; a multiplication unit, used to respectively multiply the first loss by the first weight value, the second loss by the second weight value, and the third loss by the third weight value, to obtain a first target loss, a second target loss, and a third target loss respectively; a second addition unit, used to add the first target loss, the second target loss, and the third target loss to obtain a target loss; a training unit, used to train the rainy day image generation model using the target loss until the rainy day image generation model converges.

[0205] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0206] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0207] An embodiment of the present application also provides an electronic device, including a memory and a processor of a server or a memory and a processor of a user terminal, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0208] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0209] An embodiment of the present application also provides a computer program, including a computer-readable code. When the computer-readable code runs in a server or a user terminal, a processor in the server or the user terminal executes some or all of the steps for implementing the above method.

[0210] The present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0211] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0212] It should be noted that an embodiment of the present application provides an electronic device, such as Fig.10As shown, the hardware entity of the electronic device 1000 includes: a processor 1001, a communication interface 1002 and a memory 1003, wherein: the processor 1001 generally controls the overall operation of the electronic device 1000. The communication interface 1002 can enable the electronic device to communicate with other terminals or servers through a network. The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and each module in the electronic device 1000 (for example, image data, audio data, voice communication data and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be carried out between the processor 1001, the communication interface 1002 and the memory 1003 through the bus 1004.

[0213] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A training method for a rainy day image generation model, characterized in that: The training method comprises: Acquire a training data set; the training data set includes real regular images, real rainy day images, real regular segmented images and real rainy day segmented images; determining a first loss between the real regular image and a first rainy day image; the first rainy day image is generated using the real regular image; determining a second loss between the real regular image, the real rainy day image, and the first rainy day image; determining a third loss between the true regular image, the first rainy day image, the true regular segmented image, and the true rainy day segmented image; The rainy day image generation model is trained using the first loss, the second loss, and the third loss until the rainy day image generation model converges.

2. The training method according to claim 1, characterized in that: Get the training dataset, including: Collecting an initial real conventional image and an initial real rainy day image; The initial real conventional image and the initial real rainy day image are segmented respectively, and an initial real conventional segmented image and an initial real rainy day segmented image are obtained correspondingly; Respectively scaling and converting the initial real conventional image, the initial real rainy day image, the initial real conventional segmented image, and the initial real rainy day segmented image in sequence, to obtain the real conventional image, the real rainy day image, the real conventional segmented image, and the real rainy day segmented image respectively; The training data set is determined based on the real regular image, the real rainy day image, the real regular segmented image and the real rainy day segmented image.

3. The training method according to claim 2, characterized in that: The initial real conventional image is sequentially scaled and converted to obtain the real conventional image, including: Scaling the initial real conventional image to obtain a scaled real conventional image; Performing color space conversion processing on the scaled real regular image to obtain a first real regular image; The format of the first real regular image is converted into a first format to obtain the real regular image.

4. The training method according to claim 1, characterized in that: The first rainy day image is generated by a generator module using the real regular image, wherein the generator module includes four layers of upsampling modules with different multiples and four layers of downsampling modules with different multiples. Inputting the real regular image into the generator module and outputting the first rainy day image comprises: Downsampling the real conventional image in different multiples in sequence through the four layers of downsampling modules with different multiples, and obtaining corresponding downsampling features; Upsampling the downsampled features output by the fourth downsampled module through the first upsampled module to obtain a first upsampled feature; Starting from the second upsampling module, the upsampling features output by the previous upsampling module are concatenated with the downsampling features output by the downsampling module of the corresponding multiple to obtain the upsampling features; The first rainy day image is obtained by splicing the up-sampled features output by the third up-sampling module and the down-sampled features output by the down-sampling module of the corresponding multiple through the fourth up-sampling module.

5. The training method according to any one of claims 1 to 4, characterized in that: Determining a second loss between the real regular image, the real rainy day image, and the first rainy day image includes: For a same pixel point in the image, subtract the pixel value of the first rainy day image from the pixel value of the real regular image to obtain a first pixel difference value; Subtracting the pixel value of the first rainy day image from the pixel value of the real rainy day image to obtain a second pixel difference value; Adding the absolute value of the first pixel difference to the absolute value of the second pixel difference to obtain the loss of the same pixel; The second loss is obtained by summing the losses of all pixels in the image and dividing the sum by the area of ​​the image.

6. The training method according to any one of claims 1 to 4, characterized in that: Determining a third loss between the true regular image, the first rainy day image, the true regular segmented image, and the true rainy day segmented image comprises: determining target similarity scores between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image; determining an image block loss between an image block in the real regular image and an image block in the first rainy day image; Determining a weight coefficient based on the target similarity score and a preset weighted scaling factor; The third loss is determined based on the target similarity score and the weight coefficient.

7. The training method according to any one of claims 1 to 4, characterized in that: Determining target similarity scores between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image includes: determining a first similarity between the real regular image and the first rainy day image; Determining a second similarity between the real conventional segmented image and the real rainy day segmented image; Subtract 1 from the second similarity and then multiply the result by the first similarity to obtain a first similarity score; Subtract 1 from the first similarity and then multiply the result by the second similarity to obtain a second similarity score; The first similarity score is used to divide the sum of the second similarity score and a preset value to obtain a target similarity score.

8. The training method according to any one of claims 1 to 4, characterized in that: The rainy day image generation model is trained by using the first loss, the second loss, and the third loss until the rainy day image generation model converges, including: Respectively obtaining a first weight value corresponding to the first loss, a second weight value corresponding to the second loss, and a third weight value corresponding to the third loss; Respectively multiplying the first loss by the first weight value, the second loss by the second weight value, and the third loss by the third weight value to obtain a first target loss, a second target loss, and a third target loss; Adding the first target loss, the second target loss and the third target loss to obtain a target loss; The rainy day image generation model is trained using the target loss until the rainy day image generation model converges.

9. A training device for a rainy day image generation model, characterized in that: The training device comprises: An acquisition module is used to acquire a training data set; the training data set includes real conventional images, real rainy day images, real conventional segmented images and real rainy day segmented images; a discriminator module for determining a first loss between the real regular image and a first rainy image; the first rainy image is generated using the real regular image; a discriminator module for determining a second loss between the real regular image, the real rainy image, and the first rainy image; a segmentation contrast learning module, configured to determine a third loss between the real regular image, the first rainy day image, the real regular segmented image, and the real rainy day segmented image; A training module is used to train the rainy day image generation model by using the first loss, the second loss and the third loss until the rainy day image generation model converges.

10. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and the computer program or instructions implement the steps of the method according to any one of claims 1 to 8 when executed by a computer.

Images