An image processing method, apparatus, electronic device, and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and a storage medium, wherein the image processing method can be applied to the field of intelligent driving, and the method comprises the following steps: acquiring a ground splicing image around a target vehicle; performing semantic segmentation on the ground splicing image to obtain at least one piece of ground semantic information; inputting the ground splicing image and the at least one piece of ground semantic information into a generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation image corresponding to the ground splicing image; and performing semantic segmentation on the noise-free inverse perspective transformation image to obtain at least one piece of target semantic information. The application reduces the influence of ground reflection noise in a picture on a subsequent segmentation model, improves the accuracy of a segmentation model in predicting the pixel category in the picture, and improves the accuracy of an automatic driving perception system in interpreting the environment around a vehicle.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, autonomous driving perception systems rely on hardware such as lidar and cameras to acquire information about the vehicle's surroundings. This information typically requires further processing using pre-set models to allow the perception system to interpret the vehicle's environment and assist the driving system in responding to ensure safe driving. In underground parking garages, such as... Figure 1 As shown, reflections often occur due to lighting conditions and the surface material, appearing in images captured by the camera in a fleeting moment, such as... Figure 2 As shown, these ground reflections have the same pixel value as traffic signs on the ground due to exposure issues, which confuses the model. As a result, the model treats the ground reflections as traffic landmarks on the ground, outputs incorrect perception results, affects the subsequent planning and decision-making of vehicles, and affects the safe driving of cars.

[0003] Existing technologies either treat reflective areas as background or eliminate brighter parts based on the color distribution of the image itself. However, if the image itself has bright colors, it will also be considered a reflective area and removed, affecting the semantic information contained in the image and interfering with the image's clarity. Summary of the Invention

[0004] To address the problems of the prior art, this application provides an image processing method, apparatus, electronic device, and storage medium. The technical solution is as follows:

[0005] On the one hand, an image processing method is provided, the method comprising:

[0006] Obtain a mosaic image of the ground surrounding the target vehicle;

[0007] The ground mosaic image is semantically segmented to obtain at least one piece of ground semantic information;

[0008] The ground mosaic and the at least one piece of ground semantic information are input into the generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation map corresponding to the ground mosaic.

[0009] Semantic segmentation is performed on the noise-free inverse perspective transformation image to obtain at least one target semantic information.

[0010] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0011] The first acquisition module is used to acquire a mosaic image of the ground surrounding the target vehicle;

[0012] The first semantic segmentation module is used to perform semantic segmentation on the ground mosaic map to obtain at least one piece of ground semantic information.

[0013] An image generation module is used to input the ground mosaic image and the at least one piece of ground semantic information into the generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation image corresponding to the ground mosaic image.

[0014] The second semantic segmentation module is used to perform semantic segmentation on the noise-free inverse perspective transformation image to obtain at least one target semantic information.

[0015] In one exemplary embodiment, the image generation module includes:

[0016] The first weight allocation module is used to assign weights to the at least one piece of ground semantic information to obtain the weight corresponding to each piece of ground semantic information.

[0017] The first information synthesis module is used to obtain weighted ground semantic information corresponding to each ground semantic information based on the at least one ground semantic information and the weight corresponding to each ground semantic information;

[0018] The noise-free image generation module is used to input the ground mosaic image and the weighted ground semantic information into the generator of the pre-trained generative adversarial network to generate the noise-free inverse perspective transformation image.

[0019] In one exemplary embodiment, the apparatus further includes a model training module for training a generative adversarial network, the model training module comprising:

[0020] The second acquisition module is used to acquire the sample ground mosaic image and the noise-free ground mosaic image corresponding to the sample ground mosaic image;

[0021] The sample segmentation module is used to perform semantic segmentation on the sample ground mosaic map to obtain at least one piece of sample ground semantic information.

[0022] The prediction module is used to input the sample ground mosaic map and the at least one sample ground semantic information into the generator of the initial generative adversarial network to generate a noise-free prediction map.

[0023] The loss determination module is used to input the predicted noise-free map and the noise-free ground mosaic map into the discriminator of the initial generative adversarial network, and determine the model loss based on the discrimination result of the discriminator; the discrimination result indicates the difference between the predicted noise-free map and the noise-free ground mosaic map;

[0024] The parameter adjustment module is used to adjust the network parameters of the initial generative adversarial network based on the model loss until the preset training termination condition is met, thereby obtaining the pre-trained generative adversarial network.

[0025] In one exemplary implementation, the prediction module includes:

[0026] The second weight allocation module is used to allocate weights to the at least one sample ground semantic information to obtain the weights corresponding to each sample ground semantic information.

[0027] The second information synthesis module is used to obtain weighted sample ground semantic information corresponding to each sample ground semantic information based on the at least one sample ground semantic information and the weight corresponding to each sample ground semantic information.

[0028] The prediction image generation module is used to input the sample ground mosaic map and the weighted sample ground semantic information into the generator of the initial generative adversarial network to generate the prediction noise-free map.

[0029] In one exemplary embodiment, the sample segmentation module includes:

[0030] The first segmentation model module is used to input the sample ground mosaic image into a pre-trained inverse perspective transformation image segmentation model for semantic segmentation processing to obtain the at least one sample ground semantic information.

[0031] In one exemplary implementation, the first semantic segmentation module includes:

[0032] The second segmentation model module is used to input the ground mosaic image into the pre-trained inverse perspective transformation image segmentation model to generate the at least one piece of ground semantic information;

[0033] The second semantic segmentation module includes:

[0034] The third segmentation model module is used to input the noise-free inverse perspective transformation image into the pre-trained inverse perspective transformation image segmentation model to generate the at least one target semantic information.

[0035] In one exemplary embodiment, the apparatus further includes an image processing module for processing multiple fisheye images, the image processing module comprising:

[0036] The image acquisition module is used to acquire multiple fisheye images captured by multiple fisheye cameras around the target vehicle;

[0037] The parameter determination module is used to determine the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera.

[0038] The image transformation module is used to transform the fisheye image corresponding to each fisheye camera to the ground coordinate system based on the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera, so as to obtain multiple ground sub-images.

[0039] The image stitching module is used to stitch together the multiple ground sub-images to obtain the ground stitched image.

[0040] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image processing method of any of the above aspects.

[0041] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the image processing method as described above.

[0042] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the image processing method of any of the above aspects.

[0043] This application embodiment uses the output of a pre-trained inverse perspective transform image segmentation model to guide the generative adversarial network, removing ground reflection areas from the stitched image of a fisheye image. This reduces the impact of ground reflections on subsequent segmentation models, thereby improving the accuracy of the autonomous driving perception system in interpreting the vehicle's surrounding environment. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a scene of ground reflection in an underground parking garage provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a ground reflective area being identified as a lane line according to an embodiment of this application;

[0047] Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the main operation process and module design of a current autonomous driving perception system in a parking scenario, provided by an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the main operation process and module design of an improved autonomous driving perception system provided in this application embodiment in a parking scenario;

[0050] Figure 6 This is a schematic diagram of a pre-trained generative adversarial network generating images according to an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of a de-glare module in an improved autonomous driving perception system provided in an embodiment of this application;

[0052] Figure 8 This is a schematic flowchart of a fisheye image processing method provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of a training process for a generative adversarial network provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of a training generative adversarial network provided in an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of a process for training a generative adversarial network to generate images, provided in an embodiment of this application.

[0056] Figure 12 This is a structural block diagram of an image processing apparatus provided in an embodiment of this application;

[0057] Figure 13 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0060] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0061] Please see Figure 3 The diagram illustrates a flowchart of an image processing method provided in an embodiment of this application. It should be noted that while this specification provides the operational steps of the method as described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive methods. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual systems or products, the methods can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or accompanying drawings. Specifically, as... Figure 3 As shown, the method may include:

[0062] S301, Obtain a mosaic image of the ground surrounding the target vehicle.

[0063] The ground mosaic image is obtained by stitching together multiple fisheye images acquired by multiple fisheye cameras around the target vehicle. In practice, the image processing method of this embodiment can be applied to any other image and is not limited to a mosaic image of fisheye images.

[0064] The target vehicle is this vehicle, which is equipped with an autonomous driving perception system. The target vehicle is equipped with multiple fisheye cameras around its body, typically four fisheye cameras.

[0065] Among them, stitching multiple fisheye images requires distortion removal from multiple fisheye images before stitching them together. Specifically, by pre-calculating the intrinsic and extrinsic parameters of the fisheye camera, the images from the fisheye camera are projected onto a ground coordinate system, and then multiple fisheye images are stitched together to obtain a ground stitched image.

[0066] S303, perform semantic segmentation on the ground mosaic map to obtain at least one piece of ground semantic information.

[0067] Among them, ground semantic information consists of segmented images of different categories (such as vehicles, lane lines, traffic surfaces, etc.). The category of ground semantic information corresponds to the number of ground semantic information items. Generally, the number of semantic categories in the ground mosaic image will generate the same number of ground semantic information items after segmentation.

[0068] S305, the ground mosaic map and the at least one piece of ground semantic information are input into the generator of the pre-trained generative adversarial network to generate a noise-free inverse perspective transformation map corresponding to the ground mosaic map.

[0069] In this invention, the generative adversarial network (GAN) is a conditional GAN. The pre-trained GAN is a generator network trained on the GAN based on sample ground mosaics and corresponding noise-free ground mosaics. This trained GAN takes the mosaic information of the ground mosaic and ground semantic information as input and can generate noise-free inverse perspective transformation maps conditionally based on the input ground semantic information. The output of the pre-trained GAN is used as the input to the subsequent semantic segmentation model. Compared to the main operation process and module design of current autonomous driving perception systems in parking scenarios, such as... Figure 4 As shown, this embodiment adds a reflection removal module before semantic segmentation, such as... Figure 5 As shown, this can improve the image input quality of the segmentation model, enhance the quality of the input image, thereby reducing the impact of ground reflection in the image on the subsequent segmentation model, improving the accuracy of semantic segmentation of images in underground parking garage scenarios, and thus improving the accuracy of the autonomous driving perception system in interpreting the vehicle's surrounding environment.

[0070] Among them, the noise-free inverse perspective transformation image is an inverse perspective transformation image generated by the generator of the generative adversarial network under the guidance of ground semantic information, after removing reflective noise from the ground mosaic image.

[0071] The training of generative adversarial networks will be described in detail in a later section of this invention.

[0072] In one exemplary implementation, such as Figure 6 As shown, step S305 above may include:

[0073] S601, assign weights to the at least one piece of ground semantic information to obtain the weights corresponding to each piece of ground semantic information.

[0074] The purpose of assigning weights to semantic information is to inform the generative adversarial network (GAN) which part of the semantic information to prioritize when processing an image. For example, if the semantic information of the wall is more important than that of the ground, then the ground semantic information should be given a higher weight when assigning weights. As a result, the GAN will focus on processing the ground portion of the image. In practice, the weight assignment needs to be pre-defined based on the importance of the semantic information.

[0075] S603, based on the at least one piece of ground semantic information and the weight corresponding to each piece of ground semantic information, obtain the weighted ground semantic information corresponding to each piece of ground semantic information.

[0076] The weighted ground semantic information is obtained by integrating each piece of ground semantic information and its corresponding weight, which is convenient for subsequent use as input to the generative adversarial network. In specific implementation, the ground semantic information and its weights can be integrated by weighted summation to determine the weighted ground semantic information.

[0077] S605, the ground mosaic map and the weighted ground semantic information are input into the generator of the pre-trained generative adversarial network to generate the noiseless inverse perspective transformation map.

[0078] Among them, the pre-trained generative adversarial network is a generative network trained on the sample ground mosaic map. The trained generative adversarial network can generate a noise-free inverse perspective transformation map based on the input ground mosaic map and weighted ground semantic information.

[0079] The noise-free inverse perspective transformation image is an inverse perspective transformation image generated by the generator of a generative adversarial network (GAN) after removing reflective noise from the ground mosaic image, guided by weighted ground semantic information. Specifically, the ground mosaic image and the weighted ground semantic information are concatenated and input into a pre-trained GAN. The weighted ground semantic information serves as a conditional input to the GAN, which then performs image processing under this condition.

[0080] The training of generative adversarial networks will be described in detail in a later section of this invention.

[0081] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application improve the quality of image processing by assigning weights to ground semantic information, informing the generative adversarial network which part of semantic information needs to be given more attention when processing images.

[0082] S307, perform semantic segmentation on the noise-free inverse perspective transformation image to obtain at least one target semantic information.

[0083] Among them, the noise-free inverse perspective transformation image is a ground mosaic image with reflective noise that is generated by a pre-trained generative adversarial network. The noise-free inverse perspective transformation image is used as the input of the semantic segmentation model to improve the quality of the input image and improve the accuracy of the semantic segmentation model in the semantic segmentation of images in ground reflective scenes.

[0084] The target semantic information refers to the more accurate semantic segmentation result that this embodiment of the invention aims to obtain based on a ground mosaic image with reflective noise.

[0085] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application use the output results of the pre-trained inverse perspective transformation image segmentation model to guide the generative adversarial network, remove the ground reflection area on the stitched image of the fisheye image, thereby reducing the impact of ground reflection in the image on the subsequent segmentation model, thereby improving the accuracy of the autonomous driving perception system in interpreting the vehicle's surrounding environment.

[0086] In an exemplary embodiment, step S303 above, when performing semantic segmentation on the ground mosaic map, may include the following steps:

[0087] The ground mosaic image is input into the pre-trained inverse perspective transformation image segmentation model to generate at least one piece of ground semantic information;

[0088] Step S307 above, when performing semantic segmentation on the noise-free inverse perspective transform image, may include the following steps:

[0089] The noise-free inverse perspective transformation image is input into the pre-trained inverse perspective transformation image segmentation model to generate the at least one target semantic information.

[0090] Among them, the pre-trained inverse perspective transformation image segmentation model is the semantic segmentation model selected in this embodiment for semantic segmentation of ground mosaic images and noise-free inverse perspective transformation images. For example... Figure 7 As shown, a pre-trained inverse perspective transformation image segmentation model is used to perform semantic segmentation on a ground mosaic image. The segmentation result is used as a conditional input to a generative adversarial network (GAN), guiding the GAN to process the ground mosaic image and minimize reflection noise in the input ground mosaic image. The noise-free inverse perspective transformation image generated by the pre-trained GAN is then used as input to the pre-trained inverse perspective transformation image segmentation model, reducing the impact of ground reflections on the segmentation model.

[0091] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application perform semantic segmentation twice through a pre-trained inverse perspective transformation image segmentation model, perform semantic segmentation on the ground mosaic image, use the segmentation results as guidance for generative adversarial network image processing, eliminate ground reflection to the greatest extent, and use the generated noise-free inverse perspective transformation image as input to the pre-trained inverse perspective transformation image segmentation model, improve the image input quality of the segmentation model, reduce the impact of ground reflection in the image on the segmentation model, and make the perception results of the autonomous driving perception system more accurate.

[0092] The following explains the processing of fisheye images before obtaining the mosaic image of the ground around the target vehicle. Figure 8 As shown, it can specifically include:

[0093] S801, acquire multiple fisheye images captured by multiple fisheye cameras around the target vehicle.

[0094] The target vehicle is the vehicle itself equipped with an autonomous driving perception system. There are usually four fisheye cameras around the vehicle, and the fisheye images captured can cover the scene around the vehicle, thus serving as the perception source for the vehicle's autonomous driving perception system.

[0095] S803 determines the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera.

[0096] Among them, the intrinsic and extrinsic parameters of the vehicle's surrounding fisheye camera are pre-calculated and used to convert the fisheye images captured by the fisheye camera to the ground coordinate system, which facilitates the analysis and prediction of the scene around the vehicle by the relevant models in the autonomous driving perception system.

[0097] S805 transforms the fisheye image corresponding to each fisheye camera into the ground coordinate system based on the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera, resulting in multiple ground sub-maps.

[0098] Among them, the ground coordinate system is the coordinate system of the ground plane. In specific implementation, the fisheye image needs to be distorted and then the coordinates of the fisheye image need to be transformed according to the intrinsic and extrinsic parameters of the fisheye camera.

[0099] Among them, the ground sub-image is an image transformed from a fisheye image to a ground coordinate system, which is a distortion-free image of the scene around the vehicle.

[0100] S807, the multiple ground sub-maps are stitched together to obtain the ground stitched map.

[0101] The ground mosaic map is composed of multiple ground sub-maps and is a complete, distortion-free image of the vehicle's surroundings. The vehicle's autonomous driving perception system uses this image to analyze and predict the scene around the vehicle.

[0102] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application obtain a complete and undistorted scene map of the vehicle's surroundings by removing distortion and stitching fisheye images captured by fisheye cameras around the vehicle. This map serves as the object of processing by relevant models in the vehicle's autonomous driving perception system, thereby enabling the analysis and prediction of the scene around the vehicle.

[0103] The following describes the process of training a generative adversarial network based on sample ground mosaic images, such as... Figure 9 As shown, the specific steps may include:

[0104] S901, Obtain the sample ground mosaic map and the noise-free ground mosaic map corresponding to the sample ground mosaic map.

[0105] The sample ground mosaic serves as the input to the generator of the generative adversarial network (GAN) and is the training data for the GAN.

[0106] Among them, the noise-free ground mosaic is obtained by stitching together noise-free fisheye images taken by a fisheye camera. It serves as the input to the discriminator of the generative adversarial network and is the reference ground truth value for judging the authenticity of the data generated by the generator.

[0107] S903, perform semantic segmentation on the sample ground mosaic map to obtain at least one sample ground semantic information.

[0108] Specifically, the sample ground mosaic image is input into the prior segmentation model. The segmentation model performs semantic segmentation on the input image, distinguishing between lane background and traffic lines. The result of semantic segmentation is used as the input to the generator of the generative adversarial network, which further enhances the learning of noise-free features of the background.

[0109] In an exemplary embodiment, step S903 above, when performing semantic segmentation on the sample ground mosaic map, may include the following steps:

[0110] The sample ground mosaic image is input into a pre-trained inverse perspective transformation image segmentation model for semantic segmentation processing to obtain at least one sample ground semantic information.

[0111] The pre-trained inverse perspective transformation image segmentation model is the semantic segmentation model selected in this embodiment for semantic segmentation of the sample ground mosaic image. The output of the pre-trained inverse perspective transformation image segmentation model is relatively accurate in determining pixel categories in areas other than ground traffic landmarks; therefore, it can be used as a condition for the generative adversarial network in the sample ground mosaic. Figure 1 In the same input generative adversarial network generator, the learning speed of generative adversarial network for noiseless road surface background features is accelerated.

[0112] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application perform semantic segmentation on the sample ground mosaic image through a pre-trained inverse perspective transformation image segmentation model, distinguishing the lane background and traffic lines, and using the segmentation result as the input of the generator of the generative adversarial network, thereby further enhancing the learning of noise-free features of the background part by the generative adversarial network.

[0113] S905, the sample ground mosaic map and the at least one sample ground semantic information are input into the generator of the initial generative adversarial network to generate a noise-free prediction map.

[0114] Among them, the semantic information of the sample ground is used as a guiding condition and is concatenated with the sample ground. Figure 1 Both serve as inputs to the generator of a generative adversarial network, such as Figure 10 As shown, the generator of the generative adversarial network performs image processing guided by the semantic information of the sample ground surface, removing reflection noise to obtain a noise-free predicted image. Specifically, the semantic segmentation operation involves inputting the stitched sample ground surface image into the backbone model of MobileNet V2 (a lightweight neural network) for downsampling and feature extraction. Each layer uses convolutional and residual structures for higher-dimensional feature extraction, ultimately obtaining features with a resolution of 1 / 16 of the original image size. These features are then input into an ASPP (Atrous Spatial Pyramid Pooling) structure for convolutional operations on the spatial relationships of the features at a high-dimensional scale. The final output is then upsampled by 4 times and combined with the 1 / 4-fold features generated during the previous backbone downsampling process, followed by another 4-fold upsampling to obtain a feature map with the same size as the original image. Finally, the segmentation result is obtained through a Softmax (normalization) layer. The generator processes the image through multiple convolutional operations to obtain the final output.

[0115] In one exemplary implementation, such as Figure 11 As shown, step S905 above may include:

[0116] S1101, assign weights to the at least one sample ground semantic information to obtain the weights corresponding to each sample ground semantic information.

[0117] The purpose of assigning weights to semantic information is to inform the generative adversarial network (GAN) which part of the semantic information should be prioritized when processing an image. For example, if the semantic information of the wall is more important than that of the ground, then the ground semantic information should be given a higher weight when assigning weights. As a result, the GAN will focus on processing the ground portion of the image. In practice, the weights need to be pre-defined based on the importance of the semantic information.

[0118] S1103, based on the at least one sample ground semantic information and the weight corresponding to each sample ground semantic information, obtain the weighted sample ground semantic information corresponding to each sample ground semantic information.

[0119] The weighted sample ground semantic information is obtained by integrating the sample ground semantic information and its corresponding weights, which is convenient for subsequent use as input to the generative adversarial network. In specific implementation, the sample ground semantic information and its weights can be integrated by weighted summation to determine the weighted sample ground semantic information.

[0120] S1105, the sample ground mosaic map and the weighted sample ground semantic information are input into the generator of the initial generative adversarial network to generate the predicted noise-free map.

[0121] Among them, the weighted semantic information of the sample ground is used as a guiding condition and concatenated with the sample ground. Figure 1 As input to the generator of the generative adversarial network, the generator performs image processing under the guidance of weighted sample ground semantic information to remove reflective noise, thereby obtaining a noise-free predicted image.

[0122] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application assign weights to the sample ground semantic information, informing the generative adversarial network which part of the semantic information needs to be emphasized more when processing images, thereby accelerating the learning speed of the generative adversarial network for noiseless road surface background features and improving the quality of the image processed by the generative network.

[0123] S907, The predicted noise-free map and the noise-free ground mosaic map are input into the discriminator of the initial generative adversarial network, and the model loss is determined based on the discriminator's discrimination result.

[0124] The discrimination result indicates the difference between the predicted noise-free map and the noise-free ground mosaic map.

[0125] The model loss is the loss function of the generative adversarial network, and specifically, the model loss can be expressed as:

[0126]

[0127] Where x represents the noise-free ground mosaic; z represents the sample ground mosaic; G represents the generator network; and D represents the discriminator network. The generator needs to minimize the difference between the generated predicted noise-free map and the noise-free ground mosaic, while the discriminator needs to distinguish the difference between the predicted noise-free map and the noise-free ground mosaic as much as possible.

[0128] S909, Based on the model loss, adjust the network parameters of the initial generative adversarial network until the preset training termination condition is met, and obtain the pre-trained generative adversarial network.

[0129] Specifically, the model parameters of the generator and discriminator of the conditional generative adversarial network are adjusted based on the model loss until the preset training termination condition is met, and the generative network at the end of training is used as the pre-trained generative adversarial network of this invention embodiment.

[0130] The preset training termination condition can be that the model loss value reaches a preset loss threshold, or the number of iterations reaches a preset iteration number threshold.

[0131] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application use the output of the semantic segmentation model as a condition of the generator network and concatenate it with the sample ground. Figure 1 In the same input generative adversarial network, the learning speed of the generative network for noiseless road background features is accelerated, and the generative adversarial network is trained to learn to remove reflective noise in the image, providing a pre-trained model for the reflective removal module in the autonomous driving perception system.

[0132] Corresponding to the image processing methods provided in the above embodiments, this application also provides an image processing apparatus. Since the image processing apparatus provided in this application corresponds to the image processing methods provided in the above embodiments, the implementation methods of the aforementioned image processing methods are also applicable to the image processing apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0133] Please see Figure 12 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this application. This apparatus has the function of implementing the image processing method described in the above-described method embodiments. This function can be implemented in hardware or by hardware executing corresponding software. Figure 12 As shown, the device may include:

[0134] The first acquisition module 1210 is used to acquire a ground mosaic image around the target vehicle;

[0135] The first semantic segmentation module 1220 is used to perform semantic segmentation on the ground mosaic map to obtain at least one piece of ground semantic information.

[0136] Image generation module 1230 is used to input the ground mosaic image and the at least one piece of ground semantic information into the generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation image corresponding to the ground mosaic image.

[0137] The second semantic segmentation module 1240 is used to perform semantic segmentation on the noise-free inverse perspective transformation map to obtain at least one target semantic information.

[0138] In one exemplary embodiment, the image generation module includes:

[0139] The first weight allocation module is used to assign weights to the at least one piece of ground semantic information to obtain the weight corresponding to each piece of ground semantic information.

[0140] The first information synthesis module is used to obtain weighted ground semantic information corresponding to each ground semantic information based on the at least one ground semantic information and the weight corresponding to each ground semantic information;

[0141] The noise-free image generation module is used to input the ground mosaic image and the weighted ground semantic information into the generator of the pre-trained generative adversarial network to generate the noise-free inverse perspective transformation image.

[0142] In one exemplary embodiment, the apparatus further includes a model training module for training a generative adversarial network, the model training module comprising:

[0143] The second acquisition module is used to acquire the sample ground mosaic image and the noise-free ground mosaic image corresponding to the sample ground mosaic image;

[0144] The sample segmentation module is used to perform semantic segmentation on the sample ground mosaic map to obtain at least one piece of sample ground semantic information.

[0145] The prediction module is used to input the sample ground mosaic map and the at least one sample ground semantic information into the generator of the initial generative adversarial network to generate a noise-free prediction map.

[0146] The loss determination module is used to input the predicted noise-free map and the noise-free ground mosaic map into the discriminator of the initial generative adversarial network, and determine the model loss based on the discrimination result of the discriminator; the discrimination result indicates the difference between the predicted noise-free map and the noise-free ground mosaic map;

[0147] The parameter adjustment module is used to adjust the network parameters of the initial generative adversarial network based on the model loss until the preset training termination condition is met, thereby obtaining the pre-trained generative adversarial network.

[0148] In one exemplary implementation, the prediction module includes:

[0149] The second weight allocation module is used to allocate weights to the at least one sample ground semantic information to obtain the weights corresponding to each sample ground semantic information.

[0150] The second information synthesis module is used to obtain weighted sample ground semantic information corresponding to each sample ground semantic information based on the at least one sample ground semantic information and the weight corresponding to each sample ground semantic information.

[0151] The prediction image generation module is used to input the sample ground mosaic map and the weighted sample ground semantic information into the generator of the initial generative adversarial network to generate the prediction noise-free map.

[0152] In one exemplary embodiment, the sample segmentation module includes:

[0153] The first segmentation model module is used to input the sample ground mosaic image into a pre-trained inverse perspective transformation image segmentation model for semantic segmentation processing to obtain the at least one sample ground semantic information.

[0154] In one exemplary implementation, the first semantic segmentation module includes:

[0155] The second segmentation model module is used to input the ground mosaic image into the pre-trained inverse perspective transformation image segmentation model to generate the at least one piece of ground semantic information;

[0156] The second semantic segmentation module includes:

[0157] The third segmentation model module is used to input the noise-free inverse perspective transformation image into the pre-trained inverse perspective transformation image segmentation model to generate the at least one target semantic information.

[0158] In one exemplary embodiment, the apparatus further includes an image processing module for processing multiple fisheye images, the image processing module comprising:

[0159] The image acquisition module is used to acquire multiple fisheye images captured by multiple fisheye cameras around the target vehicle;

[0160] The parameter determination module is used to determine the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera.

[0161] The image transformation module is used to transform the fisheye image corresponding to each fisheye camera to the ground coordinate system based on the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera, so as to obtain multiple ground sub-images.

[0162] The image stitching module is used to stitch together the multiple ground sub-images to obtain the ground stitched image.

[0163] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0164] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement any of the image processing methods provided in the above method embodiments.

[0165] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0166] The method embodiments provided in this application can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 13 This is a hardware structure block diagram of an electronic device running an image processing method provided in an embodiment of this application, such as... Figure 13 As shown, the internal structure of this computer device may include, but is not limited to, a processor, a network interface, and a memory. The processor, network interface, and memory within the computer device can be connected via a bus or other means, as illustrated in the embodiments of this specification. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0167] The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. The network interface may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.). Memory is the storage device in the computer device used to store programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile storage device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, including but not limited to: Windows (an operating system), Linux (an operating system), Android (a mobile operating system), iOS (a mobile operating system), etc., which are not limited in this invention; and the storage space also stores one or more instructions suitable for being loaded and executed by the processor, which can be one or more computer programs (including program code). In the embodiments of this specification, the processor loads and executes one or more instructions stored in the memory to implement the image processing method provided in the above method embodiments.

[0168] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an image processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the image processing methods provided in the above-described method embodiments.

[0169] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0170] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0172] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0173] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a mosaic image of the ground surrounding the target vehicle; The ground mosaic image is semantically segmented to obtain at least one piece of ground semantic information; The ground mosaic and the at least one piece of ground semantic information are input into the generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation map corresponding to the ground mosaic. Semantic segmentation is performed on the noise-free inverse perspective transform image to obtain at least one target semantic information; The step of inputting the ground mosaic image and the at least one piece of ground semantic information into the generator of a pre-trained generative adversarial network to generate a noiseless inverse perspective transformation image corresponding to the ground mosaic image includes: assigning weights to the at least one piece of ground semantic information to obtain the weights corresponding to each piece of ground semantic information; obtaining weighted ground semantic information corresponding to each piece of ground semantic information based on the at least one piece of ground semantic information and the weights corresponding to each piece of ground semantic information; and inputting the ground mosaic image and the weighted ground semantic information into the generator of the pre-trained generative adversarial network to generate the noiseless inverse perspective transformation image. The training process of the generative adversarial network includes: acquiring a sample ground mosaic and a corresponding noise-free ground mosaic; performing semantic segmentation on the sample ground mosaic to obtain at least one sample ground semantic information; inputting the sample ground mosaic and the at least one sample ground semantic information into the generator of the initial generative adversarial network to generate a predicted noise-free map; inputting the predicted noise-free map and the noise-free ground mosaic into the discriminator of the initial generative adversarial network, and determining the model loss based on the discriminator's discrimination result; the discrimination result indicating the difference between the predicted noise-free map and the noise-free ground mosaic; and adjusting the network parameters of the initial generative adversarial network based on the model loss until a preset training termination condition is met, thereby obtaining the pre-trained generative adversarial network.

2. The image processing method according to claim 1, characterized in that, The step of inputting the sample ground mosaic map and the at least one sample ground semantic information into the generator of the initial generative adversarial network to generate a predicted noise-free map includes: Weights are assigned to the at least one sample ground semantic information to obtain the weights corresponding to each sample ground semantic information; Based on the at least one sample ground semantic information and the weight corresponding to each sample ground semantic information, the weighted sample ground semantic information corresponding to each sample ground semantic information is obtained; The sample ground mosaic and the weighted sample ground semantic information are input into the generator of the initial generative adversarial network to generate the predicted noise-free map.

3. The image processing method according to claim 1, characterized in that, The semantic segmentation of the sample ground mosaic map to obtain at least one piece of sample ground semantic information includes: The sample ground mosaic image is input into a pre-trained inverse perspective transformation image segmentation model for semantic segmentation processing to obtain at least one sample ground semantic information.

4. The image processing method according to claim 1, characterized in that, The semantic segmentation of the ground mosaic image yields at least one piece of ground semantic information, including: The ground mosaic image is input into the pre-trained inverse perspective transformation image segmentation model to generate at least one piece of ground semantic information; The semantic segmentation of the noise-free inverse perspective transform image to obtain at least one piece of target semantic information includes: The noise-free inverse perspective transformation image is input into the pre-trained inverse perspective transformation image segmentation model to generate the at least one target semantic information.

5. The image processing method according to any one of claims 1 to 4, characterized in that, Before acquiring a mosaic map of the ground surrounding the target vehicle, the method further includes: Acquire multiple fisheye images captured by multiple fisheye cameras around the target vehicle; Determine the intrinsic parameters and extrinsic parameters for each fisheye camera; Based on the intrinsic parameters and extrinsic parameters corresponding to each fisheye camera, the fisheye image corresponding to each fisheye camera is transformed to the ground coordinate system to obtain multiple ground sub-images; The multiple ground sub-maps are stitched together to obtain the stitched ground map.

6. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a mosaic image of the ground surrounding the target vehicle; The first semantic segmentation module is used to perform semantic segmentation on the ground mosaic map to obtain at least one piece of ground semantic information. An image generation module is used to input the ground mosaic image and the at least one piece of ground semantic information into the generator of a pre-trained generative adversarial network to generate a noise-free inverse perspective transformation image corresponding to the ground mosaic image. The second semantic segmentation module is used to perform semantic segmentation on the noiseless inverse perspective transformation image to obtain at least one target semantic information. The image generation module includes: a first weight allocation module, used to allocate weights to the at least one piece of ground semantic information to obtain the weight corresponding to each piece of ground semantic information; a first information synthesis module, used to obtain weighted ground semantic information corresponding to each piece of ground semantic information based on the at least one piece of ground semantic information and the weight corresponding to each piece of ground semantic information; and a noise-free image generation module, used to input the ground mosaic image and the weighted ground semantic information into the generator of the pre-trained generative adversarial network to generate the noise-free inverse perspective transformation image. The device further includes a model training module for training a generative adversarial network (GAN). The model training module includes: a second acquisition module for acquiring a sample ground mosaic and a corresponding noise-free ground mosaic; a sample segmentation module for semantically segmenting the sample ground mosaic to obtain at least one sample ground semantic information; a prediction module for inputting the sample ground mosaic and the at least one sample ground semantic information into the generator of an initial GAN ​​to generate a predicted noise-free image; a loss determination module for inputting the predicted noise-free image and the noise-free ground mosaic into the discriminator of the initial GAN, and determining the model loss based on the discriminator's discrimination result; the discrimination result indicating the difference between the predicted noise-free image and the noise-free ground mosaic; and a parameter adjustment module for adjusting the network parameters of the initial GAN ​​based on the model loss until a preset training termination condition is met, thereby obtaining the pre-trained GAN.

7. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 5.

Images