Model training and image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a model training and image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring sample image data; wherein the sample image data comprises a first fog image corresponding to the first scene, a reference fog purification image corresponding to the first scene, and a scene information graph corresponding to the first scene; performing feature extraction on the scene information graph corresponding to the first scene to obtain scene features; performing fog purification processing on the first fog image based on the scene features by using a target purification model to obtain a target purified image corresponding to the first fog image; and based on the reference fog purification image and the target purification image, updating model parameters of the target purification model at least once to obtain a trained target purification model. According to the method, the target purification model is trained by adopting the scene information graph comprising the depth graph and / or the semantic segmentation graph, so that the target purification model can better purify the fog image.

Description

Technical Field

[0001] The present invention relates to the field of automobile safe driving, and in particular to a model training and image processing method, device, electronic equipment and storage medium. Background Art

[0002] Heavy fog reduces visibility and poses a risk to vehicle driving. In related technologies, with the popularization of vehicle-mounted cameras and vehicle-mounted central control screens, intelligent fog purification modules have emerged. However, the fog purification modules have problems of low accuracy and poor flexibility. Summary of the invention

[0003] The present application provides a model training and image processing method, device, electronic device and storage medium to solve the problems of low accuracy and poor flexibility of the mist purification module in the related art.

[0004] In order to achieve the above objectives, the present application provides a model training method, the method comprising:

[0005] Acquire sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map;

[0006] Extracting features from the scene information graph corresponding to the first scene to obtain scene features;

[0007] Using the target purification model and based on the scene features, the first fog image is subjected to fog purification processing to obtain a target purified image corresponding to the first fog image;

[0008] Based on the reference mist purification image and the target purification image, the model parameters of the target purification model are updated at least once to obtain a trained target purification model.

[0009] According to the above technical means, by obtaining sample image data including a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene, and then extracting features from the scene information map corresponding to the first scene to obtain scene features, a target purification image corresponding to the first fog image is obtained based on the scene features using a target purification model, and then the model parameters of the target purification model are updated at least once according to the reference fog-purified image and the target purification image to obtain a trained target purification model. The present application uses a scene information map including a depth map and / or a semantic segmentation map to train the target purification model, so that the target purification model can better understand the scene information in the fog image to better purify the fog image, thereby improving the fog purification capability of the trained target purification model.

[0010] Furthermore, the target purification model includes an encoder and a decoder; using the target purification model, based on the scene features, the first fog image is subjected to fog purification processing to obtain a target purified image corresponding to the first fog image, including: using the encoder, encoding the scene features and the first fog image to obtain target image features corresponding to the target purified image; using the decoder, decoding the target image features to obtain a target purified image corresponding to the first fog image.

[0011] According to the above technical means, the scene features and the first fog image are first encoded by the encoder to obtain the target image features corresponding to the target purified image, and then the target image features are decoded by the decoder to obtain the target purified image corresponding to the first fog image, thereby improving the accuracy of the target purification model in processing the image.

[0012] Furthermore, the encoder includes an encoding convolution layer and an attention mechanism layer; using the encoder, the scene features and the first fog image are encoded to obtain target image features corresponding to the target purified image, including: using the encoding convolution layer, convolving the first fog image to obtain target image sub-features; using the attention mechanism layer, fusing the scene features and the target image sub-features to obtain target image features corresponding to the target purified image.

[0013] According to the above technical means, the first fog image is first convolved through the encoding convolution layer to obtain the target image sub-features, and then the scene features and the target image sub-features are fused through the attention mechanism layer to obtain the target image features corresponding to the target purified image, thereby improving the accuracy of the target image features.

[0014] Furthermore, the decoder includes a decoding convolution layer and a guided filter layer; using the decoder, the target image features are decoded to obtain a target purified image corresponding to the first foggy image, including: using the decoding convolution layer, convolution processing is performed on the target image features to obtain a target sub-image; using the guided filter layer, filtering processing is performed on the target sub-image to obtain a target purified image corresponding to the first foggy image.

[0015] According to the above technical means, the target image features are first convolved through the decoding convolution layer to obtain a target sub-image, and then the target sub-image is filtered through the guided filter layer to obtain a target purified image corresponding to the first fog image, so that the decoder can better decode the target image features. At the same time, the guided filter is used to better remove the noise in the image, thereby improving the fog purification processing effect of the target purification model.

[0016] Furthermore, the guided filter layer includes floating-point relative precision parameters; based on the reference fog-purified image and the target purified image, the model parameters of the target purification model are updated at least once to obtain the trained target purification model, including: based on the reference fog-purified image and the target purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolutional layer, and the floating-point relative precision parameters are updated at least once to obtain the trained target purification model.

[0017] According to the above-mentioned technical means, by referring to the fog-purified image and the target purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolutional layer, and the floating-point relative precision parameters are updated at least once to obtain the trained target purification model, so that the target purification model can process fog images of different visibilities through the corresponding model parameters, thereby improving the accuracy of the target purification model.

[0018] Furthermore, the target purification model includes N layers of encoders, and the target image features corresponding to the target purified image include the output image features of the Nth layer encoder; the encoder is used to encode the scene features and the first fog image to obtain the target image features corresponding to the target purified image, including: using the first layer encoder, encoding the input supplementary features of the first layer encoder and the first fog image to obtain the output image features of the first layer encoder; the input supplementary features of the first layer encoder are scene features; using the i-th layer encoder, encoding the output image features of the i-1th layer encoder and the input supplementary features of the i-th layer encoder to obtain the output image features of the i-th layer encoder; wherein i is a positive integer greater than 1 and less than N+1, and the input supplementary features of the i-th layer encoder are obtained by pooling the input supplementary features of the i-1th layer encoder based on the input feature size corresponding to the i-th layer encoder.

[0019] According to the above technical means, the input supplementary features of the first layer encoder and the first fog image are first encoded through the first layer encoder to obtain the output image features of the first layer encoder, and then the output image features of the i-1 layer encoder and the input supplementary features of the i layer encoder are encoded through the i-th layer encoder to obtain the output image features of the i-th layer encoder. The input fog image is encoded at least once using multiple layers of encoders, which improves the accuracy of the target image features while also improving the accuracy of the target purification model.

[0020] Furthermore, the target purification model includes a supplementary feature extraction module, and the scene features include depth features and / or semantic segmentation features; feature extraction is performed on the scene information map corresponding to the first scene to obtain scene features, including: when the scene information map includes a depth map, using the supplementary feature extraction module to perform feature extraction on the depth map to obtain depth features corresponding to the depth map; when the scene information map includes a semantic segmentation map, using the supplementary feature extraction module to perform feature extraction on the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map.

[0021] According to the above-mentioned technical means, by supplementing the feature extraction module, feature extraction is performed on the depth map and / or semantic segmentation map to obtain depth features and / or semantic segmentation features. According to different situations of the scene information map, targeted extraction can be performed on the scene information map, thereby improving the flexibility of feature extraction and the accuracy of scene features.

[0022] An image processing method, characterized in that the method comprises:

[0023] Acquire a second fog image to be processed;

[0024] The second fog image is subjected to fog purification processing by using the target purification model to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any of the above-mentioned model training methods.

[0025] According to the above-mentioned technical means, the target purification model obtained by any of the above-mentioned model training methods is used to perform fog purification on the acquired second foggy image to obtain a purified image corresponding to the second foggy image. On the one hand, since the target purification model can understand the scene information in the foggy image, the second foggy image can be better purified, thereby improving the fog purification ability of the image processing method; on the other hand, since the target purification model is similar to a software plug-in, it can be called by various video image processing processes, thereby improving the flexibility of the image processing method.

[0026] A model training device, comprising:

[0027] A first acquisition unit is used to acquire sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map;

[0028] An extraction unit, used to extract features from the scene information graph corresponding to the first scene to obtain scene features;

[0029] A first determination unit is configured to perform a fog purification process on the first fog image based on scene features by using a target purification model to obtain a target purified image corresponding to the first fog image;

[0030] The updating unit is used to update the model parameters of the target purification model at least once based on the reference mist purification image and the target purification image to obtain a trained target purification model.

[0031] An image processing device, characterized in that the device comprises:

[0032] A second acquisition unit, used for acquiring a second fog image to be processed;

[0033] The second determination unit is used to use the target purification model to perform fog purification on the second fog image to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any of the above-mentioned model training methods.

[0034] An electronic device comprises a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, any one of the above methods is implemented.

[0035] A computer-readable storage medium stores a computer program, which implements any of the above methods when executed by a processor.

[0036] A computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in any of the above methods are implemented.

[0037] A vehicle comprises the above electronic device.

[0038] Beneficial effects of this application:

[0039] (1) This application uses a scene information map including a depth map and / or a semantic segmentation map to train the target purification model, so that the target purification model can better understand the scene information in the foggy image to better purify the foggy image, thereby improving the fog purification ability of the trained target purification model;

[0040] (2) The scene features and the first fog image are first encoded by an encoder to obtain target image features corresponding to the target purified image, and then the target image features are decoded by a decoder to obtain a target purified image corresponding to the first fog image, thereby improving the accuracy of the image processed by the target purification model;

[0041] (3) By referring to the fog-purified image and the target-purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative accuracy parameters are updated at least once to obtain a trained target purification model, so that the target purification model can process fog images of different visibilities through corresponding model parameters, thereby improving the accuracy of the target purification model;

[0042] (4) Using a multi-layer encoder, the input fog image is encoded at least once, which improves the accuracy of the target image features and the accuracy of the target purification model;

[0043] (5) Since the target purification model can understand the scene information in the foggy image, it can better purify the second foggy image, thereby improving the fog purification ability of the image processing method. At the same time, since the target purification model is similar to a software plug-in, it can be called by various video image processing processes, thereby improving the flexibility of the image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the implementation process of a model training method provided in an embodiment of the present application Figure 1 ;

[0045] Figure 2 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 2 ;

[0047] Figure 4 A schematic diagram of mist purification provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a target purification model provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of a training target purification model provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a data set provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of images corresponding to different parameters provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of downsampling provided in an embodiment of the present application;

[0053] Fig.10A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0054] Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0055] Fig.12 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0057] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0058] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Heavy fog reduces visibility and poses a risk to vehicle driving. In terms of manual driving, heavy fog blocks the driver's field of vision, affects the driver's judgment, and increases the possibility of traffic accidents; in terms of autonomous driving, heavy fog reduces the accuracy of commonly used target detection algorithms. Objects obscured by heavy fog may cause vehicles to "brake suddenly" or "turn sharply" to avoid collisions, which may cause major traffic accidents involving multiple vehicles rear-ending each other, endangering public transportation safety.

[0062] In the related technology, with the popularization of vehicle-mounted cameras and vehicle-mounted central control screens, intelligent fog purification modules have emerged. However, the fog purification module only uses various fog images during the training stage and does not consider the information of different traffic scenes, resulting in problems of low accuracy and poor flexibility.

[0063] The embodiment of the present application provides a model training method, by acquiring sample image data including a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene, and then extracting features from the scene information map corresponding to the first scene to obtain scene features, so as to use a target purification model to obtain a target purified image corresponding to the first fog image based on the scene features, thereby updating the model parameters of the target purification model at least once according to the reference fog-purified image and the target purified image to obtain a trained target purification model. The present application uses a scene information map including a depth map and / or a semantic segmentation map to train the target purification model, so that the target purification model can better understand the scene information in the fog image to better purify the fog image, thereby improving the fog purification capability of the trained target purification model.

[0064] Below, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application.

[0065] Figure 1 A schematic diagram of the implementation process of a model training method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes steps S101 to S104, wherein:

[0066] Step S101, obtaining sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map.

[0067] Here, the sample image data is the data used to train the target purification model. The sample image data may include, but is not limited to, at least one of: a first fog image corresponding to the first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene. The scene information map may include, but is not limited to, at least one of: a depth map, a semantic segmentation map, and the like.

[0068] The first scene refers to a scene with fog. The first fog image corresponding to the first scene refers to an image obtained from the first scene with fog. The reference fog-purified image corresponding to the first scene refers to a clean image obtained after purifying the first scene with fog. The depth map is a grayscale image, and the brightness value of each pixel in the depth map represents the depth information of the pixel in three-dimensional space. The depth map corresponding to the first scene is used to characterize the depth information corresponding to different objects in the first scene. The semantic segmentation map is a color image, and each pixel in the semantic segmentation map is assigned a label, indicating which category or object in the image the pixel belongs to. The label may include but is not limited to: at least one of: color, number, other identifiers, etc. The semantic segmentation map corresponding to the first scene is used to characterize the categories or objects corresponding to different objects in the first scene.

[0069] The method for obtaining sample image data may include but is not limited to: obtaining sample image data from a data set, obtaining sample image data through a camera, etc. For example, a data set contains a large number of annotated images, which are suitable for various computer vision tasks. You can select a data set and obtain sample image data from the data set. The data set may include but is not limited to: at least one of SynFog, ImageNet, PASCAL VOC, etc. For another example, at least one image is taken by a camera in the first scene, and the image may include but is not limited to: a first fog image corresponding to the first scene, a depth map corresponding to the first scene, and a reference fog-purified image corresponding to the first scene. Then, the reference fog-purified image corresponding to the first scene is semantically segmented to obtain a semantic segmentation map corresponding to the first scene, so that the sample image data can be obtained.

[0070] In some embodiments, the first fog image corresponding to the first scene may be at least one image corresponding to different visibilities, or may be at least one image corresponding to different situations when the vehicle lights are turned on and the vehicle lights are turned off.

[0071] In some embodiments, the reference mist-cleared image corresponding to the first scene may be at least one image corresponding to different situations when the vehicle lights are turned on and the vehicle lights are turned off.

[0072] Exemplarily, the reference fog-purified image corresponding to the first scene may include: a clean image A1 with the headlights off, and an image A4 of a clean image with the headlights on; the first fog image corresponding to the first scene may include: a fog image A2 with a visibility of 600 meters with the headlights off, a fog image A3 with a visibility of 300 meters with the headlights off, a fog image A5 with a visibility of 600 meters with the headlights on, and a fog image A6 with a visibility of 300 meters with the headlights on.

[0073] Step S102: extracting features from the scene information graph corresponding to the first scene to obtain scene features.

[0074] Here, the scene feature refers to an image feature of the scene information graph corresponding to the first scene. In some embodiments, the number of scene features may be at least one.

[0075] Scene features may include, but are not limited to, at least one of pixel features, region features, geometric features, point cloud features, surface features, etc. Pixel features may include, but are not limited to, at least one of color histograms, texture features, etc. Region features may include, but are not limited to, at least one of shape features, boundary features, region relationships, etc. Geometric features may include, but are not limited to, at least one of depth differences, normal vectors, curvatures, etc. Surface features may include, but are not limited to, at least one of surface normals, surface curvatures, etc.

[0076] The method for extracting features from the scene information graph corresponding to the first scene may include, but is not limited to: extracting features from the scene information graph corresponding to the first scene through a Histogram of Oriented Gradient (HOG), extracting features from the scene information graph corresponding to the first scene through a Scale-Invariant Feature Transform (SIFT), extracting features from the scene information graph corresponding to the first scene through an image feature extraction network, etc. For example, through HOG, the scene information graph corresponding to the first scene can be divided into small connected areas, and then the gradient or edge direction histograms of each pixel in the connected area are collected, and finally these histograms are combined to obtain scene features. For another example, through SIFT, the key points in the scene information graph corresponding to the first scene can be detected, and then the scene information graph corresponding to the first scene can be divided into multiple small areas with the key points as the center, and multiple directions of gradient histograms are created for each small area, and then the scene features are obtained according to the gradient histograms in multiple directions. For another example, the image feature extraction network can extract features from the input image. The image feature extraction network may include but is not limited to at least one of: ResNet, Pyramid Pooling Module (PPM), Convolutional Neural Networks (CNN), etc. The scene information graph corresponding to the first scene is input into the image feature extraction network, and the scene information graph corresponding to the first scene is subjected to feature extraction through the image feature extraction network to obtain scene features.

[0077] In some embodiments, the target purification model includes a supplementary feature extraction module, and the scene features include depth features and / or semantic segmentation features. When the scene information graph includes a depth map, the supplementary feature extraction module can be used to extract features from the depth map to obtain depth features corresponding to the depth map. When the scene information graph includes a semantic segmentation map, the supplementary feature extraction module can be used to extract features from the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map.

[0078] Step S103 , using the target purification model and based on the scene features, the first fog image is subjected to fog purification processing to obtain a target purified image corresponding to the first fog image.

[0079] Here, the target purification model can process the input image to obtain a target purified image corresponding to the input image. In some embodiments, the target purification model can be established based on U-Net. U-Net is a CNN architecture for image segmentation, having a symmetrical encoder-decoder structure, wherein the encoder has a feature extraction function and the decoder has a function of upsampling and reconstructing data.

[0080] In some embodiments, the depth map indicates the distance of the object in the image from the camera, and can help the target purification model understand the structure of the reference fog-purified image corresponding to the first scene from the depth direction. The semantic segmentation map can accurately distinguish the position of the object in the image, and can help the target purification model understand the structure of the reference fog-purified image corresponding to the first scene from the horizontal direction. Through the scene features, the target purification model can more accurately perform fog purification on the first fog image.

[0081] In some embodiments, the first fog image is used as input information of the target purification model, and the scene features are also input into the target purification model as auxiliary parameters of the first fog image. After the fog purification process, the target purification model outputs a target purification image corresponding to the first fog image.

[0082] In some embodiments, the target purification model includes an encoder and a decoder. The encoder can be used to encode the scene features and the first fog image to obtain the target image features corresponding to the target purified image, and then the decoder can be used to decode the target image features to obtain the target purified image corresponding to the first fog image.

[0083] Step S104: based on the reference mist purification image and the target purification image, the model parameters of the target purification model are updated at least once to obtain a trained target purification model.

[0084] Here, the updating method of the model parameters may include but is not limited to at least one of the gradient descent method, the momentum update method, the Newton momentum method, etc.

[0085] In some embodiments, a target loss value may be determined based on a reference mist purification image and a target purification image, and then the model parameters of the target purification model may be updated at least once according to the target loss value to obtain a trained target purification model. The target loss value may include, but is not limited to, at least one of MSE (mean squared error) loss, MAE (mean absolute error) loss, RMSE (root mean squared error) loss, LAD (least absolute deviation) loss, etc.

[0086] In some embodiments, the target loss value may be calculated using at least one of the Pearson Linear Correlation Coefficient (PLCC), the Spearman Rank-order Correlation Coefficient (SRCC), the Kendall Rank-order Correlation Coefficient (KRCC), etc.

[0087] In some embodiments, it may be determined whether the model parameters of the target purification model need to be updated based on the target loss value. For example, the target loss value is compared with a threshold value, and when the target loss value is greater than the threshold value, the target purification model is updated; when the target loss value is not greater than the threshold value, the target purification model is determined as the trained target purification model. For another example, the target loss value is compared with the previous target loss value, and when the target loss value is different from the previous target loss value, the model parameters of the target purification model are updated; when the target loss value is substantially equal to the previous target loss value, the target purification model is determined as the trained target purification model.

[0088] In some embodiments, when the model parameters of the target purification model are updated according to the target loss value, the target loss value can be continuously reduced through back propagation and multiple rounds of training to update the model parameters of the target purification model at least once to obtain a trained target purification model.

[0089] In some embodiments, the model parameters of the target purification model can be updated through the main function. The learning rate, the number of iterations, and the optimized learning rate are set in the main function, and then the model parameters of the target purification model are updated using the main function according to the reference mist purification image and the target purification image.

[0090] In some embodiments, the target purification model includes an encoder and a decoder, the decoder includes a decoding convolution layer and a guided filter layer, the guided filter layer includes a floating-point relative precision parameter, and the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative precision parameters can be updated at least once based on a reference fog purification image and a target purification image to obtain a trained target purification model.

[0091] In an embodiment of the present application, sample image data including a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information graph corresponding to the first scene are obtained, and then feature extraction is performed on the scene information graph corresponding to the first scene to obtain scene features, so as to use a target purification model to obtain a target purified image corresponding to the first fog image based on the scene features, thereby updating the model parameters of the target purification model at least once according to the reference fog-purified image and the target purified image to obtain a trained target purification model. The present application uses a scene information graph including a depth map and / or a semantic segmentation map to train the target purification model, so that the target purification model can better understand the scene information in the fog image to better purify the fog image, thereby improving the fog purification capability of the trained target purification model.

[0092] In some implementations, the target purification model includes an encoder and a decoder; step S103 includes step S131 and step S132, wherein:

[0093] Step S131 , using an encoder to encode the scene features and the first fog image to obtain target image features corresponding to the target purified image.

[0094] Here, the components of the encoder may include but are not limited to: at least one of an encoding convolution layer, an attention mechanism layer, an activation function, etc. The activation function is a nonlinear function that can increase the nonlinear expression capability of the target purification model. The encoder can encode the first fog image in combination with the scene features to generate a low-dimensional feature representation, and use the feature as the target image feature corresponding to the target purification image.

[0095] The target image feature refers to an image feature corresponding to the target cleansed image. In some embodiments, the number of the target image feature may be at least one.

[0096] The target image features may include but are not limited to: at least one of pixel features, region features, geometric features, point cloud features, surface features, etc. Pixel features may include but are not limited to: at least one of color histogram, texture features, etc. Region features may include but are not limited to: at least one of shape features, boundary features, region relationships, etc. Geometric features may include but are not limited to: at least one of depth difference, normal vector, curvature, etc. Surface features may include but are not limited to: at least one of surface normal, surface curvature, etc.

[0097] In some embodiments, the encoder includes an encoding convolution layer and an attention mechanism layer. The encoding convolution layer can be used to first perform convolution processing on the first fog image to obtain target image sub-features, and then the attention mechanism layer can be used to fuse the scene features and the target image sub-features to obtain the target image features corresponding to the target purified image.

[0098] In some embodiments, the target purification model includes N layers of encoders, and the target image features corresponding to the target purified image include the output image features of the Nth layer encoder. The first layer encoder can be used to encode the input supplementary features of the first layer encoder and the first fog image to obtain the output image features of the first layer encoder, wherein the input supplementary features of the first layer encoder are scene features, and then the i-th layer encoder is used to encode the output image features of the i-1th layer encoder and the input supplementary features of the i-th layer encoder to obtain the output image features of the i-th layer encoder. Wherein, i is a positive integer greater than 1 and less than N+1, and the input supplementary features of the i-th layer encoder are obtained by pooling the input supplementary features of the i-1th layer encoder based on the input feature size corresponding to the i-th layer encoder.

[0099] Step S132: using a decoder to decode the target image features to obtain a target purified image corresponding to the first fog image.

[0100] Here, the components of the decoder may include but are not limited to: at least one of a decoding convolution layer, a guided filter layer, an activation function, etc. The decoder may decode the target image features and restore the target image features to a target purified image corresponding to the first fog image.

[0101] In some embodiments, there are corresponding skip connections between the encoder and the decoder. These skip connections allow the reuse of features in the encoder, reduce redundant calculations in the decoder, and transmit high-level information directly to the bottom layer of the network to prevent the loss of high-quality details.

[0102] In some embodiments, the decoder includes a decoding convolution layer and a guided filter layer. The decoding convolution layer can be used to first perform convolution processing on the target image features to obtain a target sub-image, and then the guided filter layer can be used to perform filtering processing on the target sub-image to obtain a target purified image corresponding to the first fog image.

[0103] In an implementation manner of the present application, the scene features and the first fog image are first encoded by an encoder to obtain target image features corresponding to the target purified image, and then the target image features are decoded by a decoder to obtain a target purified image corresponding to the first fog image, thereby improving the accuracy of image processing by the target purification model.

[0104] In some embodiments, the encoder includes an encoding convolution layer and an attention mechanism layer; step S131 includes step S1311 and step S1312, wherein:

[0105] Step S1311, using the coding convolution layer, convolution processing is performed on the first fog image to obtain target image sub-features.

[0106] Here, the encoding convolution layer is used to extract image features, and the first fog image may be convolved to obtain target image sub-features. The target image sub-features include but are not limited to: at least one of pixel features, region features, geometric features, point cloud features, surface features, etc. In some embodiments, the number of target image sub-features may be at least one.

[0107] In some embodiments, the encoder may further include an activation function, by which the output of the encoding convolutional layer may be converted into a nonlinear activation value, thereby obtaining the target image sub-features according to the nonlinear activation values ​​and enhancing the discrimination of the target image sub-features.

[0108] In some embodiments, the encoder further includes a pooling layer, which can perform downsampling processing on the first fog image to reduce the size of the first fog image, thereby reducing the amount of calculation and the number of parameters.

[0109] Step S1312, using the attention mechanism layer, the scene features and the target image sub-features are fused to obtain the target image features corresponding to the target purified image.

[0110] Here, the attention mechanism is a technology widely used in deep learning, which can fuse different input information, thereby helping the target purification model to better capture the correlation between the two inputs. The scene features and the target image sub-features are fused through the attention mechanism layer, and the target image sub-features after enhancing the scene information are used as the target image features corresponding to the target purified image.

[0111] In some embodiments, the attention mechanism layer can be established based on the SKnet attention mechanism. The SKnet attention mechanism sets a set of dynamically selected convolutions, which are divided into the following three parts: Split, Fuse, and Select. Among them, Split can perform a complete convolution operation with different convolution kernel sizes on the input vector, Fuse can perform a global average pooling operation on the fused features, and Select uses two weight matrices a and b to perform weighted operations on the previous two features.

[0112] In some embodiments, when the scene features and the target image sub-features are fused using the attention mechanism layer, each scene feature has a corresponding mask, which is used to indicate the importance of the scene feature. The larger the value of the mask, the higher the weight of the corresponding scene feature, and when the scene features and the target image sub-features are fused, more information of the scene feature can be used.

[0113] In the implementation mode of the present application, the first fog image is first convolved through the encoding convolution layer to obtain the target image sub-features, and then the scene features and the target image sub-features are fused through the attention mechanism layer to obtain the target image features corresponding to the target purified image, thereby improving the accuracy of the target image features.

[0114] In some embodiments, the decoder includes a decoding convolution layer and a guided filter layer; step S132 includes step S132a and step S132b, wherein:

[0115] Step S132a, using the decoding convolution layer, convolution processing is performed on the target image features to obtain a target sub-image.

[0116] Here, the target sub-image is an image obtained by performing convolution processing on the target image features. In some embodiments, the number of the target sub-image may be at least one. The decoding convolution layer is used to perform convolution processing on the target image features to obtain the target sub-image.

[0117] In some embodiments, the decoder may further include an activation function, by which the output of the decoding convolutional layer may be converted into a nonlinear activation value, so that a target sub-image is obtained according to the nonlinear activation value, thereby enhancing the discrimination of the target sub-image.

[0118] In some embodiments, the decoder further includes a pooling layer, which can perform upsampling processing on the first fog image to increase the size of the target sub-image.

[0119] Step S132b: Using the guided filter layer, filter the target sub-image to obtain a target purified image corresponding to the first fog image.

[0120] Here, guided filtering is an image smoothing and detail enhancement technology widely used in image processing and computer vision. It can use an image (also called a guided image) to guide the filtering process of another image (also called an input image). Through guided filtering, image edge preservation filtering, image noise removal, image edge feathering, image enhancement, etc. can be achieved. In the process of fog purification, fog is equivalent to image noise. After adding a guided filter layer, it can help the target purification model to better remove fog and retain the edges and contrast of the original image.

[0121] In some implementations, when filtering is performed on a target sub-image using a guided filter layer, both the guided image and the input image may be the target sub-image.

[0122] In some embodiments, the target purification model includes N layers of decoders, each layer of the decoder includes a decoding convolution layer and a guided filtering layer. The target image features can be first convolved through the decoding convolution layer in the first layer of the decoder to obtain a target sub-image, and then the target sub-image is filtered through the guided filtering layer in the first layer of the decoder to obtain a target output sub-image corresponding to the target sub-image. Then, by using the i-th layer decoder, the target output sub-image output by the i-1-th layer decoder is convolved and filtered. After N times of processing, the target purified image corresponding to the first fog image is obtained.

[0123] In the implementation mode of the present application, the target image features are first convolved through a decoding convolution layer to obtain a target sub-image, and then the target sub-image is filtered through a guided filtering layer to obtain a target purified image corresponding to the first fog image, so that the decoder can better decode the target image features. At the same time, the guided filtering is used to better remove the noise in the image, thereby improving the fog purification processing effect of the target purification model.

[0124] In some embodiments, the guided filter layer includes a floating point relative precision parameter; step S104 includes step S14, wherein:

[0125] Step S14, based on the reference fog-purified image and the target purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative accuracy parameters are updated at least once to obtain a trained target purification model.

[0126] Here, the floating point relative precision parameter (also called the Eps parameter) is used to describe the degree of error that occurs when a computer calculates a floating point number. The Eps parameter can be any suitable size, for example, 0.01, 0.04, 0.09, etc.

[0127] In some implementations, a higher Eps parameter indicates a smaller error in the floating-point calculation result, and a lower Eps parameter indicates a larger error in the floating-point calculation result.

[0128] In some embodiments, the guided filter layer also includes a filter window radius. The Eps parameter is related to the variance. When the filter window radius is the same, the larger the Eps parameter, the smaller the variance, the sharper the edge of the target purified image, and the better the edge preservation effect. The smaller the Eps parameter, the larger the variance, and the smoother the edge of the target purified image.

[0129] In some embodiments, the greater the visibility corresponding to the first fog image, the greater the Eps parameter, and the smaller the visibility corresponding to the first fog image, the smaller the Eps parameter.

[0130] In some implementations, when the Eps parameter is updated at least once, the visibility corresponding to the first fog image is different, and the updated Eps parameter may be different.

[0131] The encoding parameters of the encoder may include but are not limited to: at least one of the encoding convolution kernel size, the number of encoding convolution layers, the learning rate, etc. The decoding parameters of the decoding convolution layer may include but are not limited to: at least one of the decoding convolution kernel size, the number of decoding convolution layers, the learning rate, etc.

[0132] In some embodiments, a target loss value may be determined based on a reference mist-purified image and a target-purified image, and then the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative precision parameters are updated at least once according to the target loss value to obtain a trained target purification model. The target loss value may include, but is not limited to, at least one of MSE loss, MAE loss, RMSE loss, LAD loss, and the like.

[0133] In some embodiments, the updating method of the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative precision parameters may include but is not limited to at least one of the gradient descent method, the momentum update method, the Newton momentum method, etc.

[0134] In an embodiment of the present application, by referring to the fog-purified image and the target purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolutional layer, and the floating-point relative precision parameters are updated at least once to obtain a trained target purification model, so that the target purification model can process fog images of different visibilities through corresponding model parameters, thereby improving the accuracy of the target purification model.

[0135] In some embodiments, the target purification model includes N layers of encoders, and the target image features corresponding to the target purified image include output image features of the Nth layer of encoders; step S132 includes step S132c and step S132d, wherein:

[0136] Step S132c, using the first layer encoder, encoding the input supplementary features of the first layer encoder and the first fog image to obtain the output image features of the first layer encoder; the input supplementary features of the first layer encoder are scene features.

[0137] Here, the output image feature of the first layer encoder refers to the image feature corresponding to the first fog image processed by the first layer encoder. In some embodiments, the number of the output image feature of the first layer encoder may be at least one.

[0138] The output image features of the first layer encoder may include, but are not limited to, at least one of: pixel features, region features, geometric features, point cloud features, surface features, etc.

[0139] In some embodiments, the first encoder layer includes a first encoding convolution layer and a first attention mechanism layer. The first encoding convolution layer can be used to perform convolution processing on the first fog image to obtain the first target image sub-features, and then the first attention mechanism layer can be used to fuse the scene features and the first target image sub-features to obtain the output image features of the first encoder layer.

[0140] The process of encoding the input supplementary features of the first layer encoder and the first fog image using the first layer encoder may refer to the specific implementation process of the aforementioned step S131.

[0141] Step S132d, using the i-th layer encoder, encodes the output image features of the i-1-th layer encoder and the input supplemental features of the i-th layer encoder to obtain the output image features of the i-th layer encoder; wherein i is a positive integer greater than 1 and less than N+1, and the input supplemental features of the i-th layer encoder are obtained by pooling the input supplemental features of the i-1-th layer encoder based on the input feature size corresponding to the i-th layer encoder.

[0142] Here, i is any appropriate positive integer greater than 1 and less than N+1, for example, 2, 3, etc.

[0143] In some embodiments, the i-th encoder includes an i-th pooling layer. Based on the input feature size corresponding to the i-th encoder, the i-th pooling layer of the i-th encoder may be used to perform pooling processing on the input supplementary features of the i-1-th encoder to obtain the input supplementary features of the i-th encoder.

[0144] The process of using the i-th layer encoder to encode the output image features of the i-1-th layer encoder and the input supplemental features of the i-th layer encoder to obtain the output image features of the i-th layer encoder can refer to the specific implementation process of the aforementioned step S131.

[0145] In the implementation manner of the present application, the first layer encoder is first used to encode the input supplementary features of the first layer encoder and the first fog image to obtain the output image features of the first layer encoder, and then the i-th layer encoder is used to encode the output image features of the i-1 layer encoder and the input supplementary features of the i-th layer encoder to obtain the output image features of the i-th layer encoder. The input fog image is encoded at least once using multiple layers of encoders, which improves the accuracy of the target image features while also improving the accuracy of the target purification model.

[0146] In some embodiments, the target purification model includes a supplementary feature extraction module, and the scene features include depth features and / or semantic segmentation features; step S102 includes step S121 and step S122, wherein:

[0147] Step S121, when the scene information map includes a depth map, a supplementary feature extraction module is used to extract features from the depth map to obtain depth features corresponding to the depth map.

[0148] Here, the supplementary feature extraction module has a feature extraction function. In some embodiments, the supplementary feature extraction module includes at least one feature extraction layer.

[0149] The depth features corresponding to the depth map may include, but are not limited to, at least one of pixel features, region features, context features, etc. The context features may include, but are not limited to, at least one of neighborhood information, global context, etc.

[0150] In some implementations, the depth map may be used as an input of a supplementary feature extraction module, so that the supplementary feature extraction module is used to perform feature extraction on the depth map to obtain depth features corresponding to the depth map.

[0151] Step S122, when the scene information map includes a semantic segmentation map, a supplementary feature extraction module is used to perform feature extraction on the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map.

[0152] Here, the semantic segmentation features corresponding to the semantic segmentation map may include but are not limited to: at least one of geometric features, point cloud features, surface features, etc. Geometric features may include but are not limited to: at least one of depth difference, normal vector, curvature, etc. Surface features may include but are not limited to: at least one of surface normal, surface curvature, etc.

[0153] In some implementations, the semantic segmentation map may be used as an input of a supplementary feature extraction module, so that the supplementary feature extraction module is used to perform feature extraction on the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map.

[0154] In the implementation manner of the present application, feature extraction is performed on the depth map and / or semantic segmentation map by supplementing the feature extraction module to obtain depth features and / or semantic segmentation features. Targeted extraction can be performed on the scene information map according to different situations of the scene information map, thereby improving the flexibility of feature extraction and the accuracy of scene features.

[0155] Based on the above embodiments, the present application also provides a model training method. Figure 2 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes steps S201 to S202, wherein:

[0156] Step S201, obtaining a second fog image to be processed.

[0157] Here, the second fog image to be processed refers to an image obtained from a foggy scene. In some embodiments, the visibility corresponding to different second fog images to be processed may be the same or different.

[0158] The method for obtaining the second fog image to be processed may include but is not limited to: obtaining sample image data through a camera, reading the second fog image to be processed from the cloud, etc. For example, the second fog image to be processed may be obtained by photographing the current scene through a camera. For another example, a large amount of data may be stored in the cloud, and the second fog image to be processed may be read from the cloud.

[0159] Step S202, using the target purification model, performing fog purification processing on the second fog image to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any model training method in claims 1 to 7.

[0160] Here, the second fog image to be processed may be input as input information into the target purification model, and the second fog image may be subjected to fog purification processing by the target purification model to obtain a purified image corresponding to the second fog image.

[0161] In some embodiments, the target purification model is obtained by any of the above-mentioned model training methods.

[0162] In some implementations, the target purification model may be deployed in a mist purification module, so that the mist purification module may be called to perform mist purification processing on the second mist image to obtain a purified image corresponding to the second mist image.

[0163] In some embodiments, the mist purification module may be called by turning on the mist purification function.

[0164] In some embodiments, a purified image corresponding to the second fog image can be displayed on the central control screen of the vehicle, and the image displayed on the central control screen can be detected by an object detection module. When the object detection module detects that the image displayed on the central control screen of the vehicle is the purified image corresponding to the second fog image, the fog purification function can be turned off.

[0165] In the implementation example of the present application, the target purification model obtained by any of the above-mentioned model training methods is used to perform fog purification on the acquired second fog image to obtain a purified image corresponding to the second fog image. On the one hand, since the target purification model can understand the scene information in the fog image, the second fog image can be better purified, thereby improving the fog purification ability of the image processing method; on the other hand, since the target purification model is similar to a software plug-in, it can be called by various video image processing processes, thereby improving the flexibility of the image processing method.

[0166] The following describes the application of the model training method provided in an embodiment of the present application in an actual scenario, taking a vehicle equipped with a fog purification module as an example.

[0167] Heavy fog reduces visibility and poses a risk to vehicle driving. In terms of manual driving, heavy fog blocks the driver's field of vision, affects the driver's judgment, and increases the possibility of traffic accidents; in terms of autonomous driving, heavy fog reduces the accuracy of commonly used target detection algorithms. Objects obscured by heavy fog may cause vehicles to "brake suddenly" or "turn sharply" to avoid collisions, which may cause major traffic accidents involving multiple vehicles rear-ending each other, endangering public transportation safety.

[0168] In the related technology, with the popularization of vehicle-mounted cameras and vehicle-mounted central control screens, intelligent fog purification modules have emerged. However, the fog purification module only uses various fog images during the training stage and does not consider the information of different traffic scenes, resulting in problems of low accuracy and poor flexibility.

[0169] The embodiment of the present application provides a model training method, by acquiring sample image data including a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene, and then extracting features from the scene information map corresponding to the first scene to obtain scene features, so as to use a target purification model to obtain a target purified image corresponding to the first fog image based on the scene features, thereby updating the model parameters of the target purification model at least once according to the reference fog-purified image and the target purified image to obtain a trained target purification model. The present application uses a scene information map including a depth map and / or a semantic segmentation map to train the target purification model, so that the target purification model can better understand the scene information in the fog image to better purify the fog image, thereby improving the fog purification capability of the trained target purification model.

[0170] like Figure 3 As shown, the method includes steps S301 to S309, wherein: Figure 3 A schematic diagram of the implementation process of a model training method provided in the embodiment of the present application Figure 2The method includes steps S301 to S307, wherein:

[0171] Step S301, obtaining sample image data from the SynFog dataset;

[0172] Here, the sample image data includes a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene, wherein the scene information map includes a depth map and a semantic segmentation map.

[0173] Step S302, using a supplementary feature extraction module to perform feature extraction on the depth map to obtain depth features corresponding to the depth map, and using the supplementary feature extraction module to perform feature extraction on the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map;

[0174] Step S303, using the coding convolution layer, performing convolution processing on the first fog image to obtain target image sub-features;

[0175] Step S304, using the attention mechanism layer, the depth features, the semantic segmentation features and the target image sub-features are fused to obtain the target image features corresponding to the target purified image;

[0176] Step S305, using the decoding convolution layer, convolution processing is performed on the target image features to obtain a target sub-image;

[0177] Step S306, using the guided filter layer, filtering the target sub-image to obtain a target purified image corresponding to the first fog image;

[0178] Step S307 , based on the minimum absolute value deviation loss between the reference mist-purified image and the target purified image, the model parameters of the target purification model are updated at least once to obtain a trained target purification model.

[0179] Figure 4 A schematic diagram of mist purification provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, where:

[0180] When fog is detected in front of the vehicle, the fog purification function is turned on 41;

[0181] Here, the fog purification function can be made into an icon button under the weather bar on the car's central control screen. When the driver finds that the fog affects driving, he can turn on the fog purification by clicking the button or voice control.

[0182] Collecting the fog image to be processed 42 through the car camera;

[0183] Inputting the fog image 42 to be processed into the fog purification module 43, and performing fog purification processing on the fog image 42 to be processed by the fog purification module 43 to obtain a purified image corresponding to the fog image 42 to be processed;

[0184] The purified image corresponding to the fog image 42 to be processed is displayed on the central control screen 44. At the same time, the image displayed on the central control screen 44 is detected by the target detection module 45. When the target detection module 45 detects that the image displayed on the central control screen 44 is the purified image corresponding to the fog image 42 to be processed, the fog purification function 41 can be turned off to end the fog purification.

[0185] Figure 5 A schematic diagram of a target purification model provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the target purification model includes a supplementary feature extraction module 51, an encoder 52 and a decoder 53, wherein:

[0186] The supplementary feature extraction module 51 can extract features from the depth map and the semantic segmentation map, and input the extracted features (corresponding to the rectangles in the map) into the corresponding encoder 52;

[0187] The encoder 52 can extract the features of the fog image by encoding the convolutional layer, such as Figure 5 As shown, the encoder 52 includes 5 layers of encoders. Each downsampling layer will reduce the image by half and double the number of convolution kernels. Each layer of the encoder includes: encoding convolution layer 521, SKNet attention mechanism layer 522 and pooling layer 523. Only the number of the first layer of encoder is marked in the figure, and the components of each layer are the same. The 5 encoding convolution layers (3×3, ReLU) correspond to the convolution kernel sizes of 3×3, 3×3, 3×3, 3×3, 3×3, and the number of convolution kernels is 64, 128, 256, 512, and 1024. Each 3×3 convolution is followed by a ReLU activation function and a 2×2 maximum pooling (max pool 2×2), and the number of output layers is 64, 128, 256, 512, and 1024 respectively.

[0188] The decoder 53 can perform convolution processing on the input image through the decoding convolution layer, such as Figure 5 As shown in FIG. 1 , the decoder 53 includes 5 layers of decoders. Each upsampling layer doubles the size of the image. Each layer of the encoder includes: a decoding convolution layer 531, a guided filter layer 532, and an upsampling layer 533. Only the number of the first layer of decoders is marked in the figure, and the components of each layer are the same. The convolution kernel size is 3×3, the number of convolution kernels is 1024, 512, 256, 128, 64, and the number of output layers is 1024, 512, 256, 128, 64, respectively.

[0189] There are corresponding jump connections between the encoder and decoder. These jump connections allow the feature maps in the encoder to be reused, reduce redundant calculations in the decoder, and transmit high-level information directly to the bottom layer of the network to prevent the loss of high-quality details. Figure 5 As shown, only the skip connection 54 between the first layer encoder and the fifth layer decoder is identified in the figure, and the skip connection of each other layer is the same as the skip connection 54.

[0190] Figure 6 A schematic diagram of a training target purification model provided in an embodiment of the present application, such as Figure 6 As shown, where:

[0191] Input the fog image 61 to the target purification model 62;

[0192] Performing fog purification processing on the fog image 61 through the target purification model 62 to obtain a target purified image corresponding to the fog image;

[0193] Input the clean image 63 into the target purification model 62, and calculate the loss between the target purification image and the clean image 63;

[0194] The model parameters of the target purification model are updated at least once through the loss of the target purified image and the clean image 63 to obtain a trained target purification model 64 .

[0195] Figure 7 A schematic diagram of a data set provided in an embodiment of the present application, such as Figure 7 As shown, where:

[0196] The dataset includes images of the same scene in different situations, specifically: a clean image 71 with the lights off, an image 72 of a clean image with the lights on, a foggy image 73 with a visibility of 600 meters with the lights off, a foggy image 74 with a visibility of 600 meters with the lights on, a foggy image 75 with a visibility of 300 meters with the lights off, a foggy image 76 with a visibility of 300 meters with the lights on, a foggy image 77 with a visibility of 150 meters with the lights off, a foggy image 78 with a visibility of 150 meters with the lights on, a depth map 79 of the scene, and a semantic segmentation map 710 of the scene.

[0197] Figure 8 A schematic diagram of an image corresponding to different parameters provided in an embodiment of the present application, such as Figure 8 As shown, r represents the filter window radius, where:

[0198] Image 81 is the guided filtering effect of r=2 and Eps parameter=0.01, image 82 is the guided filtering effect of r=4 and Eps parameter=0.01, image 83 is the guided filtering effect of r=8 and Eps parameter=0.01, image 84 is the guided filtering effect of r=2 and Eps parameter=0.04, image 85 is the guided filtering effect of r=4 and Eps parameter=0.04, image 86 is the guided filtering effect of r=8 and Eps parameter=0.04, image 87 is the guided filtering effect of r=2 and Eps parameter=0.09, image 88 is the guided filtering effect of r=4 and Eps parameter=0.09, and image 89 is the guided filtering effect of r=8 and Eps parameter=0.09.

[0199] Here, the Eps parameter is related to the variance. The smaller the variance, the larger the Eps parameter, and the larger the variance, the smaller the Eps parameter. Figure 8 As shown in the figure, when r is consistent, the larger the Eps parameter is, the sharper the edge of the image is and the better the edge preservation effect is. Correspondingly, the smaller the Eps parameter is, the smoother the edge of the image is.

[0200] Fig. 9 A schematic diagram of downsampling provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, where:

[0201] Performing blur processing on the image 91 to obtain a first blurred image 92 corresponding to the image 91;

[0202] Downsampling the first blurred image 92 to obtain a first downsampled image 93 corresponding to the image 91;

[0203] Performing blur processing on the first down-sampled image 93 to obtain a second blurred image 94 corresponding to the first down-sampled image 93;

[0204] Downsampling the second blurred image 94 to obtain a second downsampled image 95 corresponding to the first downsampled image 93;

[0205] You can set different downsampling times according to different requirements, and perform downsampling processing by referring to the above process.

[0206] Based on the above embodiments, the present application also provides a model training device. Fig.10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the model training device 1000 includes a first acquisition unit 1001, an extraction unit 1002, a first determination unit 1003 and an update unit 1004, wherein:

[0207] The first acquisition unit 1001 is used to acquire sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-cleared image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map;

[0208] An extraction unit 1002 is used to extract features from a scene information graph corresponding to the first scene to obtain scene features;

[0209] The first determining unit 1003 is configured to perform a fog purification process on the first fog image based on scene features by using a target purification model to obtain a target purified image corresponding to the first fog image;

[0210] The updating unit 1004 is used to update the model parameters of the target purification model at least once based on the reference mist purification image and the target purification image to obtain a trained target purification model.

[0211] In some embodiments, the target purification model includes an encoder and a decoder; the first determination unit 1003 is also used to use the encoder to encode the scene features and the first fog image to obtain the target image features corresponding to the target purified image; and use the decoder to decode the target image features to obtain the target purified image corresponding to the first fog image.

[0212] In some embodiments, the encoder includes an encoding convolution layer and an attention mechanism layer; the first determination unit 1003 is also used to use the encoding convolution layer to perform convolution processing on the first fog image to obtain target image sub-features; and use the attention mechanism layer to fuse the scene features and the target image sub-features to obtain target image features corresponding to the target purified image.

[0213] In some embodiments, the decoder includes a decoding convolution layer and a guided filtering layer; the first determination unit 1003 is also used to use the decoding convolution layer to perform convolution processing on the target image features to obtain a target sub-image; and use the guided filtering layer to filter the target sub-image to obtain a target purified image corresponding to the first fog image.

[0214] In some embodiments, the guided filter layer includes floating-point relative precision parameters; the update unit 1004 is also used to update the encoding parameters of the encoder, the decoding parameters of the decoding convolution layer, and the floating-point relative precision parameters at least once based on the reference fog purification image and the target purification image to obtain a trained target purification model.

[0215] In some embodiments, the target purification model includes N layers of encoders, and the target image features corresponding to the target purified image include the output image features of the Nth layer encoder; the first determination unit 1003 is also used to use the first layer encoder to encode the input supplementary features of the first layer encoder and the first fog image to obtain the output image features of the first layer encoder; the input supplementary features of the first layer encoder are scene features; using the i-th layer encoder, the output image features of the i-1th layer encoder and the input supplementary features of the i-th layer encoder are encoded to obtain the output image features of the i-th layer encoder; wherein i is a positive integer greater than 1 and less than N+1, and the input supplementary features of the i-th layer encoder are obtained by pooling the input supplementary features of the i-1th layer encoder based on the input feature size corresponding to the i-th layer encoder.

[0216] In some embodiments, the target purification model includes a supplementary feature extraction module, and the scene features include depth features and / or semantic segmentation features; the extraction unit 1002 is also used to use the supplementary feature extraction module to perform feature extraction on the depth map when the scene information map includes a depth map, so as to obtain depth features corresponding to the depth map; when the scene information map includes a semantic segmentation map, the supplementary feature extraction module is used to perform feature extraction on the semantic segmentation map to obtain semantic segmentation features corresponding to the semantic segmentation map.

[0217] Based on the above embodiments, the present application also provides an image processing device. Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the image processing device 1100 includes a second acquisition unit 1101 and a second determination unit 1102, wherein:

[0218] A second acquisition unit 1101 is used to acquire a second fog image to be processed;

[0219] The second determination unit 1102 is used to use the target purification model to perform fog purification processing on the second fog image to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any model training method in claims 1 to 7.

[0220] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0221] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0222] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the computer program.

[0223] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. The computer-readable storage medium can be transient or non-transient.

[0224] The present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0225] It should be noted that Fig.12 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.12 As shown, the hardware entity of the electronic device 1200 includes: a processor 1201, a communication interface 1202 and a memory 1203, wherein:

[0226] The processor 1201 generally controls the overall operations of the electronic device 1200 .

[0227] The communication interface 1202 may enable the electronic device 1200 to communicate with other terminals or servers via a network.

[0228] The memory 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and each module in the electronic device 1200 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or RAM. Data can be transmitted between the processor 1201, the communication interface 1202, and the memory 1203 through the bus 1204.

[0229] Here, the electronic device may be a vehicle computer in a vehicle.

[0230] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0231] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0232] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0233] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0234] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0235] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0236] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks.

[0237] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0238] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-cleared image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map; Extracting features from the scene information graph corresponding to the first scene to obtain scene features; Using the target purification model, based on the scene features, the first fog image is subjected to fog purification processing to obtain a target purified image corresponding to the first fog image; Based on the reference mist purification image and the target purification image, the model parameters of the target purification model are updated at least once to obtain a trained target purification model.

2. The model training method according to claim 1, characterized in that: The target purification model includes an encoder and a decoder; The using the target purification model to perform fog purification processing on the first fog image based on the scene features to obtain a target purified image corresponding to the first fog image includes: Using the encoder, encoding the scene features and the first fog image to obtain target image features corresponding to the target purified image; The decoder is used to decode the target image features to obtain a target purified image corresponding to the first fog image.

3. The model training method according to claim 2, characterized in that: The encoder includes an encoding convolutional layer and an attention mechanism layer; The step of encoding the scene features and the first fog image using the encoder to obtain target image features corresponding to the target purified image includes: Using the coding convolution layer, convolution processing is performed on the first fog image to obtain target image sub-features; The scene features and the target image sub-features are fused using the attention mechanism layer to obtain target image features corresponding to the target purified image.

4. The model training method according to claim 2, characterized in that: The decoder comprises a decoding convolution layer and a guided filtering layer; The step of decoding the target image features by using the decoder to obtain a target purified image corresponding to the first fog image includes: Using the decoding convolution layer, convolution processing is performed on the target image features to obtain a target sub-image; The target sub-image is filtered using the guided filter layer to obtain a target purified image corresponding to the first fog image.

5. The model training method according to claim 4, characterized in that: The guided filter layer includes floating point relative precision parameters; The updating of the model parameters of the target purification model at least once based on the reference mist purification image and the target purification image to obtain a trained target purification model includes: Based on the reference fog-purified image and the target purified image, the encoding parameters of the encoder, the decoding parameters of the decoding convolutional layer, and the floating-point relative accuracy parameters are updated at least once to obtain the trained target purification model.

6. The model training method according to claim 2, characterized in that: The target purification model includes N layers of encoders, and the target image features corresponding to the target purified image include output image features of the Nth layer of encoders; The step of encoding the scene features and the first fog image using the encoder to obtain target image features corresponding to the target purified image includes: Using a first-layer encoder, encoding the input supplementary features of the first-layer encoder and the first fog image to obtain output image features of the first-layer encoder; the input supplementary features of the first-layer encoder are the scene features; Using the i-th layer encoder, the output image features of the i-1-th layer encoder and the input supplementary features of the i-th layer encoder are encoded to obtain the output image features of the i-th layer encoder; wherein i is a positive integer greater than 1 and less than N+1, and the input supplementary features of the i-th layer encoder are obtained by pooling the input supplementary features of the i-1-th layer encoder based on the input feature size corresponding to the i-th layer encoder.

7. The model training method according to any one of claims 2 to 6, characterized in that: The target purification model includes a supplementary feature extraction module, and the scene features include depth features and / or semantic segmentation features; The extracting features of the scene information graph corresponding to the first scene to obtain scene features includes: In a case where the scene information map includes the depth map, using the supplementary feature extraction module to perform feature extraction on the depth map to obtain the depth feature corresponding to the depth map; In the case where the scene information map includes the semantic segmentation map, the supplementary feature extraction module is used to perform feature extraction on the semantic segmentation map to obtain the semantic segmentation features corresponding to the semantic segmentation map.

8. An image processing method, characterized in that: The method comprises: Acquire a second fog image to be processed; The second fog image is subjected to fog purification processing by using a target purification model to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any one of the model training methods of claims 1 to 7.

9. A model training device, characterized in that: The device comprises: A first acquisition unit, configured to acquire sample image data; wherein the sample image data includes a first fog image corresponding to a first scene, a reference fog-purified image corresponding to the first scene, and a scene information map corresponding to the first scene; the scene information map includes a depth map and / or a semantic segmentation map; an extraction unit, configured to extract features from the scene information graph corresponding to the first scene to obtain scene features; a first determining unit, configured to perform a fog purification process on the first fog image based on the scene features by using a target purification model to obtain a target purified image corresponding to the first fog image; An updating unit is used to update the model parameters of the target purification model at least once based on the reference mist purification image and the target purification image to obtain a trained target purification model.

10. An image processing device, characterized in that: The device comprises: A second acquisition unit, used for acquiring a second fog image to be processed; The second determination unit is used to use a target purification model to perform fog purification on the second fog image to obtain a purified image corresponding to the second fog image; the target purification model is obtained by any model training method in claims 1 to 7.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

12. A vehicle, characterized in that: The vehicle includes the electronic device of claim 11.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method according to any one of claims 1 to 8 are implemented.

Images