An image processing method, a model training method, an apparatus, and a storage medium

Abstract

The present application provides an image processing method, a model training method, an apparatus, and a storage medium. The image processing method includes: preprocessing a real-time image to obtain a first processed image; the real-time image is an image including a first type of object and a second type of object, and the first processed image is an image excluding the first type of object; performing an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; the second processed image is an image excluding the second type of object; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between a plurality of channel features corresponding to the first processed image. The present application processes the real-time image collected by the imaging device, and through the correlation information between the image preprocessing and the channel features constructed by the image processing model, performs two-level processing on the real-time image to remove environmental noises with different granularities in the image, and can improve the image quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image processing method, a model training method, a device, and a storage medium. Background Art

[0002] The quality of images taken outdoors is affected by the environment, and there will be environmental noise in the captured images. For example, in rainy, foggy, or hazy weather, due to weather reasons, the lens of the imaging device will be stained with raindrops, fog, or dust. Coupled with the scattering of atmospheric light and scene light by raindrops, fog, or dust in the captured scene or target, the captured images often exhibit visible quality degradation such as color distortion, blurring, and low contrast. For such images, computer vision tasks such as object detection, object recognition, tracking, and segmentation become very complex.

[0003] In the field of autonomous driving, the autonomous driving system obtains information about the vehicle itself and the surrounding environment through the perception system, and analyzes, calculates, and processes the collected data information through the processor, so as to make decisions to control the execution system to realize actions such as vehicle acceleration, deceleration, and steering. The perception system can process the information of the vehicle's surrounding environment in a timely manner, including but not limited to detecting dynamic targets, semantically interpreting the environmental content, and classifying the working conditions of the driving environment. These tasks generally require further processing of the point cloud or image information transmitted back by the lidar or in-vehicle camera through a preset model to complete the interpretation of the vehicle's surrounding environment by the perception system and assist the driving system to make responses to ensure the safe driving of the vehicle. The input of the perception system depends on the results captured by the camera. Since the camera obtains information in a passive perception manner and does not emit and receive signals itself, the quality of the environment itself will directly affect the quality of the images captured by the camera. Therefore, the driving decisions made by the perception system based on the image processing results are greatly affected by the environment. How to improve the perception ability of the perception system in a harsh environment has become one of the urgent problems to be solved in current autonomous driving. Summary of the Invention

[0004] Aiming at the above problems of the prior art, the purpose of the present invention is to provide an image processing method, a model training method, a device, and a storage medium. The technical solution of the present application can be used in the field of autonomous driving to process the images captured by the imaging device, so as to improve the image quality and further improve the perception ability of the perception system.

[0005] According to one aspect of the present application, an image processing method is provided, and the method includes:

[0006] Preprocess the real-time image to obtain a first processed image; the real-time image is an image including a first type of object and a second type of object, and the first processed image is an image that does not include the first type of object;

[0007] Performing an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; the second processed image is an image that does not include second type objects; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the association information between a plurality of channel features corresponding to the first processed image.

[0008] In a possible implementation manner, the method further includes:

[0009] In the case of obtaining a second processed image corresponding to the current moment, performing an image fusion process on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain a third processed image corresponding to the current moment; the target image corresponding to the previous moment is the real-time image corresponding to the previous moment, or the third processed image corresponding to the previous moment.

[0010] In a possible implementation manner, the preprocessing the real-time image to obtain the first processed image includes:

[0011] Performing object recognition on the real-time image based on an object recognition model;

[0012] In the case of the existence of the first type objects, performing an object removal operation on the real-time image based on a first preset neural network to obtain the first processed image.

[0013] In a possible implementation manner, the performing an image extraction operation on the first processed image based on the image processing model to obtain the second processed image includes:

[0014] Performing global feature extraction on the first processed image to obtain a plurality of channel features corresponding to the first processed image;

[0015] Performing association information extraction on the plurality of channel features based on the channel feature extraction layer to obtain the association information between the plurality of channel features;

[0016] Obtaining the second processed image based on the association information.

[0017] In a possible implementation manner, the performing an image fusion process on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain a third processed image corresponding to the current moment includes:

[0018] Obtaining the target image corresponding to the previous moment;

[0019] Performing feature decomposition on the target image corresponding to the previous moment to obtain a feature vector and a transformation matrix of the target image corresponding to the previous moment in a preset spatial dimension;

[0020] Based on the transformation matrix and the preset spatial dimension, perform a spatial transformation on the second processed image to obtain a feature vector of the second processed image in the preset spatial dimension;

[0021] Based on the feature vector and transformation matrix of the target image corresponding to the previous moment, the feature vector of the second processed image, and the transformation matrix, calculate the eigenvalues of the target image corresponding to the previous moment and the second processed image respectively;

[0022] Perform a weighted average on the eigenvalues of the target image corresponding to the previous moment and the second processed image to obtain a fused eigenvalue;

[0023] Based on the transformation matrix and the preset spatial dimension, perform an inverse transformation on the fused eigenvalue to obtain a third processed image corresponding to the current moment.

[0024] In a possible implementation, when the real-time image is the first frame image collected by an image acquisition device, the method further includes:

[0025] Based on an object recognition model, perform object recognition on the real-time image;

[0026] In the case where the first type of object does not exist, use the real-time image as the target image corresponding to the current image.

[0027] In a possible implementation, after performing object recognition on the real-time image based on the object recognition model, the method further includes:

[0028] In the case where the first type of object exists, perform a feature extraction operation on the real-time image based on a first preset neural network to obtain the first processed image;

[0029] Based on an image processing model, perform an image extraction operation on the first processed image to obtain a second processed image;

[0030] Use the second processed image as the target image corresponding to the current image.

[0031] According to another aspect of the present application, there is provided a method for training a rain and fog noise removal model for training the above-mentioned image processing model, the method including:

[0032] Obtain training sample images and a basic image processing model; the training sample images include original images with rain and fog noise, environmental original images without rain and fog noise obtained by preprocessing the original images, and rain and fog noise images; the basic image processing model includes a global feature extraction layer and a channel feature extraction layer;

[0033] Input the original image into the global feature extraction layer to extract the global features of the training sample image, and obtain multiple channel features corresponding to the original image;

[0034] Take the multiple channel features as the input of the channel feature extraction layer, establish the mapping relationship between the multiple channel features, and separate the rain and fog noise features of the original image based on the mapping relationship to obtain the predicted noise map and the predicted environment map corresponding to the original image;

[0035] Calculate the first loss value between the predicted noise map and the rain and fog noise image and the second loss value between the predicted environment map and the original environment map based on a preset rule;

[0036] Perform backpropagation calculation on the basic image processing model based on the first loss value and the second loss value to obtain the basic image processing model with updated parameters;

[0037] Take the basic image processing model with updated model parameters as the current basic image processing model, and return to execute the steps of obtaining the training sample image and the basic image processing model until the preset training completion condition is reached to obtain the image processing model.

[0038] According to another aspect of the present application, there is provided an image processing device, the device includes:

[0039] The first image processing module is used to preprocess the real-time image to obtain a first processed image; the real-time image is an image including a first type of object and a second type of object, and the first processed image is an image that does not include the first type of object;

[0040] The second image processing module is used to perform an image extraction operation on the first processed image based on the image processing model to obtain a second processed image; the second processed image is an image that does not include the second type of object; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image.

[0041] According to another aspect of the present application, there is provided a storage medium, in which at least one instruction or at least one program segment is stored, and the at least one instruction or the at least one program segment is loaded and executed by a processor to implement any of the above feasible image processing methods or to implement the above image processing model training method.

[0042] The image processing method, model training method, device and storage medium provided by the present application have the following beneficial effects:

[0043] The image processing method of the present application preprocesses a real-time image to obtain a first processed image; performs an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; the second processed image is an image that does not include second-type objects; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image. Using the technical solution of the present application, the real-time image collected by the imaging device can be processed, and through the image preprocessing and the correlation information between the channel features constructed by the image processing model, the real-time image can be processed at two levels, the environmental noise with different granularities in the image can be removed, and thus the image quality can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings according to these drawings without creative efforts.

[0045] Figure 1 A schematic diagram of an application environment provided by an embodiment of the present application;

[0046] Figure 2 A schematic flowchart of an image processing method provided by an embodiment of the present application;

[0047] Figure 3 A schematic flowchart of another image processing method provided by an embodiment of the present application;

[0048] Figure 4 A schematic flowchart of another image processing method provided by an embodiment of the present application;

[0049] Figure 5 A schematic block diagram of an image processing model provided by an embodiment of the present application;

[0050] Figure 6 A schematic flowchart of a rain and fog removal model training method provided by an embodiment of the present application;

[0051] Figure 7 A schematic block diagram of an image processing device provided by an embodiment of the present application;

[0052] Figure 8 A schematic block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0055] Please refer to Figure 1 , Figure 1 which is a schematic diagram of an application environment provided by an embodiment of the present application. As Figure 1 shown, the vehicle autonomous driving system includes a perception system, a decision-making system, and an execution system. The autonomous driving system obtains the vehicle's own information and surrounding environment information through the perception system, and transmits the collected data information to the decision-making system. The decision-making system analyzes, calculates, and processes the collected data information through a processor to make a decision; the decision-making system controls the corresponding execution units of the execution system by sending control information to perform actions such as accelerating, decelerating, and steering the vehicle. Among them, the perception system usually includes a camera device, a radar device, or a positioning device, etc.; the execution system usually includes an electronic braking unit, an electronic drive unit, and an electric steering unit, etc.

[0056] The camera device configured in the vehicle can be a monocular camera, a binocular camera, or a surround-view camera. The real-time image collected by the camera is transmitted to the decision-making system after being denoised by the image processing module. Through the analysis and processing of the image data information by the decision-making system, the electronic braking unit, the electronic drive unit, or the electric steering unit of the execution system is controlled to realize actions such as accelerating, decelerating, and steering the vehicle. Among them, the analysis and processing of the image by the decision-making system include, but are not limited to, one or more of object detection, semantic analysis, and working condition classification.

[0057] However, in practical applications, since the input of the perception system depends on the results captured by the camera, and the way the camera obtains information is passive perception. The camera itself does not emit and receive transmitted signals. Therefore, the quality of the environment itself will directly affect the quality of the images captured by the camera. As a result, the driving decisions made by the autonomous driving system based on the image processing results are greatly affected by the environment. The image processing method provided in the embodiments of the present application can be applied to the perception system to process the real-time images captured by the imaging device. By using the technical solution of the present application, through the correlation information between the channel features of image preprocessing and the construction of the image processing model, the real-time images are processed at two levels, which can remove environmental noises with different granularities in the images, and thus improve the image quality. The specific description is as follows:

[0058] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of an image processing method provided in the embodiments of the present application, including:

[0059] S210, preprocess the real-time image to obtain a first processed image; the real-time image is an image including a first type of object and a second type of object, and the first processed image is an image that does not include the first type of object;

[0060] In the embodiments of the present application, the image processing method is used to process the image frames in the video stream captured by the imaging device of the vehicle. The real-time image is the image at the current moment captured by the imaging device in real time, and the image frame before the image at the current moment is the image at the previous moment. The first type of object and the second type of object may be noises with different granularities of the same category; for example, in a rain and fog scenario, the first type of object may be raindrops, and the second type of object may be rain and fog noises embedded in the original environmental image; in a sand and dust scenario, the first type of object may be dust particles, and the second type of object may be dust and fog embedded in the original environmental image, etc. The preprocessing process is to remove the first type of object with a relatively large and easily recognizable granularity.

[0061] It should be noted that the first type of object and the second type of object existing in the image may be caused by the occlusions on the lens, or may be caused by environmental factors within the viewing angle of the imaging device.

[0062] S220, perform an image extraction operation on the first processed image based on the image processing model to obtain a second processed image; the second processed image is an image that does not include the second type of object; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image.

[0063] In the embodiments of the present application, the image processing model is used to remove the noise with relatively small granularity embedded in the original environmental image in the first processed image. This type of noise has a large coverage range, a high degree of fusion with the scene image, is not easy to identify, and is very difficult to remove. Therefore, in this embodiment, the second type of object in the image is regarded as a noise map superimposed on the original environmental image without noise.

[0064] In the embodiments of the present application, by constructing the channel feature extraction layer of the image processing model and performing high-dimensional fusion feature extraction on the channel features based on the deep learning of the neural network, the superposition relationship of the noise between different channel features is obtained, as well as how part of the distortion superimposed on the original environmental image is generated, and the association information between different channel features in the first processed image is constructed. Thus, the noise map superimposed on the original environmental image can be separated to obtain the second processed image that does not include the second type of object. Among them, the channel features may include, but are not limited to, abstract features such as color, shape, and texture extracted from the image, and different channel features contain noise factors with different weights.

[0065] In one embodiment, the channel feature extraction layer of the image processing model may be a fusion attention network between different channel features constructed based on the attention mechanism of the neural network of Transformer. The fusion attention network learns the superposition mechanism of the noise factors in different channel features to obtain an image processing model capable of separating the original environmental image and the noise map.

[0066] The image processing model of the embodiments of the present application can be used for, but not limited to, removing rain and fog noise or dust and fog noise with relatively small granularity embedded in the original environmental image.

[0067] Please refer to Figure 3 , in one embodiment, after step S220, the method further includes:

[0068] S330, in the case of obtaining the second processed image corresponding to the current moment, performing image fusion processing on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain the third processed image corresponding to the current moment; the target image corresponding to the previous moment is the real-time image corresponding to the previous moment, or the third processed image corresponding to the previous moment.

[0069] In practical applications, when there is no object of the first type in the real-time image of the previous moment, it is considered that there is no noise factor in the real-time image of the previous moment, and the real-time image of the previous moment is directly used as the target image corresponding to the previous moment. When there is an object of the first type in the real-time image of the previous moment, it is considered that the real-time image of the current moment includes the above-mentioned object of the first type and the object of the second type. After processing the real-time image of the previous moment by using the above steps S210 and S220, a second processed image corresponding to the real-time image of the previous moment is obtained; then, through step S330, the second processed image is subjected to image fusion processing with a third processed image corresponding to the real-time image before the real-time image of the previous moment, so as to obtain a third processed image corresponding to the previous moment; and the third processed image is used as the target image corresponding to the previous moment. In the embodiment of the present application, by fusing the target image corresponding to the previous moment in the process of processing the real-time image of the current moment, it is possible to take into account the situations where the real-time image of the previous moment includes and does not include an object of the first type, and obtain a third processed image corresponding to the current moment.

[0070] Particularly, when there is an object of the first type in the real-time image of the current moment, by performing image fusion processing on the second processed image corresponding to the current moment and the target image corresponding to the previous moment, it is possible to utilize the correlation relationship in the video image time series to correct the real-time image of the current moment, reduce the interference of noise in the image of the current moment, further improve the image quality, and thus improve the perception ability of the decision-making system for the image content.

[0071] Please refer to Figure 4 , in a specific embodiment, step S210 may include:

[0072] S2110, performing object recognition on the real-time image based on an object recognition model;

[0073] S2120, when there is an object of the first type, performing an object removal operation on the real-time image based on a first preset neural network to obtain a first processed image.

[0074] In the embodiment of the present application, the object recognition model may be a convolutional neural network capable of recognizing an object of the first type, which is trained by an original image containing an object of the first type and an original image with the object of the first type removed.

[0075] In the embodiment of the present application, the first preset neural network may be an image processing model pre-trained based on a generative adversarial network (GAN). Through the adversarial training between two models (a generative network G and a discriminative network D), the model can finally restore the image containing an object of the first type to a first processed image that does not include an object of the first type.

[0076] In one embodiment, an object recognition model is used to identify raindrops. The first preset neural network is an image processing model pre-trained based on a generative adversarial network (GAN) and is used to remove raindrops in a real-time image to obtain the original environmental image of the real-time image.

[0077] Further, step S220 may include:

[0078] S2210, perform global feature extraction on the first processed image to obtain multiple channel features corresponding to the first processed image;

[0079] In the embodiments of the present application, common network structures for extracting image features can be selected. For example, convolutional neural network structures such as resnet and mobilenet. Through global feature extraction, shallow abstract features such as color, shape, and texture in the image are extracted to obtain corresponding channel features.

[0080] S2220, perform associated information extraction on the multiple channel features based on a channel feature extraction layer to obtain the associated information between the multiple channel features;

[0081] The associated information between the multiple channel features can be the hierarchical superposition relationship of the second type of object in different channel features learned by the Transformer attention network based on establishing connections among multiple channel features such as color, shape, and texture in the image or space obtained by global feature extraction. For example, the second type of object with different weights in different channel features, and the superposition relationship of the weights of the second type of object in different channel features is deeply learned through the multi-head attention structure of the Transformer.

[0082] S2230, obtain a second processed image based on the associated information.

[0083] In this step, the first processed image is restored using the above-mentioned associated information between the multiple channel features to obtain a second processed image that does not include the second type of object.

[0084] It should be noted that based on the above processing process, in the embodiments of the present application, a network structure based on channel features for extracting the fusion features of each feature in different channel features is constructed to obtain the associated information between the multiple channel features. This associated information includes not only shallow abstract image features but also the correlation between the levels of multi-scale features extracted, thereby improving the attention degree to important information in the image. Thus, in the embodiments of the present application, a second processed image that is more in line with the real scene can be obtained.

[0085] In one embodiment, the second type of object is rain and fog noise.

[0086] In another embodiment, the second type of object is dust and fog noise.

[0087] Further, step S330 may include:

[0088] S3310, obtaining the target image corresponding to the previous moment;

[0089] It should be noted that when the image of the previous moment does not include the first type of object, the target image corresponding to the previous moment is the real-time image of the previous moment. When the image of the previous moment includes the first type of object, the target image corresponding to the previous moment is the third processed image corresponding to the previous moment.

[0090] S3320, performing feature decomposition on the target image corresponding to the previous moment to obtain the feature vector and transformation matrix of the target image corresponding to the previous moment in the preset spatial dimension;

[0091] S3330, performing spatial transformation on the second processed image based on the transformation matrix and the preset spatial dimension to obtain the feature vector of the second processed image in the preset spatial dimension;

[0092] S3340, respectively calculating the eigenvalues of the target image corresponding to the previous moment and the second processed image based on the feature vector and transformation matrix of the target image corresponding to the previous moment, and the feature vector and transformation matrix of the second processed image;

[0093] S3350, performing weighted averaging on the eigenvalues of the target image corresponding to the previous moment and the second processed image to obtain a fused eigenvalue;

[0094] S3360, performing inverse transformation on the fused eigenvalue based on the transformation matrix and the preset spatial dimension to obtain the third processed image corresponding to the current moment.

[0095] Exemplarily, in the embodiments of the present application, PCA (Principal Component Analysis) may be used for feature decomposition.

[0096] PCA feature decomposition is a method for dimensionality reduction of information (such as image information, feature information, etc., which can be represented in multiple dimensions). The main idea is to find several specific variables to replace the original numerous variables, so that these specific variables can represent the information volume of the original variables as much as possible and are not related to each other. This method can also represent information in the feature space, combine the eigenvalues, and then perform an inverse transformation on the feature space to return to the original information space to achieve information fusion. The information of the image after PCA transformation is mainly concentrated in the first few principal component components. Discard the principal component components with small information volume in the transformation domain, transform the original large amount of hyperspectral data into a small number of components, while reducing the data dimensionality, maximizing the preservation of the information of the original data.

[0097] In the embodiment of the present application, by fusing the target image corresponding to the previous moment, which is a noise-free image, with the second processed image at the current moment through the above steps, the image features of the noise-free second processed image obtained at the current moment can be fused with the noise-free image obtained at the previous moment, so that the real-time image at the previous moment and the real-time image at the current moment can be used to further correct the image quality in terms of temporal correlation. In this way, the advantages in time can be obtained and utilized as much as possible, and the interference of noise on the original image can be reduced at the first time.

[0098] In practical applications, for the first frame image collected by the imaging device, since there is no image at the previous moment, there is no target image corresponding to the previous moment. Therefore, in the embodiment of the present application, a further solution is provided.

[0099] In one embodiment, when the real-time image is the first frame image collected by the image acquisition device, the method may further include:

[0100] S410, perform object recognition on the real-time image based on the object recognition model;

[0101] In the embodiment of the present application, the object recognition model may be a convolutional neural network trained by the original image containing the first type of object and the original image with the first type of object removed, which can recognize the first type of object.

[0102] S420, when there is no first type of object, use the real-time image as the target image corresponding to the current image.

[0103] Further, after step S410, the method may further include:

[0104] S430, when there is a first type of object, perform feature extraction operation on the real-time image based on the first preset neural network to obtain the first processed image;

[0105] S440. Perform an image extraction operation on the first processed image based on the image processing model to obtain a second processed image;

[0106] S450. Use the second processed image as the target image corresponding to the current image.

[0107] In the above method of the embodiment of the present application, corresponding target images are respectively set based on whether there is a first type of object in the first frame image, which solves the problem that there is no third processed image or real-time image corresponding to the image at the previous moment for fusion before the first frame image, and provides a general method capable of processing all image frames collected by a vehicle camera device.

[0108] The embodiment of the present application further provides a training method for an image processing model, which can be used to train the image processing model in the above method embodiment.

[0109] In a specific embodiment, the image processing model is used to remove rain and fog noise from the original environmental image. Please refer to Figure 6 , and the embodiment of the present application provides a training method for a rain and fog removal model. The method may include:

[0110] S110. Obtain training sample images and a basic image processing model; the training sample images include original images with rain and fog noise, environmental original images without rain and fog noise obtained by preprocessing the original images, and rain and fog noise images; the basic image processing model includes a global feature extraction layer and a channel feature extraction layer;

[0111] Please refer to Figure 5 , the figure shows the basic composition of a basic image processing model in an embodiment, including a global feature extraction layer and a channel feature extraction layer. Among them, the global feature extraction layer may include multiple layers of convolutional neural networks to extract global features in the image from multiple angles. The input of the channel feature extraction layer is the output of the global feature extraction layer, and the attention network of the Transformer is used to establish connections for multiple channel features input by the global feature extraction to capture the hierarchical superposition relationship of the second type of object in different channel features.

[0112] The original image with rain and fog noise may be the original image collected by a camera device, and the environmental original image without rain and fog noise and the rain and fog noise image may be obtained by separating the features of the collected original image. For example, use Photoshop or other professional drawing software to separate the image features;

[0113] In some other embodiments, the training sample images may also be obtained by sequentially photographing the environmental original image, the rain and fog noise image, and the image after superimposing the environmental original image and the rain and fog noise.

[0114] S120. Input the original image into the global feature extraction layer to extract the global features of the training sample image, and obtain multiple channel features corresponding to the original image.

[0115] Exemplarily, common network structures for extracting image features can be selected, such as convolutional neural network structures like resnet and mobilenet. Through global feature extraction, shallow abstract features such as color, shape, and texture in the image are extracted to obtain corresponding channel features.

[0116] S130. Use the multiple channel features as the input of the channel feature extraction layer, establish the mapping relationship between the multiple channel features, and separate the rain and fog noise features of the original image based on the mapping relationship to obtain the predicted noise map and the predicted environment map corresponding to the original image.

[0117] Through the multi-head attention structure of Transformer, the superposition relationship of the weights of the second type of object in different channel features is deeply learned. The constructed fusion attention network between different channel features learns the superposition mechanism of noise factors in different channel features, and uses the learned superposition mechanism to predict the original environment map and the noise map of the original image to obtain the predicted original environment map and the noise map.

[0118] S140. Calculate the first loss value between the predicted noise map and the rain and fog noise image and the second loss value between the predicted environment map and the original environment map based on the preset rules.

[0119] In the embodiments of the present application, cross-entropy loss can be used to calculate the loss value. Those skilled in the art can also select other loss functions for calculation according to needs.

[0120] S150. Perform backpropagation calculation on the basic image processing model based on the first loss value and the second loss value to obtain the basic image processing model with updated parameters.

[0121] Through the above steps, the first parameter update of the basic image model is completed.

[0122] S160. Use the basic image processing model with updated model parameters as the current basic image processing model, and return to execute the steps of obtaining the training sample image and the basic image processing model until the preset training completion condition is reached to obtain the image processing model.

[0123] Repeat the above steps multiple times until the preset training times are reached, or the loss value meets the set threshold to end the model training process.

[0124] It should be noted that the above training sample images can be a pre-prepared training image set, which includes multiple groups of training sample images, and one group of training sample images is used for each training. The above training sample images can also be obtained multiple times during the training process. When necessary, target training sample images are generated according to the training effect to reduce the number of trainings and improve the training efficiency.

[0125] In the image processing model training method according to the embodiment of the present application, the global feature extraction layer and the channel feature extraction layer of the basic image processing model are trained through training sample images, and the superposition relationship of noises in the image is deeply learned, so that noises with smaller granularity in the image can be removed, and an environmental original image more in line with the real scene can be obtained. The image processing model obtained through the above steps can be used in the above image processing method to further improve the image processing efficiency of the image processing method of the present application.

[0126] In another specific embodiment, the image processing model is used to remove the dust and fog background in the environmental original image. The difference between this training process and the above rain and fog noise removal model lies in the training sample images. In the training method of the dust and fog removal model, the training sample images include the original image with dust and fog noise, the environmental original image without dust and fog noise and the dust and fog noise image obtained by preprocessing the original image; the remaining processes can all refer to the above steps S110 to S160. This will not be elaborated in the embodiments of the present application.

[0127] The embodiment of the present application further provides an image processing device. Please refer to Figure 7 , the device may include:

[0128] The first image processing module 10 is used to preprocess the real-time image to obtain a first processed image; the real-time image is an image including a first type of object and a second type of object, and the first processed image is an image without the first type of object;

[0129] The second image processing module 20 is used to perform an image extraction operation on the first processed image based on the image processing model to obtain a second processed image; the second processed image is an image without the second type of object; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image.

[0130] In one embodiment, the device may further include:

[0131] A third image processing module, configured to perform image fusion processing on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain a third processed image corresponding to the current moment when the second processed image corresponding to the current moment is obtained; the target image corresponding to the previous moment is the real-time image corresponding to the previous moment or the third processed image corresponding to the previous moment.

[0132] In one embodiment, the first image processing module 10 may include:

[0133] An object recognition unit, configured to perform object recognition on a real-time image based on an object recognition model;

[0134] An object removal unit, configured to perform an object removal operation on the real-time image based on a first preset neural network to obtain a first processed image when there is a first type of object.

[0135] In one embodiment, the second image processing module 20 may include:

[0136] A global feature extraction unit, configured to perform global feature extraction on the first processed image to obtain multiple channel features corresponding to the first processed image;

[0137] A channel feature extraction unit, configured to extract correlation information between multiple channel features based on a channel feature extraction layer to obtain correlation information between the multiple channel features;

[0138] A second image processing unit, configured to obtain a second processed image based on the correlation information.

[0139] In one embodiment, the third image processing module may include:

[0140] A target image acquisition unit, configured to acquire a target image corresponding to the previous moment;

[0141] A first feature decomposition unit, configured to perform feature decomposition on the target image corresponding to the previous moment to obtain a feature vector and a transformation matrix of the target image corresponding to the previous moment in a preset spatial dimension;

[0142] A second feature decomposition unit, configured to perform a spatial transformation on the second processed image based on the transformation matrix and the preset spatial dimension to obtain a feature vector of the second processed image in the preset spatial dimension;

[0143] A calculation unit, configured to calculate the eigenvalues of the target image corresponding to the previous moment and the second processed image respectively based on the feature vector and the transformation matrix of the target image corresponding to the previous moment, and the feature vector and the transformation matrix of the second processed image;

[0144] The fusion processing unit is configured to perform weighted averaging on the feature values of the target image and the second processed image corresponding to the previous moment to obtain a fused feature value;

[0145] The third image processing unit is configured to perform inverse transformation on the fused feature value based on the transformation matrix and the preset spatial dimension to obtain a third processed image corresponding to the current moment.

[0146] For the specific limitations of the image processing device, reference can be made to the limitations of the image processing method in the above text respectively, which will not be elaborated here. Each module of the above image processing method and image processing model training method can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above modules.

[0147] Both the image processing method and the image processing model training method of the present application can be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for enabling a processor to implement various aspects of the present application are uploaded.

[0148] The embodiment of the present application further provides an electronic device, which includes a processor and a memory. At least one instruction or at least one program segment is stored in the memory, and the at least one instruction or the at least one program segment is loaded and executed by the processor to implement the image processing method and the image processing model training method provided in the above control method embodiment. Figure 8 is a hardware structure block diagram of an electronic device provided by an embodiment of the present application. As Figure 8As shown, the electronic device 800 can vary significantly due to different configurations or performances. It may include one or more central processing units (CPUs) 810 (the processor 810 may include, but is not limited to, processing devices such as a microprocessor MCU or a field-programmable gate array FPGA), a memory 830 for storing data, and one or more storage media 820 for storing application programs 823 or data 822 (such as one or more mass storage devices). Among them, the memory 830 and the storage media 820 can be transient storage or persistent storage. The program stored in the storage media 820 may include one or more modules, and each module may include a series of instruction operations on the electronic device. Further, the central processor 810 can be configured to communicate with the storage media 820 and execute a series of instruction operations in the storage media 820 on the electronic device 800. The electronic device 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0149] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by the communication provider of the electronic device 800. In one example, the input / output interface 840 includes a network interface controller (NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the input / output interface 840 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0150] Those of ordinary skill in the art can understand that Figure 8 the structure shown is only schematic and does not limit the structure of the above electronic device. For example, the electronic device 800 may also include more or fewer components than Figure 8 shown, or have a different configuration from Figure 8 shown.

[0151] The memory can be used to store software programs and modules. By running the software programs and modules stored in the memory, the processor can execute various functional applications and data processing. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0152] The message processing method provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal, a server or a similar computing device.

[0153] The embodiments of the present application also provide a computer-readable storage medium. The storage medium can be disposed in the server to store at least one instruction or at least one segment of program related to implementing the vehicle information processing method or the vehicle remote control method in the method embodiments. The at least one instruction or the at least one segment of program is loaded and executed by the processor to implement the above-mentioned image processing method or the image processing model training method.

[0154] Optionally, in this embodiment, the above storage medium can be located in at least one of multiple network servers in a computer network. Optionally, in this embodiment, the above storage medium can include, but is not limited to: USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks or optical discs and other various media that can store program codes.

[0155] It should be noted that: the above sequence of the embodiments of the present application is only for description and does not represent the superiority or inferiority of the embodiments. And the above specific embodiments of the present application have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0157] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware or by a program instructing the relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, or the like.

[0158] The above are only the preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

[0159] The above are only the preferred embodiments of this invention and are not intended to limit this invention. For those skilled in the art, it is obvious that this invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of this invention, this invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of this invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in this invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0160] In addition, it should be understood that although this specification is described according to the embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An image processing method, characterized in that, Including: Preprocessing a real-time image to obtain a first processed image; The real-time image is an image including a first type of object and a second type of object, the first processed image is an image without the first type of object, and the first type of object and the second type of object are noises of different granularities of the same category; Performing an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; the second processed image is an image without the second type of object; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image, and the correlation information includes the superposition relationship of the weights of the second type of object in different channel features; The performing an image extraction operation on the first processed image based on the image processing model to obtain a second processed image includes: Performing global feature extraction on the first processed image to obtain multiple channel features corresponding to the first processed image; Extracting correlation information between the multiple channel features based on the channel feature extraction layer to obtain the correlation information between the multiple channel features; Obtaining the second processed image based on the correlation information.

2. The method according to claim 1, wherein The method further includes: In the case of obtaining a second processed image corresponding to the current moment, performing image fusion processing on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain a third processed image corresponding to the current moment; the target image corresponding to the previous moment is the real-time image corresponding to the previous moment, or the third processed image corresponding to the previous moment.

3. The method according to claim 1, wherein The preprocessing the real-time image to obtain a first processed image includes: Performing object recognition on the real-time image based on an object recognition model; In the case of the existence of the first type of object, performing an object removal operation on the real-time image based on a first preset neural network to obtain the first processed image.

4. The method according to claim 2, characterized in that The performing image fusion processing on the second processed image corresponding to the current moment and the target image corresponding to the previous moment to obtain a third processed image corresponding to the current moment includes: Obtaining the target image corresponding to the previous moment; Performing feature decomposition on the target image corresponding to the previous moment to obtain a feature vector and a transformation matrix of the target image corresponding to the previous moment in a preset spatial dimension; Performing a spatial transformation on the second processed image based on the transformation matrix and the preset spatial dimension to obtain a feature vector of the second processed image in the preset spatial dimension; Calculating the eigenvalues of the target image corresponding to the previous moment and the second processed image respectively based on the feature vector and the transformation matrix of the target image corresponding to the previous moment, the feature vector of the second processed image, and the transformation matrix; Performing weighted averaging on the eigenvalues of the target image corresponding to the previous moment and the second processed image to obtain a fusion eigenvalue; Performing an inverse transformation on the fusion eigenvalue based on the transformation matrix and the preset spatial dimension to obtain a third processed image corresponding to the current moment.

5. The method according to claim 2, wherein When the real-time image is the first frame image acquired by the image acquisition device, the method further includes: Performing object recognition on the real-time image based on an object recognition model; When there is no object of the first type, using the real-time image as the target image corresponding to the current image.

6. The method according to claim 5, wherein After performing object recognition on the real-time image based on the object recognition model, the method further includes: When there is an object of the first type, performing a feature extraction operation on the real-time image based on a first preset neural network to obtain the first processed image; Performing an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; Using the second processed image as the target image corresponding to the current image.

7. An image processing model training method for training the image processing model as described in claim 1, including: Obtaining training sample images and a basic image processing model; The training sample images include original images with rain and fog noise, and environment original images and rain and fog noise images obtained by preprocessing the original images; the basic image processing model includes a global feature extraction layer and a channel feature extraction layer; Inputting the original images into the global feature extraction layer to perform global feature extraction on the training sample images to obtain multiple channel features corresponding to the original images; Using the multiple channel features as the input of the channel feature extraction layer, establishing a mapping relationship between the multiple channel features, and separating the rain and fog noise features of the original images based on the mapping relationship to obtain a predicted noise map and a predicted environment map corresponding to the original images; Calculating a first loss value between the predicted noise map and the rain and fog noise image and a second loss value between the predicted environment map and the environment original image based on a preset rule; Performing backpropagation calculation on the basic image processing model based on the first loss value and the second loss value to obtain an updated parameter basic image processing model; Using the updated parameter basic image processing model as the current basic image processing model, and returning to the step of obtaining training sample images and a basic image processing model to execute until a preset training completion condition is reached, thereby obtaining the image processing model.

8. An image processing apparatus, characterized in that, Including: A first image processing module for preprocessing a real-time image to obtain a first processed image; The real-time image is an image including an object of the first type and an object of the second type, the first processed image is an image that does not include an object of the first type, and the object of the first type and the object of the second type are different granularity noises of the same category; A second image processing module for performing an image extraction operation on the first processed image based on an image processing model to obtain a second processed image; the second processed image is an image that does not include an object of the second type; the image processing model includes a channel feature extraction layer, and the channel feature extraction layer is used to extract the correlation information between multiple channel features corresponding to the first processed image, and the correlation information includes the superposition relationship of the weights of the object of the second type in different channel features; The second image processing module includes: extracting global features from the first processed image to obtain a plurality of channel features corresponding to the first processed image; extracting correlation information between the plurality of channel features based on the channel feature extraction layer to obtain the correlation information between the plurality of channel features; and obtaining the second processed image based on the correlation information.

9. A storage medium, characterized in that, At least one instruction or at least one program segment is stored in the storage medium, and the at least one instruction or the at least one program segment is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 6, or to implement the image processing model training method according to claim 7.

Images