Ultra-lightweight image defogging and recognition network model, image defogging and recognition method

Abstract

The present application discloses an ultra-lightweight image defogging and recognition network model, through which image defogging and recognition are achieved, including a bidirectional GAN ​​network model and a target detection network model connected in sequence. The bidirectional GAN ​​network model defogs the foggy image and outputs a clear image to the target detection network model for feature recognition processing. The target detection network model undergoes row pruning and retraining. The training process is to train the original images of the training set multiple times. Before each training, the original images are downsampled by a preset multiple. After each training, the scaling coefficients of the batch normalization layer are sorted and compared, and the previous layer of convolution kernels corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning. The above-mentioned target detection network model is further pruned on the basis of the current micro-recognition model, which greatly reduces the scale of the ultra-lightweight image defogging and recognition network model, and can be deployed on the end-side platform with limited computing power and power consumption resources.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an ultra-lightweight image defogging and recognition network model, and an image defogging and recognition method. Background Art

[0002] Haze weather will reduce the clarity of the images collected by the camera, making it difficult for the computer to recognize the features of objects in the image. Therefore, before recognizing the image, the image must be dehazed. Currently, both dehazing and recognition of images can be achieved through neural network models.

[0003] For dehazing, the commonly used neural network model is Generative Adversarial Networks (GAN), which uses an unsupervised learning method to learn the mapping relationship between foggy images and clear images through a large amount of training data, and then achieves dehazing of foggy images. For recognition processing, the commonly used neural network model is the YOLO (You Only Look Once) target detection model, which divides the input image into S*S grids, each grid is responsible for detecting objects whose center falls inside itself, and directly regresses the target's regression box and predicted category at multiple positions in the input image to achieve recognition of image object features.

[0004] The current YOLO target detection model is relatively large. The neural network model obtained by applying it to image dehazing and recognition is not suitable for deployment on end-side platforms with limited computing power and power consumption resources, such as mobile phones or autonomous driving-related devices. Summary of the invention

[0005] In order to solve the problem that the current YOLO target detection model is relatively large and the neural network model obtained when it is applied to the image defogging and recognition process is not suitable for deployment on the terminal platform with limited computing power and power consumption resources, the present application discloses an ultra-lightweight image defogging and recognition network model and an image defogging and recognition method through the following embodiments.

[0006] The first aspect of the present application discloses an ultra-lightweight image defogging and recognition network model, including: a bidirectional GAN ​​network model and a target detection network model connected in sequence;

[0007] The bidirectional GAN ​​network model is used to process the input image to be defogged and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image;

[0008] The target detection network model is a Yolo-Tiny-S network model that has been retrained through row pruning. During the row pruning retraining process, the original images in the target detection network model training set are trained multiple times. Before each training, the original images are downsampled by a preset multiple. After each training, the scaling coefficients of the batch normalization layer are sorted and compared, and the convolution kernels of the previous layer corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning.

[0009] The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit connected in sequence. The clear image is input into the target detection network model through the front extraction unit. The feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit connected in sequence. The output ends of the front extraction unit and the middle extraction unit are both connected to the input end of the feature fusion splicing unit.

[0010] Optionally, the front extraction unit includes a DBL-S combination subunit and a plurality of MDBL-S combination subunits connected in sequence;

[0011] The middle extraction unit includes a plurality of the MDBL-S combination subunits and one DBL-S combination subunit connected in sequence;

[0012] The first rear extraction unit and the second rear extraction unit each include a DBL-S combination subunit and a convolution subunit connected in sequence;

[0013] The feature fusion and splicing unit includes an upsampling subunit and a feature splicing subunit, the input end of the upsampling subunit is connected to the output end of the middle extraction unit, and the output end of the upsampling subunit and the output end of the front extraction unit are connected to the input end of the feature splicing subunit;

[0014] The DBL-S combined subunit consists of a dark network convolutional layer, a batch normalization layer, and a leakage corrected linear layer connected in sequence;

[0015] The MDBL-S combination subunit is composed of a maximum pooling layer and the DBL-S combination subunit connected in sequence.

[0016] Optionally, the bidirectional GAN ​​network model includes an input module, a generation module and a discrimination module;

[0017] The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator;

[0018] The clear image input port, the first generation unit and the first discriminator are connected in sequence to extract and reconstruct features for the clear image, and the fog image input port, the second generation unit and the second discriminator are connected in sequence to extract and reconstruct features for the fog image;

[0019] The first generation unit includes a first encoder, a shared latent space, and a first decoder connected in sequence, and the second generation unit includes a second encoder, the shared latent space, and a second decoder connected in sequence;

[0020] The shared latent space is used to store high-level features, and output the high-level features to the first decoder and the second decoder, wherein the high-level features include the high-level features extracted by the first encoder for the clear image and the high-level features extracted by the second encoder for the foggy image;

[0021] The training set and the validation set of the bidirectional GAN ​​network model both include paired clear images and foggy images.

[0022] Optionally, the first encoder and the second encoder respectively include a first convolution block, a second convolution block and a first coupling residual block connected in sequence;

[0023] The first decoder and the second decoder respectively include a second coupled residual block, a first stride convolution block and a second stride convolution block connected in sequence;

[0024] The output terminal of the first coupled residual block is connected to the input terminal of the shared latent space, and the output terminal of the shared latent space is connected to the input terminal of the second coupled residual block;

[0025] The output end of the first convolution block is jump-connected to the input end of the second stride convolution block;

[0026] The output end of the second convolution block is jump-connected to the input end of the first-step long convolution block.

[0027] Optionally, the first coupled residual block and the second coupled residual block are respectively composed of a plurality of cascaded sub-residual blocks;

[0028] The sub-residual block at any level includes a first convolutional layer, an activation function layer and a second convolutional layer connected in sequence, which are used to process the output results of the sub-residual block of the previous level and the output results of the sub-residual block of the previous level, and output the processing results to the sub-residual block of the next level and the sub-residual block of the next next level.

[0029] Optionally, the first discriminator and the second discriminator both include a three-layer discriminant network and a one-layer granularity discriminant network, and any layer of the discriminant network includes three convolutional layers and one activation function layer.

[0030] Optionally, the bidirectional GAN ​​network model is optimized by the following loss functions: generative adversarial loss function, MSE loss function and total variational loss function.

[0031] The second aspect of the present application discloses a method for image defogging and recognition, the method comprising:

[0032] Extract the depth information corresponding to different pixels in the image to be defogged;

[0033] Performing depth processing on the image to be defogged according to the depth information;

[0034] Inputting the image to be defogged after deep processing into a pre-built ultra-lightweight image defogging and recognition network model;

[0035] Obtain the image defogging and recognition results output by the ultra-lightweight image defogging and recognition network model.

[0036] Optionally, the extracting depth information corresponding to different pixels in the image to be defogged includes:

[0037] Convert the image to be defogged to obtain the brightness and saturation of the image;

[0038] Depth information corresponding to different pixels in the image to be defogged is generated according to the brightness and saturation.

[0039] The third aspect of the present application discloses a computer device, comprising:

[0040] Memory for storing computer programs;

[0041] A processor is used to implement the steps of the image defogging and recognition method as described in the second aspect of the present application when executing the computer program.

[0042] The fourth aspect of the present application discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is processed and executed, the steps of the image defogging and recognition method described in the second aspect of the present application are implemented.

[0043] The present application discloses an ultra-lightweight image defogging and recognition network model, and image defogging and recognition are achieved through the network model. The network model is composed of a bidirectional GAN ​​network model and a target detection network model connected in sequence. The bidirectional GAN ​​network model is used to process the input image to be defogged and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image. The target detection network model is a Yolo-Tiny-S network model that has been retrained with row pruning. During the row pruning retraining process, the original images in the training set of the target detection network model are trained multiple times. Before each training, the original images are downsampled by a preset multiple. After each training is completed, the scaling coefficients of the batch normalization layer are sorted and compared, and the previous layer of convolution kernels corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning. The target detection network model disclosed in this application has been further pruned on the basis of the current micro-recognition model Yolov3-tiny, which greatly reduces the scale of the ultra-lightweight image dehazing and recognition network model. It can be easily deployed on end-side platforms with limited computing power and power consumption resources or in mobile chips such as vehicle-mounted cameras, thereby completing high-speed target detection and being more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a schematic diagram of the structure of the ultra-lightweight image defogging and recognition network model disclosed in the embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of the target detection network model disclosed in the embodiment of the present application;

[0047] Figure 3 It is another structural schematic diagram of the target detection network model disclosed in the embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of the structure of the existing dehazing convolutional neural network;

[0049] Figure 5 A schematic diagram of the structure of a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of another bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0051] Figure 7A schematic diagram of a shared potential space in a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of a first generation unit and a second generation unit in a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of the structure of a first coupled residual block and a second coupled residual block in a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of the structure of a first discriminator and a second discriminator in a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of a bidirectional GAN ​​network model for calculating cross-domain conversion consistency disclosed in an embodiment of the present application;

[0056] Fig.12 A schematic diagram of an application of a bidirectional GAN ​​network model disclosed in an embodiment of the present application;

[0057] Fig.13 A schematic diagram of the workflow of an image defogging and recognition method disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to solve the problem that the current YOLO target detection model is relatively large and the neural network model obtained when it is applied to the image defogging and recognition process is not suitable for deployment on the terminal platform with limited computing power and power consumption resources, the present application discloses an ultra-lightweight image defogging and recognition network model and an image defogging and recognition method through the following embodiments.

[0059] See also Figure 1 The first embodiment of the present application discloses an ultra-lightweight image defogging and recognition network model, including: a bidirectional GAN ​​network model and a target detection network model connected in sequence.

[0060] The bidirectional GAN ​​network model is used to process the input image to be defogged and output a clear image, i.e., a defogged image.

[0061] The target detection network model is used to perform feature recognition processing on the clear image. The target detection network model is a Yolo-Tiny-S network model that has been retrained through row pruning. During the row pruning retraining process, the original images in the target detection network model training set are trained multiple times. Before each training, the original images are downsampled by a preset multiple. After each training, the scaling coefficients of the batch normalization layer are sorted and compared, and the previous layer of convolution kernels corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning.

[0062] In this embodiment, in order to enable the target detection network model to better complete fast detection, the size of the model is compressed and the input image size is reduced. However, the convergence performance of the model needs to be guaranteed during the compression process. Therefore, the row pruning and retraining method is used to train the model. The specific process includes:

[0063] The first step is to directly train the original images in the training set. At this time, the resolution of the original images is relatively large. Data enhancement is used during the training process, and a larger scaling ratio is adopted to create training images of different sizes.

[0064] The second step is to sort and compare the scaling factors of the batch normalization layer after training, and prune the channels with smaller scaling factors and compression requirements. Specifically, the convolution kernels of the previous layer corresponding to the channels with smaller scaling factors are removed, and only the convolution kernels corresponding to the channels with larger scaling factors in the batch normalization layer are retained.

[0065] In the third step, the original image is downsampled by two times, and the first and second steps are repeated to train again. During the training process, the original image is enhanced by reducing the scaling ratio to avoid the low resolution of some objects during the scaling process, which affects the convergence of the model. It should be noted that in the process of sorting and comparing the pruned channels of the batch normalization layer, the proportion of pruned channels is increased to further reduce the channels.

[0066] The fourth step is to downsample the original image to a higher multiple, repeat the first and second steps to train again, until the final computational effort and input image size meet the resource and speed requirements. For example: the original image is reduced from the input image resolution of 512*512 to 128*128, and the number of model channels is reduced to about half of the currently common ultra-lightweight object detection model Yolov3-tiny.

[0067] When the model converges to a good accuracy, the trained ultra-lightweight target detection model Yolo-Tiny-S is connected to the output port of the bidirectional GAN ​​network model to form an integrated defogging and recognition model, that is, an ultra-lightweight end-to-end image defogging and recognition network model. Through this model, direct target recognition can be performed on existing real foggy images in an end-to-end manner.

[0068] This embodiment proposes for the first time an ultra-lightweight target detection model Yolo-Tiny-S (micro Yolo detection model), which is jointly used with a bidirectional GAN ​​network model to complete target detection. The ultra-lightweight target detection model is based on the lightweight target detection model Yolov3-tiny, and performs high-proportion pruning training on iterative channels for small input images based on a batch normalization layer. This training scheme is proposed for the first time in this scheme, and the resulting target detection network model is much smaller in scale than the Faster-RCNN used in previous systems. The pruning iterative algorithm can effectively converge a network that can quickly recognize smaller images, which is beneficial to the model in actual application scenarios, such as real-time target detection on resource-constrained platforms such as Internet of Vehicles, mobile phones, or autonomous driving.

[0069] See also Figure 2 The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit connected in sequence. The clear image is input into the target detection network model through the front extraction unit. The feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit connected in sequence. The output ends of the front extraction unit and the middle extraction unit are both connected to the input end of the feature fusion splicing unit.

[0070] See also Figure 3 The feature fusion and splicing unit includes an upsampling subunit and a feature splicing subunit. The input end of the upsampling subunit is connected to the output end of the middle extraction unit, and the output end of the upsampling subunit and the output end of the front extraction unit are connected to the input end of the feature splicing subunit.

[0071] The front extraction unit includes a DBL-S combination subunit and a plurality of MDBL-S combination subunits connected in sequence. In this embodiment, the front extraction unit includes a DBL-S combination subunit and four MDBL-S combination subunits connected in sequence.

[0072] The middle extraction unit includes a plurality of MDBL-S combination subunits and one DBL-S combination subunit connected in sequence. In this embodiment, the front extraction unit includes two MDBL-S combination subunits and one DBL-S combination subunit connected in sequence.

[0073] The first rear extraction unit and the second rear extraction unit each include a DBL-S combination subunit and a convolution (Conv) subunit connected in sequence.

[0074] The DBL-S combined subunit is composed of a dark network convolution layer (DarkNet convolution layer), a batch normalization layer (BN layer) and a leaky corrected linear layer (LeakyRelu layer) connected in sequence. Among them, S refers to Slim, which is used to indicate that the corresponding module has been streamlined, and the dark network convolution layer is a dark network that has undergone channel simplification.

[0075] The MDBL-S combination subunit is composed of a maximum pooling layer (MaxPool layer) and the DBL-S combination subunit which are connected in sequence.

[0076] Combination Figure 2 and Figure 3 , take the clear image after defogging as input, complete feature extraction through the combination of the front extraction unit and the middle extraction unit, and then obtain two different feature maps of the same size from the middle and back of the network for prediction. Specifically, each grid on a feature map predicts 3 prediction boxes, and each prediction box requires five basic parameters (x, y, w, h, confidence) (predicted x position, predicted y position, predicted box width, predicted box height, predicted confidence), which represent the center point position, width and height of the prediction box, and the confidence of the existence of the box, and output a probability for each category. After that, the feature map derived from the back of the network is upsampled using the feature splicing subunit, and then connected to the feature map in the middle of the network (i.e., the output end of the front extraction unit) to form a new fused feature map. The feature map directly generated by the back of the network (i.e., the output end of the middle extraction unit) and the new fused feature map are sent to the first back extraction unit and the second back extraction unit respectively to generate the final prediction classification and regression box, so that the clear image after defogging can be recognized. Figure 3 The y1 and y2 in the figure represent two branches with the same function. Each of y1 and y2 contains the information of the predicted classification result and the regression box. The predicted classification result refers to the type of object detected in the image, and the regression box is the box that represents the position of the detected object.

[0077] For image dehazing, the commonly used dehazing convolutional neural network is Generative Adversarial Networks (GAN). This network model adopts an unsupervised learning method and produces high-accuracy output through the mutual game learning of two modules: the generation module and the discrimination module. The generation module includes an encoder and a decoder, such as Figure 4 As shown, the encoder is used to extract the feature vector from the input image, and the decoder is used to restore the low-level features from the feature vector.

[0078] However, in the prior art, only one-way conversion from foggy images to clear images is performed through GAN, so GAN only contains a one-way mapping relationship from foggy domain to clear domain. After the foggy image is processed by the encoder and decoder, the output clear image usually has halo effects and artificial defects, and the information of the original image is not well preserved.

[0079] This embodiment involves a bidirectional GAN ​​network model, see Figure 5 ,The network model includes an input module, a generation module and a discrimination module.

[0080] The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator.

[0081] The clear image input port, the first generation unit and the first discriminator are connected in sequence for feature extraction and reconstruction of the clear image, and the fog image input port, the second generation unit and the second discriminator are connected in sequence for feature extraction and reconstruction of the fog image.

[0082] like Figure 5 The bidirectional GAN ​​network model shown can defog foggy images and also reversely add fog to clear images, enhancing the consistency of changes across domains, and thus better constraining the conversion effects between each other, making the defogging images more natural in specific applications and enhancing the defogging effect on real foggy images. However, in the existing defogging technology, only GAN is used to perform a one-way conversion from foggy images to clear images, and only the mapping relationship from foggy domain to clear domain can be output, resulting in halo effects and artificial defects in the output of the image, and the final original image information is not well preserved.

[0083] join Figure 6 In the bidirectional GAN ​​network model designed in this embodiment, the first generation unit includes a first encoder, a shared latent space and a first decoder connected in sequence, and the second generation unit includes a second encoder, the shared latent space and a second decoder connected in sequence.

[0084] The shared latent space is used to store high-level features and output the high-level features to the first decoder and the second decoder. The high-level features include the high-level features extracted by the first encoder for the clear image and the high-level features extracted by the second encoder for the foggy image.

[0085] The training set and the validation set of the bidirectional GAN ​​network model both include paired clear images and foggy images.

[0086] In this embodiment, it is assumed that the foggy image and the clear image share a latent space, so the feature connection from the foggy image to the clear image can be completed through the latent space. Figure 7 , the clear image X and the fog image Y, which can be connected through a shared latent space Z, to restore the images in the two domains. Based on this theoretical basis, the bidirectional GAN ​​network model disclosed in this embodiment can process real fog images and synthetic fog images well, and restore and reconstruct the fog images by first transforming the encoding and then decoding, combined with a bidirectional generative adversarial network. The bidirectional GAN ​​network model is trained and verified through paired clear images and fog images, including a bidirectional mapping relationship between the fog domain and the clear domain, and can process images in different domains, effectively ensuring the authenticity of the image reconstruction.

[0087] This embodiment discloses a bidirectional GAN ​​network model as a defogging network for effectively processing real dense fog. The network structure is novel and effective, and can extract depth information of fog images at different depths. It has great advantages in remote and dense fog processing, and can greatly improve the accuracy of subsequent object recognition. Due to the limitation of the defogging model in previous methods, there is no way to directly train with real fog images, so the corresponding defogging results cannot adapt well to the detection in actual foggy scenes. The bidirectional GAN ​​network model can use real images for end-to-end training, so that it can better utilize the information of real fog images to assist in improving the detection effect in real foggy scenes. Through the joint training of real and synthetic data sets, it can better utilize the fog image information in real scenes, avoiding the difficulty of network training caused by the lack of paired training data. The learning of real information enables the subsequent target detection network model to achieve good generalization effects in real scenes.

[0088] For further information, see Figure 8 As shown, the first encoder and the second encoder respectively include a first convolution block, a second convolution block and a first coupling residual block which are connected in sequence.

[0089] The first decoder and the second decoder respectively include a second coupled residual block, a first step convolution block and a second step convolution block which are connected in sequence.

[0090] An output terminal of the first coupled residual block is connected to an input terminal of the shared latent space, and an output terminal of the shared latent space is connected to an input terminal of the second coupled residual block.

[0091] The output end of the first convolution block is jump-connected to the input end of the second stride convolution block.

[0092] The output end of the second convolution block is jump-connected to the input end of the first-step long convolution block.

[0093] Among them, the first convolution block and the second convolution block are used to extract high-level features of the input image, and the first coupled residual block and the second coupled residual block are responsible for learning the detailed information of different features of the image, which is convenient for the restoration and reconstruction of the image. The first step convolution and the second step convolution complete the reconstruction process of the image from high-level features to the output image (clear image or foggy image). The encoder and decoder are closely connected through jump connections. After each layer of convolution blocks, there will be a jump link connected to the corresponding stride convolution block. The jump connection links the output feature map of the encoder's convolution block to the input of the corresponding decoder and splices it at the dimensional level, so that the feature map output by the decoder can include the feature information of the original image. The jump connection enables the detailed texture of the image to be better learned. Through the connection of two layers of jump connections in high and low dimensions, different position information of the image can be processed separately, further ensuring the consistency of the output image and the input image, as well as the authenticity of the reconstruction.

[0094] For further information, see Fig. 9 , the first coupled residual block and the second coupled residual block are respectively composed of a plurality of cascaded sub-residual blocks.

[0095] The sub-residual block at any level includes a first convolutional layer, an activation function layer and a second convolutional layer connected in sequence, which are used to process the output results of the sub-residual block of the previous level and the output results of the sub-residual block of the previous level, and output the processing results to the sub-residual block of the next level and the sub-residual block of the next next level.

[0096] The coupled residual block can enhance the network's ability to process information and better fit the potential identity mapping of the deep network. In this embodiment, each sub-residual block consists of two convolutional layers and one activation function layer. Fig. 9 The output and input of three sub-residual blocks are cascaded. Each level of sub-residual blocks processes the input of this level and the low-dimensional input result of the previous level to extract feature information of different dimensions, and sends the calculation results to the next level sub-residual block and the next level sub-residual block respectively. Through the close connection of multiple layers of sub-residual blocks, the residual information of fog features of different dimensions can be learned.

[0097] After encoding and decoding by the encoder and decoder, the generated image is obtained, and the generated image is input into the discrimination module for discrimination. Fig.10The first discriminator and the second discriminator respectively include three layers of discriminant networks. Each layer of the discriminant network contains three convolutional layers and one layer of activation function. Each layer of the discriminant network downsamples the input of the previous level. The output results of the three-layer discriminant network with different granularities are compared with the real feature map to determine whether the network output meets expectations. Penalties are imposed on training results that do not meet expectations, and the discriminant loss is calculated and returned to the network to update parameters. A multi-level discriminant network is used to sample and compare the output results generated by the bidirectional GAN ​​network model through three layers of features, and coarse-grained and fine-grained comparisons are performed, which increases the accuracy of the discriminant module.

[0098] For the bidirectional GAN ​​network model, this embodiment adopts a loss function, and the auxiliary model can be better expressed. The optimized loss functions include the generative adversarial loss function, the MSE loss function and the total variational loss function.

[0099] In order to achieve good training results, the network needs the loss function to include generative adversarial loss, which comes from whether the image comes from the generator G(I) or the real data J. The classic generative adversarial loss is defined as follows:

[0100] L GAN (G,D)=E I (logD(J))+E I (log(1-D(G(I));

[0101] Combination Fig.11 , the bidirectional GAN ​​network model generates adversarial losses in both directions and calculates the cross-domain conversion consistency as follows:

[0102] L adv =L GAN (G c (I c ),Dis h )+L GAN (G h (I h ),Dis c );

[0103] Among them, LGAN() represents the classic adversarial generation loss, and Ih represents the input fog Figure X , Ic represents the input clear image Y, Gc() represents the corresponding generative network in the clear scene, Gh() represents the corresponding generative network in the foggy scene, Disc represents the discriminant module in the clear scene, and Dish represents the discriminant module in the foggy scene. By sharing the latent space theory and combining the bidirectional generative adversarial loss, the authenticity of the reconstructed image and the similarity of the results of cross-domain image conversion can be guaranteed.

[0104] In the training of synthetic fog images, the MSE loss function is used to ensure that the predicted images are similar to the real images. The MSE loss function is as follows:

[0105] L MSE =|G c (I c )-J h || 1 +|G h (I h )-J c || 1 ;

[0106] Among them, Jh and Jc represent the fog image data and clear image data in the real scene respectively, and the loss is defined as the difference between the pixels of the generated fog (clear) image and the real fog (clear) image. ||()||1 represents the mean (mean square error) of the sum of the squares of the errors of the corresponding pixels on the image.

[0107] The total variation loss function is used in the training of fog images to eliminate related artificial defects, make the image visually better, and preserve the texture and details of the image. The total variation loss function is as follows:

[0108]

[0109] in, and Represents the horizontal and vertical gradient difference.

[0110] Through the above training, when the model converges to a good accuracy, the obtained training weights can be saved to obtain the final bidirectional GAN ​​defogging network model. At this time, the model can defog the existing foggy image in an end-to-end manner.

[0111] See also Fig.12 When the bidirectional GAN ​​network model disclosed in this embodiment is deployed in an actual application scenario, the brief steps of defogging are as follows:

[0112] 1. Input the real fog image in the real scene.

[0113] 2. Preprocess the image, convert it from RGB space to HSV space, combine known parameters, and output the depth information of the image.

[0114] 3. Combine the depth information, process the fog image, decode the fog image and store it in the shared latent space, encode the image in the shared latent space, and generate a clear defogging image in the real scene.

[0115] In order to enhance the consistency of cross-domain conversion and obtain more realistic dehazed images, the existing unidirectional GAN ​​network was improved. Based on the same shared latent space, the two unidirectional GAN ​​networks of clear image → latent space → foggy image and foggy image → latent space → clear image were fused into one module through two sets of encoders and decoders to form a bidirectional GAN ​​network model that can generate foggy / clear images. The cross-domain conversion losses of foggy image → clear image → foggy image and clear image → foggy image → clear image are calculated through the corresponding discriminator, which can better learn the distribution of foggy and clear images.

[0116] The bidirectional GAN ​​network model disclosed in this embodiment, based on the shared latent space assumption, proposes a bidirectional defogging network that encodes first and then decodes. This network can process images in different domains, has strong adaptability to image conversion between cross-domains, can process fog information at different depths, and has strong advantages in remote and dense fog processing. The application of jump links, coupled residual blocks and multi-layer discriminant structures greatly improves the robustness of the network and can more accurately fit the image distribution of the defogging clear image. Real fog information can be used, and depth information can be extracted after preprocessing, which is convenient for processing fog at deep concentrations and has good expression in real scenes.

[0117] The bidirectional GAN ​​network model disclosed in the above embodiment can be used to perform defogging processing on the image. In practical applications, first, the depth information corresponding to different pixels in the image to be defogged is extracted to perform depth processing on the image to be defogged; the image to be defogged after depth processing is input into the pre-built bidirectional GAN ​​network model to obtain the defogged image output by the bidirectional GAN ​​network model.

[0118] This application discloses an ultra-lightweight image defogging and recognition network model, and realizes image defogging and recognition through the network model. The network model consists of a bidirectional GAN ​​network model and a target detection network model connected in sequence. The target detection network model is further pruned on the basis of the current micro-recognition model Yolov3-tiny, which greatly reduces the scale of the ultra-lightweight image defogging and recognition network model. It can be easily deployed on the end-side platform with limited computing power and power consumption resources or in mobile chips such as vehicle-mounted cameras, thereby completing high-speed target detection, which is more practical.

[0119] The ultra-lightweight image defogging and recognition network model disclosed in this application includes two modules, defogging and recognition. The defogging module adopts a bidirectional GAN ​​network model, and the recognition module adopts Yolo-Tiny-S as a target detection network model. These two modules can be directly connected on the basis of independent training to perform fog scene target detection with a high recognition rate and a fast speed, avoiding the huge manpower and time consumption of annotating the real fog image data set for the target detection task during the joint training process. Since the defogging module can combine the depth information of the real fog image and learn the real distribution of the fog, the connection between the previous defogging module and the object recognition network does not require the joint training of the target detection data set under the real fog image (this data set is currently missing), but can directly connect the separately trained defogging module and recognition module to form an integrated defogging and recognition working system in the real scene.

[0120] See also Fig.13 The second embodiment of the present application discloses a method for image defogging and recognition, the method comprising:

[0121] Step S11, extracting depth information corresponding to different pixels in the image to be defogged.

[0122] Step S12: performing depth processing on the image to be defogged according to the depth information.

[0123] Specifically, the format of the image to be defogged is converted, and based on OpenCV, the image to be defogged is converted from RGB format to HSV format, so as to extract the brightness v and saturation s of the image to be defogged. Among them, OpenCV is a cross-platform computer vision and machine learning software library released based on the BSD license (open source). Specifically, in the shadow detection algorithm, the image in RGB format is often converted to HSV format. For the shadow area, its chroma and saturation do not change much compared with the original image, mainly the brightness information changes greatly. By converting the RGB format to the HSV format, the H, S, and V components can be obtained, thereby obtaining the values ​​of chroma, saturation, and brightness.

[0124] Depth information corresponding to different pixels in the image to be defogged is generated according to the brightness v and the saturation s.

[0125] In actual operation, according to the brightness v and saturation s, the existing depth information formula is used for calculation, and the known parameters θ0, θ1, θ2 are linearly calculated to obtain the depth information d corresponding to different pixels in the image to be defogged.

[0126] The depth information calculation formula is as follows:

[0127] d(X)=θ0+θ1v(X)+θ2s(X)+ε(X);

[0128] Among them, X represents the image to be dehazed, ε(X) is a random variable representing the random error of the model, and ε can be regarded as a random graph.

[0129] The input original image to be defogged is processed in depth through depth information to enhance the performance of deep areas, such as increasing the brightness and contrast of distant areas to make the distant view clearer.

[0130] Step S13, inputting the image to be defogged after deep processing into a pre-built ultra-lightweight image defogging and recognition network model.

[0131] Step S14, obtaining the image defogging and recognition results output by the ultra-lightweight image defogging and recognition network model.

[0132] The ultra-lightweight image defogging and recognition network model disclosed in the present application includes two modules, defogging and recognition. The bidirectional GAN ​​network model of the defogging module can well extract the depth information of the real fog image and the synthetic fog image for processing. By first transforming the encoding and then decoding, the image is interpreted into a shared latent space, and then reconstructed. The reconstructed clear image is then sent to the recognition module. The Yolo-Tiny-S model of the recognition module can effectively perform object recognition to achieve the purpose of object recognition after defogging.

[0133] The third embodiment of the present application discloses a computer device, including:

[0134] Memory, used to store computer programs.

[0135] A processor is used to implement the steps of the image defogging and recognition method as described in the second embodiment of the present application when executing the computer program.

[0136] The fourth embodiment of the present application discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is processed and executed, the steps of the image defogging and recognition method described in the second embodiment of the present application are implemented.

[0137] This application aims to solve the problem of low recognition accuracy of foggy images in object recognition tasks, and proposes a network model that integrates image defogging and image recognition for a micro-object recognition network for the first time through the above embodiments. By improving the training model of the end-to-end defogging network, a network model for defogging and recognition that uses mixed training of real pictures and synthetic pictures is proposed. The model can extract the depth information of the picture for analysis, so that the end-to-end defogging network can correctly analyze the depth details and dense fog information of the image, so that the picture can have a better defogging processing effect in dense fog scenes, and the defogging model can also maintain a good effect in the subsequent micro-target detection network.

[0138] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.

Claims

1. An ultra-lightweight image defogging and recognition network model, characterized in that: include: A bidirectional GAN ​​network model and a target detection network model connected in sequence; The bidirectional GAN ​​network model is used to process the input image to be defogged and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image; The target detection network model is a Yolo-Tiny-S network model that has been retrained through row pruning. During the row pruning retraining process, the original images in the target detection network model training set are trained multiple times. Before each training, the original images are downsampled by a preset multiple. After each training, the scaling coefficients of the batch normalization layer are sorted and compared, and the convolution kernels of the previous layer corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning. The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit connected in sequence. The clear image is input into the target detection network model through the front extraction unit. The feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit connected in sequence. The output ends of the front extraction unit and the middle extraction unit are both connected to the input end of the feature fusion splicing unit.

2. The ultra-lightweight image defogging and recognition network model according to claim 1, characterized in that: The front extraction unit includes a DBL-S combination subunit and a plurality of MDBL-S combination subunits connected in sequence; The middle extraction unit includes a plurality of the MDBL-S combination subunits and one DBL-S combination subunit connected in sequence; The first rear extraction unit and the second rear extraction unit each include a DBL-S combination subunit and a convolution subunit connected in sequence; The feature fusion and splicing unit includes an upsampling subunit and a feature splicing subunit, the input end of the upsampling subunit is connected to the output end of the middle extraction unit, and the output end of the upsampling subunit and the output end of the front extraction unit are connected to the input end of the feature splicing subunit; The DBL-S combined subunit consists of a dark network convolutional layer, a batch normalization layer, and a leakage corrected linear layer connected in sequence; The MDBL-S combination subunit is composed of a maximum pooling layer and the DBL-S combination subunit connected in sequence.

3. The ultra-lightweight image defogging and recognition network model according to claim 1, characterized in that: The bidirectional GAN ​​network model includes an input module, a generation module and a discrimination module; The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator; The clear image input port, the first generation unit and the first discriminator are connected in sequence to extract and reconstruct features for the clear image, and the fog image input port, the second generation unit and the second discriminator are connected in sequence to extract and reconstruct features for the fog image; The first generation unit includes a first encoder, a shared latent space, and a first decoder connected in sequence, and the second generation unit includes a second encoder, the shared latent space, and a second decoder connected in sequence; The shared latent space is used to store high-level features, and output the high-level features to the first decoder and the second decoder, wherein the high-level features include the high-level features extracted by the first encoder for the clear image and the high-level features extracted by the second encoder for the foggy image; The training set and the validation set of the bidirectional GAN ​​network model both include paired clear images and foggy images.

4. The ultra-lightweight image defogging and recognition network model according to claim 3, characterized in that: The first encoder and the second encoder respectively include a first convolution block, a second convolution block and a first coupling residual block connected in sequence; The first decoder and the second decoder respectively include a second coupled residual block, a first step length convolution block and a second step length convolution block connected in sequence; The output terminal of the first coupled residual block is connected to the input terminal of the shared latent space, and the output terminal of the shared latent space is connected to the input terminal of the second coupled residual block; The output end of the first convolution block is jump-connected to the input end of the second stride convolution block; The output end of the second convolution block is jump-connected to the input end of the first-step long convolution block.

5. The ultra-lightweight image defogging and recognition network model according to claim 4, characterized in that: The first coupled residual block and the second coupled residual block are respectively composed of a plurality of cascaded sub-residual blocks; The sub-residual block at any level includes a first convolutional layer, an activation function layer and a second convolutional layer connected in sequence, which are used to process the output results of the sub-residual block of the previous level and the output results of the sub-residual block of the previous level, and output the processing results to the sub-residual block of the next level and the sub-residual block of the next next level.

6. The ultra-lightweight image defogging and recognition network model according to claim 3, characterized in that: The first discriminator and the second discriminator both include a three-layer discriminant network and a one-layer granularity discriminant network, and any layer of the discriminant network includes three convolutional layers and one activation function layer.

7. The ultra-lightweight image defogging and recognition network model according to claim 3, characterized in that: The bidirectional GAN ​​network model is optimized through the following loss functions: generative adversarial loss function, MSE loss function and total variational loss function.

8. A method for image defogging and recognition, characterized in that: The method comprises: The bidirectional GAN ​​network model is used to process the input image to be dehazed and extract the depth information corresponding to different pixels in the image to be dehazed; According to the depth information, depth processing is performed on the image to be defogged to output a clear image; Performing feature recognition processing on the clear image using a target detection network model to obtain a recognition result of the clear image; The target detection network model is a Yolo-Tiny-S network model that has been retrained through row pruning. During the row pruning retraining process, the original images in the target detection network model training set are trained multiple times. Before each training, the original images are downsampled by a preset multiple. After each training, the scaling coefficients of the batch normalization layer are sorted and compared, and the convolution kernels of the previous layer corresponding to the channels whose scaling coefficients are less than the preset scaling threshold are removed to achieve pruning. The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit connected in sequence. The clear image is input into the target detection network model through the front extraction unit. The feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit connected in sequence. The output ends of the front extraction unit and the middle extraction unit are both connected to the input end of the feature fusion splicing unit.

9. The image defogging and recognition method according to claim 8, characterized in that: The extracting depth information corresponding to different pixels in the image to be defogged includes: Convert the image to be defogged to obtain the brightness and saturation of the image; Depth information corresponding to different pixels in the image to be defogged is generated according to the brightness and saturation.

10. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image defogging and recognition method as described in claim 8 or 9 when executing the computer program.

Images