Intelligent stereoscopic image defogging method

Abstract

The application discloses a kind of intelligent stereoscopic image defogging methods, method includes: for the input foggy left and right views, the feature extraction module is constructed respectively to extract the feature of foggy left and right views;Disparity-guided fog perception mechanism is constructed, the disparity between left and right views is calculated, and the distribution of fog in different areas in network perception scene is guided;Build cross-view global interaction module, enhance features by extracting global correlation information between views;The feature extraction module, disparity-guided fog perception mechanism and cross-view global interaction module are trained, and the output result of network is constrained using absolute error loss function and perception loss function during training process;Input has fog stereoscopic image, excavate global correlation information between views, utilize disparity relationship to guide network to perceive fog density, and output clear stereoscopic image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and image defogging, and particularly relates to an intelligent stereo image defogging method. BACKGROUND

[0002] Under foggy weather, the image acquisition process of the sensor will be affected by atmospheric particle scattering, resulting in problems such as low contrast, color distortion and texture detail loss in the collected foggy image. Such problems not only affect the visual experience of the viewer, but also bring challenges to many computer vision applications. As a bottom task of computer vision, image defogging aims to filter out the fog in the foggy image to restore the detailed information in the image, and has wide application in the fields of automatic driving, video monitoring and unmanned aerial photography. According to the different data processed, the image defogging task can be divided into single image defogging task and stereo image defogging task.

[0003] Stereo images are composed of left and right views taken from different positions and angles, which can provide viewers with vivid stereo visual perception. In order to restore clear stereo images, one approach is to defog the foggy left and right views respectively based on single image defogging methods. This approach ignores the inter-viewpoint information of the left and right views, making it difficult to ensure the depth consistency of the stereo images before and after defogging, and the defogging effect still needs to be improved. Therefore, it is of great significance and value to research stereo image defogging methods and effectively exploit the inter-viewpoint information of the left and right views to restore high-quality clear stereo images.

[0004] In recent years, with the continuous development of deep learning and computer vision technology, more and more researchers have begun to explore stereo image defogging methods based on deep learning. Song et al. proposed a stereo matching and defogging synchronous network SSMDN, which improves the quality of the reconstructed image by jointly using a stereo matching model and an image defogging model. Pang et al. proposed a stereo refinement defogging network SRDNet, which realizes stereo image defogging in a coarse-to-fine manner. However, existing stereo image defogging methods only extract consistency information from the matching regions of the left and right views, ignoring the relevance information in the non-matching regions. In addition, the farther the region in the scene is from the camera, the greater the impact of the fog on the image acquisition process, and the relationship between scene depth and fog density has not been effectively explored. Therefore, exploiting the global inter-viewpoint relevance in stereo images and exploring the intrinsic relationship between scene depth and fog density can help to reconstruct high-quality clear stereo images. SUMMARY

[0005] The existing stereo image defogging method only extracts consistent information from the matching area of left and right views to assist defogging, ignoring the correlation information of the non-matching area. In addition, the farther the area in the scene from the camera, the greater the influence of fog during image acquisition, and the relationship between scene depth and fog density has not been effectively explored. In order to solve the above problems, the present application provides an intelligent stereo image defogging method, which uses the global modeling capability of the multi-head attention mechanism Transformer to mine the global correlation information in the stereo image, and uses the disparity relationship to guide the network to perceive the fog density, as described in detail below:

[0006] An intelligent stereo image defogging method, the method comprising:

[0007] For the input foggy left and right views, a feature extraction module is constructed to extract the features of the foggy left and right views respectively;

[0008] A disparity-guided fog perception mechanism is constructed to calculate the disparity between the left and right views, and guide the network to perceive the distribution of fog in different areas of the scene;

[0009] A cross-view global interaction module is constructed to enhance the features by extracting global correlation information between views;

[0010] The feature extraction module, the disparity-guided fog perception mechanism and the cross-view global interaction module are trained, and during the training process, the absolute error loss function and the perception loss function are used to constrain the output results of the network;

[0011] Input the foggy stereo image, mine the global correlation information between views, use the disparity relationship to guide the network to perceive the fog density, and output the clear stereo image.

[0012] The fog perception mechanism is: using a disparity estimation model to calculate the left disparity map p l and the right disparity map p r , the formula is as follows:

[0013] p l , p r =M p (I l , I r )

[0014] Where M p represents a disparity estimation model;

[0015] Based on the disparity maps p l and p r , generate fog perception embedding vectors e l and e r , and e l and e r are combined with the features fl and f r Perform fusion; achieve fog-aware embedding vector e through feature attention. l e r Features f of the left and right views with fog l f r The adaptive fusion is expressed by the following formula:

[0016] e l =Conv l (p l )

[0017] e r =Conv r (p r )

[0018] a l b l =Split(Softmax(MLP(GAP(e l +f l ))))

[0019] a r b r =Split(Softmax(MLP(GAP(e r +f r ))))

[0020] f l h =a l e l +b l f l +f l

[0021]

[0022] Among them, Conv l (·) and Conv r (·) represent the symbols used to generate e. l and e r The convolutional layer, a l a r b l and b r The GAP(·) operation represents global average pooling, the MLP(·) operation represents a multilayer perceptron, the Softmax(·) operation represents the softmax activation function, and the Split(·) operation represents the weight splitting operation. l h and These represent the fog density perception features of the left and right views, respectively.

[0023] wherein the cross-view global interaction module comprises an inter-attention unit, a self-attention unit and a multi-layer perceptron, and first obtains a vector q l r and v r as follows:

[0024] ql=Conv q (f l h )

[0025]

[0026]

[0027] wherein Conv q , Conv k and Conv v respectively represent convolution layers for obtaining q l , k r and v r , f l h and is a fog density perception feature based on left and right views;

[0028] The inter-attention unit is used to interact the left and right view features, and is expressed as follows:

[0029]

[0030]

[0031] wherein represents matrix multiplication, T represents a transpose operation of a matrix, B represents a learnable relative position encoding, M represents a global correlation matrix, f l c represents an output of the inter-attention unit;

[0032] The left view feature f l h is sent into the self-attention unit to extract intra-view information for feature enhancement, and an output f l s of the self-attention unit is obtained, and is expressed as follows:

[0033]

[0034] wherein SelfAttn l (·) represents the self-attention unit; f l c and f l s ​The feature mapping and nonlinear transformation are performed by a multi-layer perception after the cascade to obtain the final feature representation f l 1 :

[0035] f l 1 = MLP(cat(f l c , f l s ))

[0036] wherein f l 1 represents the output of the first cross-view global interaction module, cat(·) represents the concatenation operation, and MLP(·) represents the multi-layer perception.

[0037] wherein the method further comprises: stacking multiple cross-view global interaction modules to gradually extract global correlation information, as follows:

[0038]

[0039]

[0040] wherein f l N and respectively represent the output of the Nth cross-view global interaction module for the left view and the right view, and respectively represent the Nth cross-view global interaction module for the left view and the right view.

[0041] The clear left and right images J l and J r are recovered by using the reconstruction layer, as follows:

[0042] K l , B l = Split(Conv recon (p l ))

[0043] K r , B r = Split(Conv recon (p r ))

[0044]

[0045]

[0046] wherein K l and K r respectively represent the K reconstruction matrix of the left and right views, and Bl and B l denote the B reconstruction matrix of left and right views respectively, Conv recon denote the reconstruction layer convolution, used to generate the K reconstruction matrix and the B reconstruction matrix, the reconstruction layer convolution parameters of left and right views are shared, and denote the left and right clear images obtained through the reconstruction layer respectively.

[0047] The beneficial effects of the technical solutions provided by the present application are:

[0048] 1. The present application proposes an intelligent stereo image defogging method through the feature expression ability of deep learning, fully excavates the global correlation information between views, and uses the disparity relationship to guide the network to perceive the fog density, thereby realizing the defogging of stereo images.

[0049] 2. The present application designs a cross-view global interaction module, uses the global modeling ability of the Transformer to model the correlation between views, and improves the expression ability of view features; at the same time, a disparity-guided fog perception mechanism is designed, which introduces the disparity information into the defogging process, guides the network to perceive the fog density, so as to realize more accurate defogging effect.

[0050] 3. The present application can obtain better performance than existing stereo image defogging methods through experiments on a stereo image defogging dataset. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a flowchart of an intelligent stereo image defogging method. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0053] The present application provides an intelligent stereo image defogging method, as shown in Figure 1 The method comprises the following steps:

[0054] 1. Constructing a feature extraction module

[0055] For the input foggy left and right views I l and I r , the feature extraction module is constructed to extract the features f l and f r of the foggy left and right views respectively. The process can be expressed as follows:

[0056]

[0057]

[0058] wherein, and respectively represent feature extraction modules for extracting left and right view features.

[0059] 2. Constructing a disparity-guided fog perception mechanism

[0060] In order to explore the relationship between scene depth and fog density, a disparity-guided fog perception mechanism is designed, which guides the network to perceive the distribution of fog in different regions of the scene by calculating the disparity between left and right views. Specifically, the embodiment of the present application first calculates the left disparity map p l and the right disparity map p r using a disparity estimation model, which can be expressed as follows:

[0061] p l , p r = M p (I l , I r )

[0062] wherein, M p represents a disparity estimation model.

[0063] Then, fog perception embedding vectors e l and e r are generated based on the disparity maps p l and p r , and e l and e r are fused with the features f l and f r of the foggy left and right views. The embodiment of the present application adopts a soft fusion method, and realizes the adaptive fusion of the fog perception embedding vectors e l , e r and the features f l , f r of the foggy left and right views through feature attention, which can be expressed as follows:

[0064] e l = Conv l (p l )

[0065] e r = Conv r (p r )

[0066] a l , b l = Split(Softmax(MLP(GAP(e l +f l )))

[0067] a r, b r = Split(Softmax(MLP(GAP(e r + f r )))

[0068] f l h = a l e l + b l f l + f l

[0069]

[0070] where Conv l (·) and Conv r (·) denote the convolutional layers for generating e l and e r , a l , a r , b l and b r represent the feature fusion weights, GAP(·) denotes the global average pooling operation, MLP(·) denotes the multi-layer perceptron, Softmax(·) denotes the softmax activation function, Split(·) denotes the weight splitting operation, f l h and denote the fog density perception features of the left and right views, respectively.

[0071] 3. Constructing a cross-view global interaction module

[0072] For the fog density perception features f l h and The embodiment of the present application constructs a cross-view global interaction module to enhance the features by extracting the global correlation information between views. Since the network architecture of the left and right views is consistent, the cross-view global interaction module of the left view will be introduced as an example.

[0073] The cross-view global interaction module consists of three parts, namely the mutual attention unit, the self-attention unit and the multi-layer perceptron. The cross-view global interaction module constructed based on the fog density perception features f l h and of the left and right views first obtains vectors q l , k r and v r , which are expressed as follows:

[0074] q l = Conv q (fl h )

[0075]

[0076]

[0077] where Conv q , Conv k and Conv v represent the convolutional layers for obtaining q l , k r and v r , respectively.

[0078] Then, in order to mine the global correlation information between the left and right views, the mutual attention unit is used to interact the left and right view features, which can be expressed as follows:

[0079]

[0080]

[0081] where denotes the matrix multiplication, T denotes the transpose operation of the matrix, B denotes the learnable relative position encoding, M denotes the global correlation matrix, f l c denotes the output of the mutual attention unit.

[0082] At the same time, the left view feature f l h is sent into the self-attention unit to extract the intra-view information for feature enhancement, and the output f l s of the self-attention unit is obtained, which can be expressed as follows:

[0083] f l s = SelfAttn l (f l h )

[0084] where SelfAttn l (·) denotes the self-attention unit.

[0085] Finally, f l c and f l s are concatenated and then passed through a multi-layer perceptron for further feature mapping and nonlinear transformation to obtain the final feature representation f l 1 :

[0086] f l1 = MLP(cat(f l c , f l s ))

[0087] where f l 1 represents the output of the first cross-view global interaction module, cat(·) represents the concatenation operation, and MLP(·) represents the multi-layer perceptron.

[0088] To capture rich correlation information between the left view and the right view, embodiments of the present application stack multiple cross-view global interaction modules to extract global correlation information step by step, which is expressed by the following formula:

[0089]

[0090]

[0091] where f l N and f represent the output of the Nth cross-view global interaction module for the left view and the right view, respectively. and f represent the Nth cross-view global interaction module for the left view and the right view, respectively. In a specific implementation, stacking 16 cross-view global interaction modules has the best performance.

[0092] For the output f l N and f of the cross-view global interaction module, embodiments of the present application recover clear left and right images J l and Jr by using the reconstruction layer, and the reconstruction process is expressed by the following formula.

[0093] K l , B l = Split(Conv recon (p l ))

[0094] K r , B r = Split(Conv recon (p r ))

[0095]

[0096]

[0097] where K l and K r represent the K reconstruction matrices of the left view and the right view, respectively, and Bl and B l denote the B reconstructed matrix of left and right views, Conv recon denote the reconstructed layer convolution, used to generate the K reconstructed matrix and the B reconstructed matrix, the reconstructed layer convolution parameters of left and right views are shared, and denote the left and right clear images obtained through the reconstructed layer.

[0098] 4. Training the intelligent stereo image defogging network.

[0099] In the training process, the intelligent stereo image defogging network includes a feature extraction module, a disparity-guided haze perception mechanism and a cross-view global interaction module. At the same time, in order to enhance the texture details and perceptual similarity of the stereo image, the neural network is trained using the mean absolute error (MAE) loss function and the perceptual loss function. Specifically, the formula expression of the mean absolute error (MAE) loss function is as follows:

[0100]

[0101] wherein M represents the batch size, and denote the left and right clear images recovered by the i-th network in the same batch, and denote the i-th real left and right clear images in the same batch.

[0102] The perceptual loss function uses the multi-scale features extracted by the pre-trained loss network to calculate the visual difference between the reconstructed clear stereo image and the real clear stereo image. The embodiment of the present application introduces VGG16 pre-trained on ImageNet as the loss network, so the formula expression of the perceptual loss function is as follows:

[0103]

[0104] wherein φ j (·) denotes the feature map extracted from the j-th stage of the loss network, and the total number of j is set to 3 in the embodiment of the present application. and denote the feature maps obtained by sending and into the loss network.

[0105] Finally, the loss function expression used by the training network is as follows:

[0106] L=L MSE +L Perceptual

[0107] 5. Application of the intelligent stereo image defogging network

[0108] After the training of the intelligent stereo image defogging network is completed, the foggy stereo image is sent into the network, the global correlation information between views is mined, and the disparity relationship is used to guide the network to perceive the fog density, so that the intelligent stereo image defogging network can effectively perform defogging processing on the foggy stereo image, and output clear stereo images.

[0109] The types of the devices in the embodiments of the present application are not limited, except for the special description, and the devices that can complete the above functions are all acceptable.

[0110] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0111] The above is only a preferred embodiment of the present application, and does not limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An intelligent stereoscopic image defogging method, characterized in that, The method comprises: For the input foggy left and right views, a feature extraction module is constructed to extract the features of the foggy left and right views respectively; A disparity-guided fog perception mechanism is constructed to calculate the disparity between the left and right views and guide the network to perceive the distribution of fog in different regions of the scene; A cross-view global interaction module is constructed to enhance the features by extracting global correlation information between the views; The feature extraction module, the disparity-guided fog perception mechanism and the cross-view global interaction module are trained, and during the training process, an absolute error loss function and a perception loss function are used to constrain the output results of the network; The input foggy stereo image is used to mine the global correlation information between the views, the network is guided to perceive the fog density by using the disparity relationship, and a clear stereo image is output; The fog perception mechanism comprises: calculating a left disparity map using a disparity estimation model and a right disparity map , which is expressed by the following equation: ； wherein, denotes the disparity estimation model; Based on the parallax map And Generate fog-aware embedding vector And And fuse And With the features of the foggy left and right views And Adaptive fusion of fog-aware embedding vector , With the features of the foggy left and right views , Adaptive fusion, the formula is as follows: ； ； ； ； ； ； wherein, and denote convolutional layers for generating and respectively, , , and denote feature fusion weights, denotes a global average pooling operation, denotes a multi-layer perceptron, denotes a softmax activation function, denotes a weight splitting operation, and denote fog density perception features for left and right views respectively.

2. The method of claim 1, wherein, The cross-view global interaction module comprises a mutual attention unit, a self-attention unit and a multi-layer perception machine, First, the vectors are acquired , and as follows: ； ； ； wherein, , and represent convolutional layers for obtaining , and , and are fog density perception features based on left and right views; The mutual attention unit is used to interact the features of the left and right views, and the expression is as follows: ； ； wherein, denotes matrix multiplication, denotes the transpose operation of a matrix, denotes learnable relative position encodings, M denotes a global correlation matrix, denotes the output of the cross-attention unit; The left view features are sent into a self-attention unit, extract intra-view information for feature enhancement, and obtain the output of the self-attention unit , expressed as follows: ； wherein, denotes a self-attention unit; and concatenated are mapped by a multi-layer perceptron for feature mapping and non-linear transformation to obtain the final feature representation : ； wherein, denotes the output of the first cross-view global interaction module, denotes a concatenation operation, denotes a multi-layer perceptron.

3. The method of claim 2, wherein, The method further comprises: stacking multiple cross-view global interaction modules to gradually extract global correlation information, as follows: ； ； wherein, and respectively represent the output of the first cross-view global interaction module for the left and right views, and respectively represent the first cross-view global interaction module for the left and right views; Recovering clear left and right images using a reconstruction layer and As follows: ； ； ； ； wherein, and K reconstruction matrices of left and right views, respectively, and B reconstruction matrices of left and right views, respectively, denotes a reconstruction layer convolution for generating the K reconstruction matrices and the B reconstruction matrices, reconstruction layer convolution parameters being shared for left and right views, and denote left and right sharp images obtained by the reconstruction layer, respectively.

Images