Training method of dual knowledge embedded data driven defogging model

Abstract

The invention relates to a training method of a dual knowledge embedded data driven defogging model, and belongs to the field of automatic driving and auxiliary driving. The method comprises the steps of high-quality reference true value data set division, low-quality degradation data generation, dual knowledge generation, training input data set integration, defogging network training, defogging model optimization and defogging model integration. According to the knowledge embedding training method provided by the invention, knowledge embedding of a data level and a model level can be realized, the problem that a current defogging model is deviated to a specific mode of training data is solved, the defogging performance of the model in a degradation environment is improved, and meanwhile, the stability of the model in different environments is enhanced; the method has important application value in the fields of auxiliary driving, automatic driving, autonomous sensing robots and the like.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving and assisted driving, and specifically relates to a training method for a dual knowledge embedded data-driven defogging model. Background Art

[0002] Extreme haze and heavy fog weather will affect the intelligent perception functions of autonomous driving, assisted driving and other systems, resulting in reduced performance of the perception system, blurred observation images, and poor perception of the observed objects, affecting the accuracy of target detection and recognition, and further causing the decision-making system to make wrong judgments.

[0003] At present, the atmospheric scattering model based on dual theory has achieved remarkable results under strong haze conditions. However, due to the lack of data supervision, the knowledge-based dehazing method has the problem of insufficient flexibility and generalization. At the same time, the data-driven dehazing model tends to process specific patterns of training data and cannot be applied to extreme haze conditions.

[0004] In order to solve the above problems, the present invention proposes a training method for dual knowledge embedding defogging network, which decomposes the entire processing process into manageable sub-models, which are used to learn dual prior knowledge and improve image quality respectively. In this mode, a key factor to achieve good performance is the tight integration of different sub-modules. Therefore, the method designs a training architecture at the model level and an objective function at the data level. During the training process, the objective function is minimized based on the training architecture to achieve a close integration of the sub-models at the model level and the data level. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is how to provide a training method for a dual knowledge embedded data-driven defogging model, which is used to improve the perception quality of observed objects under extreme haze conditions, and can improve both the performance of the defogging model and the stability of the defogging model in different degraded environments.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention proposes a training method for a dual knowledge embedding data-driven defogging model, which includes the following steps:

[0009] S1, high-quality reference truth data set division step: according to the needs, the high-quality reference truth data set Y is divided into sub-data sets Y1 and Y2, so that Y = Y1 ∪ Y2;

[0010] S2, low-quality degraded data generation step: the high-quality data set Y1 is based on the context feature degrader G1 to generate the low-quality data set X1, that is, G1:Y1→X1; the high-quality data set Y2 is based on the texture feature degrader G2 to generate the low-quality data set X2, that is, G2:Y2→X2;

[0011] S3, dual knowledge generation step: the low-quality data set X2 is calculated based on the knowledge model M1 to obtain the processing result set X 12 , that is, M1:X2→X 12 ;

[0012] S4, training input data set integration step: the data set X 12 As the data enhancement part of the dataset X1, it is integrated to complete the knowledge embedding at the data level, that is, X = X1 ∪ X 12 ;

[0013] S5, dehazing network training step: input the training data set X into the network model M2 to generate the prediction output Right now

[0014] S6. Dehazing model optimization step: During the training process, minimize the predicted output results The loss function between the high-quality data set Y is used to optimize the dehazing model;

[0015] S7, defogging model integration step: connect the knowledge model M1 with the network model M2, and realize the knowledge embedding at the model level during use, that is, M2(M1):I in →I output .

[0016] (III) Beneficial effects

[0017] The present invention proposes a training method for a dual knowledge embedded data-driven defogging model. The knowledge embedding training method proposed in the present invention can realize knowledge embedding at the data level and the model level, solve the problem that the current defogging model tends to be biased towards specific patterns of training data, improves the defogging performance of the model in a degraded environment, and at the same time enhances the stability of the model in different environments. It has important application value in the fields of assisted driving, automatic driving, and autonomous perception robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The flowchart of the training method of dual knowledge embedding defogging network of the present invention;

[0019] Figure 2 A training architecture diagram of a dual knowledge embedding dehazing model of the present invention;

[0020] Figure 3A schematic diagram of a dual encoding-decoding structure of the present invention;

[0021] Figure 4 A schematic diagram of a defogging network model of the present invention;

[0022] Figure 5 Schematic diagram of a dehazing model embedded with dual knowledge according to the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0024] The present invention relates to a training method for embedding dual knowledge into a defogging network. Specifically, a training method is designed to realize a dual encoding-decoding structure and a defogging backbone model closely embedded in the data layer and to minimize the objective function during the training process to embed the dual prior knowledge into the data information, thereby improving the processing performance of the defogging network model in extreme haze weather.

[0025] The present invention discloses a defogging network training method embedded with dual prior knowledge, the method comprising: (1) a high-quality reference truth data set division step. According to needs, the high-quality reference truth data set Y is divided into sub-data sets Y1 and Y2, so that Y=Y1∪Y2. (2) a low-quality degraded data generation step. The high-quality data set Y1 is generated into a low-quality data set X1 based on a context feature degrader G1, that is, G1:Y1→X1; the high-quality data set Y2 is generated into a low-quality data set X2 based on a texture feature degrader G2, that is, G2:Y2→X2. (3) a dual knowledge generation step. The low-quality data set X2 is calculated based on a model M1 to obtain a processing result set X1. 12 , that is, M1:X2→X 12 (4) Training input data set integration step. 12 As the data enhancement part of the dataset X1, it is integrated to complete the knowledge embedding at the data level, that is, X = X1 ∪ X 12 (5) Dehazing network training step. The training data set X is input to the network model M2 to generate the predicted output Y M2 ,Right now (6) Dehazing model optimization step. During the training process, the predicted output is minimized. The loss function between the high-quality dataset Y and the dehazing model is used to optimize the dehazing model. (7) Dehazing model integration step. The knowledge model M1 is connected to the network M2 to achieve knowledge embedding at the model level during use, that is, M2(M1):I in →I output .

[0026] The knowledge embedding training method proposed in the present invention can realize knowledge embedding at the data level and the model level, solve the problem that the current defogging model tends to be biased towards specific patterns of training data, improves the defogging performance of the model in a degraded environment, and at the same time enhances the stability of the model in different environments. It has important application value in the fields of assisted driving, autonomous driving, and autonomous perception robots.

[0027] The purpose of the present invention is to provide a training method for embedding dual knowledge into a defogging network model, which is used to improve the perception quality of observed objects under extreme haze conditions, thereby improving both the performance of the defogging model and the stability of the defogging model in different degraded environments.

[0028] To achieve the above object, the present invention proposes a training method for embedding dual knowledge into a defogging network model, the method comprising:

[0029] S1, high-quality reference truth data set division step: according to the needs, the high-quality reference truth data set Y is divided into sub-data sets Y1 and Y2, so that Y = Y1 ∪ Y2;

[0030] S2, low-quality degraded data generation step: the high-quality data set Y1 is based on the context feature degrader G1 to generate the low-quality data set X1, that is, G1:Y1→X1; the high-quality data set Y2 is based on the texture feature degrader G2 to generate the low-quality data set X2, that is, G2:Y2→X2;

[0031] S3, dual knowledge generation step: the low-quality data set X2 is calculated based on the knowledge model M1 to obtain the processing result set X 12 , that is, M1:X2→X 12 ;

[0032] S4, training input data set integration step. 12 As the data enhancement part of the dataset X1, it is integrated to complete the knowledge embedding at the data level, that is, X = X1 ∪ X 12 ;

[0033] S5, dehazing network training step. The training data set X is input to the network model M2 to generate the predicted output Y M2 ,Right now

[0034] S6. Dehazing model optimization step: During the training process, minimize the predicted output results The loss function between the high-quality data set Y is used to optimize the dehazing model;

[0035] S7, defogging model integration step: connect the knowledge model M1 with the network model M2, and realize the knowledge embedding at the model level during use, that is, M2(M1):Iin →I output .

[0036] Embodiment 1:

[0037] Figure 1 This is a flow chart of a training method for embedding dual knowledge into a defogging network of the present invention. Figure 1 As shown, the method includes:

[0038] (1) High-quality truth data set division step: According to the needs, the high-quality reference truth value Y is divided into sub-datasets Y1 and Y2, so that Y = Y1 ∪ Y2.

[0039] In a specific implementation, the number of n high-quality data sets Y is divided into the number of n1 data sets Y1 and the number of n2 data sets Y2, so that n=n1+n2, and n1≈n2 holds.

[0040] (2) Low-quality degraded data generation step. Figure 2 As shown, the high-quality data set Y1 is used to generate a low-quality data set X1 based on the context feature degenerater G1, that is, G1:Y1→X1; the high-quality data set Y2 is used to generate a low-quality data set X2 based on the texture feature degenerater G2, that is, G2:Y2→X2.

[0041] In the specific implementation, the context feature degrader G1 is used to destroy the context features of the Y1 data set to obtain a low-quality data set X1; the texture feature degrader G2 is used to degrade the texture features of the data set Y2 based on the degradation formula to obtain a low-quality data set X2. The context feature degrader and the texture feature degrader can be trained based on generative models such as CycleGAN and Duffusion.

[0042] Context feature degenerater: The model proposed in the paper “what else can fool deep learning? Addressing Color Constancy Errors on Deep Neural Network Performance” can be used to degenerate the context features of the observed objects in the image by changing the color constancy of the image. Alternatively, a context feature degenerater can be obtained by training based on generative models such as CycleGAN and Duffusion.

[0043] Texture feature degrader: It can be established based on the atmospheric degradation model, or trained based on generative models such as CycleGAN and Duffusion.

[0044] (3) Embedding knowledge generation step. Figure 3 As shown, the low-quality data set X2 is calculated based on the knowledge model M1 to obtain the processing result set X12 , that is, M1:X2→X 12 .

[0045] In the specific implementation, a dual encoder-decoder M1 is used to process the texture features of the data set X2. Figure 3 It is a texture feature dual encoder-decoder of the present invention. First, dual mapping T1 is performed to map the data set X2 to the dual space, and then the normalization layer Norm (·) constrains the solution space; then the encoder-decoder E1 and D1 with a cross-link mechanism are used to enhance the texture features in the dual space, and cross-links are used between the encoder and the decoder to splice the same-size features S1 to avoid feature loss during downsampling; finally, the texture feature enhancement result X based on dual knowledge is obtained through dual inverse mapping T2. 12 ,Right now:

[0046] X 12 =M1(X2)

[0047] =T2D1(E1(Norm(T1·X2),θ E1 ),S1,θ D1 )

[0048] Among them, S1 represents the same size feature in the encoding process, {θ E1 ,θ D1} represents model parameters.

[0049] (4) Training input data set integration step. 12 It is integrated as the data enhancement part of dataset X1 to complete the knowledge embedding at the data level.

[0050] In the specific implementation, the data set X 12 Integration with X1 enables

[0051] (5) Dehazing model training steps. Figure 4 As shown, the training data set X is input to the network model M2 to generate the predicted output Right now

[0052] In the specific implementation, it first passes through the encoder E2, whose shallow layer extracts texture features, and the deep layer obtains context features through a larger receptive field and generates potential representations; then the decoder D2 generates outputs through deconvolution operations; finally, the same-size features are spliced ​​through the cross-link mechanism, including the same-size features S2 generated by the encoder E2 and the same-size features T2 S1 generated by the previous dual encoder ( Figure 3 ), T2 is used to inversely map the feature S1 from the dual space into a feature that can be received by the decoder D2, that is:

[0053] Y M2 =M2(X)

[0054] =D2(E2(X,θ E2 ),S2,T2S1,θ D2 )

[0055] {θ E2 ,θ D2} represents model parameters.

[0056] (6) Dehazing model optimization step. During the training process, the predicted output is minimized. The loss function between the high-quality data set Y is used to optimize the network model and solve the network parameters {θ E1 ,θ D1 ,θ E2 ,θ D2}.

[0057] In specific implementation, the loss function L loss Includes: Linear combination L1 norm and gradient norm And by finding the minimum value to optimize the dehazing model, the knowledge embedding at the data level is completed, that is:

[0058]

[0059] Where θ={θ E1 ,θ D1 ,θ E2 ,θ D2} represents the model parameters, λ is a hyperparameter and has λ=0.05.

[0060] (7) Dehazing model integration step. Figure 5 As shown, the knowledge model M1 is connected with the dehazing model M2 to realize model-level knowledge embedding during use.

[0061] In the specific implementation, when the input observation image I in After that, the dehazing model will generate the result I output ,Right now:

[0062] I output =M2(M1(I in ,θ E1 ,θ D1 ),θ E2 ,θ D2 )

[0063] Embodiment 2:

[0064] A dehazing network training method embedding dual prior knowledge includes the following steps:

[0065] (1) Division of high-quality reference truth datasets.

[0066] (2) Low-quality degraded data generation.

[0067] (3) Dual knowledge generation.

[0068] (4) Training input dataset integration.

[0069] (5) Dehazing network training.

[0070] (6) Dehazing model optimization.

[0071] (7) Dehazing model integration.

[0072] Furthermore, in step (2), the prior knowledge of dual enhancement is embedded into the data-driven dehazing model from the data level to solve the problem that the current dehazing model tends to enhance the specific pattern of training data, thereby improving the performance and stability of the dehazing enhancement model.

[0073] Furthermore, in the step (3), a dual encoding-decoding structure is adopted, a normalization layer constraint solution space is set in the dual space, a small-size receptive field is used to obtain the dual texture features of the observed object, and a large-scale receptive field is used to obtain the spatial features of the observed object.

[0074] Furthermore, in step (5), since the downsampling feature extraction of the encoding network will lose the spatial features, and the upsampling of the decoder cannot recover the lost features, the same-scale features in different models are spliced ​​across connections to avoid the loss of spatial features, thereby suppressing artifact noise.

[0075] Furthermore, in step (6), an objective function that embeds dual knowledge is used. In order to embed dual prior knowledge, the dehazing process is decomposed into manageable sub-models. In this model, the loose coupling of sub-models can lead to unstable performance. Therefore, the loss function of the dual encoding-decoding model is used as a sub-function of the objective function, and different training tasks are marked. During the training process, the model parameters are optimized by finding the minimum value to achieve a tight integration of different sub-models.

[0076] The knowledge embedding training method proposed in the present invention can realize knowledge embedding at the data level and the model level, solve the problem that the current defogging model tends to be biased towards specific patterns of training data, improves the defogging performance of the model in a degraded environment, and at the same time enhances the stability of the model in different environments. It has important application value in the fields of assisted driving, autonomous driving, and autonomous perception robots.

[0077] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dual knowledge embedded data driven dehazing model training method, characterized in that: The method comprises the following steps: S1, high-quality reference truth data set division step: according to the needs, the high-quality reference truth data set Y is divided into sub-data sets Y1 and Y2, so that Y = Y1 ∪ Y2; S2, low-quality degraded data generation step: the high-quality data set Y1 is based on the context feature degrader G1 to generate the low-quality data set X1, that is, G1:Y1→X1; the high-quality data set Y2 is based on the texture feature degrader G2 to generate the low-quality data set X2, that is, G2:Y2→X2; S3, dual knowledge generation step: the low-quality data set X2 is calculated based on the knowledge model M1 to obtain the processing result set X 12 , that is, M1:X2→X 12 ; S4, training input data set integration step: the data set X 12 As the data enhancement part of the dataset X1, it is integrated to complete the knowledge embedding at the data level, that is, X = X1 ∪ X 12 ; S5, dehazing network training step: input the training data set X into the network model M2 to generate the prediction output Right now S6. Dehazing model optimization step: During the training process, minimize the predicted output results The loss function between the high-quality data set Y is used to optimize the dehazing model; S7, defogging model integration step: connect the knowledge model M1 with the network model M2, and realize the knowledge embedding at the model level during use, that is, M2(M1):I in →I output .

2. The dual knowledge embedded data driven defogging model training method according to claim 1, characterized in that: The S1 includes: dividing the number of high-quality data sets Y, which is n, into the number of data sets Y1, which is n1, and the number of data sets Y2, which is n2, so that n=n1+n2, and n1≈n2 holds.

3. The dual knowledge embedded data driven defogging model training method according to claim 2 is characterized in that: The S2 includes: using the context feature degenerator G1 to destroy the context features of the Y1 data set to obtain a low-quality data set X1; using the texture feature degenerator G2 to degenerate the texture features of the data set Y2 based on the degradation formula to obtain a low-quality data set X2.

4. The dual knowledge embedded data driven defogging model training method according to claim 3 is characterized in that: The context feature degenerater and texture feature degenerater are trained based on CycleGAN and Duffusion generative models.

5. The training method of the dual knowledge embedding data driven defogging model according to any one of claims 1 to 4, characterized in that: The S3 includes: using a dual encoder-decoder M1 to process the texture features of the data set X2: firstly, performing a dual mapping T1 to map the data set X2 to the dual space, and then normalizing the layer Norm(·) to constrain the solution space; then using an encoder E1 and a decoder D1 with a cross-link mechanism to enhance the texture features in the dual space, and using a cross-link between the encoder and the decoder to splice the same-size features S1; finally, obtaining the texture feature enhancement result X based on the dual knowledge through a dual inverse mapping T2. 12 ,Right now: X 12 =M1(X2) =T2D1(E1(Norm(T1·X2),θ E1 ),S1,θ D1 ) Among them, S1 represents the same size feature in the encoding process, {θ E1 ,θ D1 } represents model parameters.

6. The dual knowledge embedded data driven defogging model training method according to claim 5, characterized in that: The S5 includes: First, the encoder E2 extracts texture features in the shallow layer, and the deep layer obtains context features through a larger receptive field and generates potential representations. Then the decoder D2 generates outputs through deconvolution operations. Finally, the same-size features are concatenated through a cross-link mechanism, including the same-size feature S2 generated by the encoder E2 and the same-size feature T2 S1 generated by the previous dual encoder. T2 is used to inversely map the feature S1 from the dual space to a feature that can be received by the decoder D2, that is: {θ E2 ,θ D2 } represents model parameters.

7. The training method of the dual knowledge embedding data driven defogging model according to claim 6 is characterized in that: S6 includes: during the training process, minimizing the predicted output result Y M2 The loss function between the high-quality data set Y is used to optimize the network model and solve the network parameters {θ E1 ,θ D1 ,θ E2 ,θ D2 }.

8. The dual knowledge embedded data driven defogging model training method according to claim 7, characterized in that: Loss function L loss Includes: Linear combination L1 norm and gradient norm And by finding the minimum value to optimize the dehazing model, the knowledge embedding at the data level is completed, that is: Where θ={θ E1 ,θ D1 ,θ E2 ,θ D2 } represents the model parameters and λ is the hyperparameter.

9. The dual knowledge embedded data driven defogging model training method according to claim 8, characterized in that: λ=0.

05.

10. The training method of the dual knowledge embedding data driven defogging model according to claim 8, characterized in that: The step S7 includes: when the observation image I is input in After that, the dehazing model will generate the result I output ,Right now: I output =M2(M1(I in ,i E1 ,i D1 ),θ E2 ,i D2 )。

Images