Unmanned driving scene generation method based on StyleGAN-V algorithm

Abstract

The invention discloses an unmanned driving scene generation method based on a StyleGAN-V algorithm, and the method comprises the implementation steps: firstly, collecting a real severe weather driving scene video, and carrying out the preprocessing of the video, and constructing a data set; then preliminarily constructing a video generation model; a time sequence feature extraction module combining time sequence excitation and time sequence displacement is constructed, and time sequence features of the video are extracted; introducing a joint loss function constraint in the generation process; and finally, calculating an improved loss function, updating video generation model parameters, and continuously training until the generation model is completed. According to the method, the driving scene in the severe weather can be realistically simulated, and through experimental verification, the method can effectively improve the continuity of the generated video and the capturing capability of dynamic characteristics, so that the authenticity of the driving simulation scene in the severe weather is improved, the sensing capability of subtle action changes in the video is improved, and the driving experience is improved. And better performance is achieved in the aspects of dynamic characteristics and picture continuity of the video.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a method for generating an unmanned driving scene based on a StyleGAN-V algorithm. Background Art

[0002] With the rapid growth of the global automobile population and the frequent occurrence of traffic accidents, driverless technology has become an important solution to address traffic safety challenges. Driverless driving controls vehicle driving in an intelligent way, thereby minimizing traffic accidents, ensuring traffic safety and improving traffic efficiency. Driverless technology testing usually uses virtual driving scenario testing for verification and training. This virtual scenario testing provides a safe and cost-effective testing environment to improve the generalization of its performance. However, driving in severe weather conditions remains a huge challenge, so simulating severe weather such as rain, snow and fog in virtual scenarios is crucial to testing the performance of driverless technology in various complex environments.

[0003] Most existing methods use simulation platforms to create virtual scenes. For example, the document with application number "202410788219.6" discloses "a method, device and computer program product for generating materials for virtual simulation scenes of autonomous driving", which creates simulation scenes by inputting relevant keywords and parameters into a pre-trained image generation model. Its disadvantage is that the generated scenes are insufficient in details and diversity, affecting the authenticity and applicability of the test results. Another document with application number "202210172378.4" discloses "a method and device for generating unmanned driving simulation scenes", which collects local environment images and splits them into various environmental element images, then virtually generates lane element images based on local maps, and then synthesizes these element images into unmanned driving simulation scenes. Its disadvantage is that there are significant differences between the generated scenes and the real scenes, especially in the generation of complex and harsh scenes. The common problem with the above schemes is that it is difficult to simulate and generate severe weather conditions, and it is impossible to reproduce complex environments and details with high quality. Summary of the invention

[0004] The present invention proposes an unmanned driving scene generation method based on the StyleGAN-V algorithm to solve the problems in the prior art that it is difficult to simulate and generate severe weather conditions and cannot reproduce complex environments and details with high quality.

[0005] To achieve the above object, the technical solution of the present invention is as follows: a method for generating an unmanned driving scene based on the StyleGAN-V algorithm, comprising the following steps:

[0006] Step 1: Collect real videos of driving scenes in severe weather, preprocess them, and build a dataset;

[0007] Step 2: Preliminary construction of the video generation model, the steps include:

[0008] 2.1 Construct a motion feature extraction module based on a deep autoencoder structure;

[0009] 2.2 Use the pre-trained parameters to initialize the motion feature extraction module;

[0010] 2.3 The motion features of the driving scene videos in the dataset are extracted using the motion feature extraction module, and the high-dimensional motion data is mapped to a low-dimensional feature space;

[0011] 2.4 Use the obtained motion features to constrain the generated image and complete the preliminary construction of the video generation model;

[0012] Step 3: construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video;

[0013] Step 4: Introduce joint loss function constraints in the generation process;

[0014] Step 5: Calculate the improved loss function: After completing the forward propagation, use the stochastic gradient descent method to back-propagate the model, continuously adjust the model parameters, and gradually minimize the loss function;

[0015] Step 6: Update the video generation model parameters and continue training to determine whether the number of iterations has been reached and converged. If so, the training is completed to generate the model; if not, return to step 1 to continue training.

[0016] Furthermore, in the above step 2.1, the input of the motion feature extraction module combined with the deep autoencoder structure consists of two parts: the current motion T and the previous t frame motion; the motion feature extraction module combined with the deep autoencoder is deformed by the deep autoencoder structure, and the past motion sequence p is added as input in the deep autoencoder structure; the current motion T is the motion data of a single frame, which serves as the x input of the network model, and the previous t frame motion is the motion data of the previous t frames of the current motion T, which serves as the input of the past motion sequence p in the network model.

[0017] Furthermore, in the above step 2.4, in the encoding and decoding stage of the deep self-encoding structure, the original motion data is first mapped to the hidden layer features, and then restored to the original motion data to generate reconstructed motion data; by establishing a constraint relationship between the reconstructed data and the input data, the loss function of the network is constructed, which is defined as follows:

[0018]

[0019] Furthermore, in the above step 3, the steps of constructing the timing excitation module are:

[0020] First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W), and the global spatial information of the input features is represented by spatial average pooling, which can be expressed as formula (6):

[0021]

[0022] Among them, X represents the input feature map, represents the compressed feature map, c and t represent the number of channels and time dimension of the input feature map, respectively, and i and j represent the spatial height and width of the input feature map, respectively;

[0023] Then, the Sigmoid activation function is used to obtain the weight distribution on different time frames, and the weight distribution is multiplied with the original feature map to obtain the feature map after time series excitation. The formula (7) of time series excitation is as follows:

[0024]

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The present invention uses videos of driving scenes in bad weather for training and learning, which can realistically simulate driving scenes in bad weather and can be used for unmanned driving virtual simulation tests, thus avoiding the problems of the difficulty in simulating bad weather conditions and the inability to generate bad weather environments caused by the existing use of simulation platforms to create scenes;

[0027] (2) Aiming at the problem that the video generation model has limited ability to capture motion features, the present invention constructs a motion feature extraction module based on deep autoencoding. The method of the present invention uses the motion data of the current frame and the previous frame for feature extraction at the same time. Since the timing information and the autoencoding mechanism are introduced, a more reliable feature representation capability can be provided. By learning the features of the original motion data, the quality of video generation and the continuity of the picture motion are improved;

[0028] (3) In view of the problems such as texture adhesion and visual anomalies in the video generation model, the present invention constructs a temporal feature extraction module that combines temporal excitation and temporal displacement. The temporal feature extraction module in the method of the present invention can dynamically assign weights to the features of different time frames and introduce a displacement operation in the time dimension. By adopting the operation of combining temporal excitation and temporal displacement, the fusion and interaction of temporal information can be promoted, so as to better capture the temporal changes and dynamic features in the video, and improve the quality and authenticity of the generated video. In step three of the present invention, the feature map after temporal excitation expands the original feature map along the time and channel dimensions. Because the temporal excitation module does not act on a specific spatial position, but on different temporal positions, the specific spatial dimension is not displayed. Through this innovative design, the temporal excitation module can effectively improve the coherence of the generated video and the ability to capture dynamic features. The core idea of ​​the temporal displacement structure constructed in step three is to divide the time dimension into multiple segments and perform a displacement operation in each segment, so that the model can have better temporal modeling capabilities and innovatively improve the perception of subtle action changes in the video.

[0029] (4) The channel-based feature shifting method designed by the present invention mixes the feature information of adjacent time frames, which is equivalent to realizing the convolution operation in the time dimension. The fusion of time information is realized by the inter-channel fusion capability of the subsequent 2D convolution. After combining the temporal excitation and the temporal displacement, the final temporal feature extraction module is obtained. The fusion module assigns different activation weights to different times, and then the adjacent frames directly obtain part of each other's information through the temporal shift. This design not only retains the features with large changes, but also enables different time features to directly interact with each other and learn the temporal feature information. This method combines the video training of severe weather scenes, the deep autoencoder motion feature extraction module, and the temporal feature extraction module of temporal excitation and temporal displacement, thereby showing significant advantages in generating unmanned driving severe weather scenes. The present invention not only improves the authenticity of the driving simulation scene in severe weather, but also achieves better performance in the dynamic features and picture continuity of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall flow chart of the present invention;

[0031] Figure 2 This is a diagram of the deep autoencoder structure;

[0032] Figure 3 It is the structural diagram of the timing excitation module;

[0033] Figure 4 Network training flow chart;

[0034] Figure 5A schematic diagram of generating an unmanned driving scenario in severe weather conditions in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the attached examples. Obviously, the described embodiments are only part of the embodiments of the present invention, which are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0036] See also Figure 1 The basic idea of ​​the present invention is: first, collect videos of various severe weather conditions, pre-process these videos, and complete the establishment of the data set; second, construct a motion feature extraction module combined with deep autoencoding, improve the original motion feature extraction method, and improve the feature representation capability; then, construct a temporal feature extraction module that combines temporal excitation and temporal displacement, constrain the generation, and improve the coherence of the generated pictures; finally, fine-tune the model and constrain the generation with a joint loss function to guide the training of the generator network to obtain better generation effects.

[0037] Based on the above basic ideas, the present invention provides a method for generating unmanned driving severe weather scenes based on improved StyleGAN-V. The StyleGAN-V algorithm performs well in generating videos and can generate high-quality traffic scenes. The specific implementation steps are as follows.

[0038] Step 1: Collect real-world videos of driving in bad weather, preprocess them, and build a dataset. The details are as follows:

[0039] Step 1.1 Real-world severe weather driving scene videos should contain video clips from different cities and multiple weather conditions to ensure the diversity and richness of the training data.

[0040] Step 1.2 Video preprocessing uses inter-frame interpolation. Inter-frame interpolation generates more video frames by inserting new frames between existing frames, making the video look smoother. This can not only increase the frame rate of the video and improve the smoothness of the video, but also enhance the richness and diversity of the video dataset when the amount of data is limited.

[0041] Step 2: Preliminary construction of the video generation model, the steps include:

[0042] Step 2.1 builds a motion feature extraction module based on a deep autoencoder structure and uses it as one of the components of the generative adversarial network motion feature capture and motion migration. The motion migration network establishes motion synthesis constraints and uses the motion features obtained by the motion extraction module to perform motion migration synthesis to improve the quality of motion synthesis. The specific construction process is as follows:

[0043] Well-known deep autoencoder structures such as Figure 2 The input of the motion feature extraction module of the present invention combined with the known deep autoencoder structure consists of two parts: the current motion T and the motion of the previous t frames; the main network is deformed by the autoencoder, and in the deep autoencoder structure

[0044] In this paper, the past motion sequence p is innovatively added as input. The current motion T is the motion data of a single frame, which is used as the x input of the network model designed by this method. The previous t frame motion is the motion data of the previous t frames of the current motion T, which is used as the input of the past motion sequence p in the network model, so that the network can extract the temporal features of the motion sequence. The entire network model can be divided into the encoding stage and the decoding stage.

[0045] In the encoding stage of the deep self-encoding structure, the network maps the input raw motion data to the feature space, and according to the input current motion x∈R n×d and the past motion sequence p∈R n×(d×t) , t is the motion sequence of the previous t frames of the current motion T. Finally, the function of the network in the encoding stage is defined as:

[0046]

[0047] Where h∈R n×c Represents the hidden layer features of the original motion data; W xf ∈R d×k is the weight connecting the current motion x and the current encoding f; W1 fh ∈R k×c is the weight connecting the current code f and the hidden layer feature h: W pf ∈R (d×t)×k is the weight connecting the past motion sequence p and the current code f; W2 fh ∈R k×c is the weight connecting the past code f and the hidden feature h; b1∈R c is the bias vector; ReLU is the nonlinear activation function.

[0048] In the decoding stage of the deep autoencoder structure, the network restores the hidden motion features h to the original motion space to obtain the reconstruction of the motion capture The function of its decoding stage is:

[0049]

[0050] in, is the reconstructed output obtained by the motion feature extraction module combined with the deep autoencoder structure; W xf '∈R c×k is the weight connecting the hidden layer feature h and the current decoding f'; To connect the current decoded f′ with the reconstructed motion The weight of pf '∈R (d×t)×k is the weight connecting the past motion sequence with the past decoding f′; To connect the past decoding f′ and reconstructing motion The weight of b2∈R d is the bias vector; the selection of k, n, d and t is the same as that in the encoding stage.

[0051] The parameters obtained from the pre-training in step 2.2 are used to initialize the feature extraction module;

[0052] Use the network to pre-train the parameters of the feature extraction model so that the network can converge quickly. Add the past motion sequence based on the network structure, use ν to represent the visible layer, h to represent the hidden layer, and p to represent the past motion sequence, and define the energy function as:

[0053]

[0054] Among them, θ is the parameter of the model; ν i is the data of the i-th visible layer node; h j is the jth hidden layer node data; a i , b j Bias for the i-th visible layer and the j-th hidden layer; is the weight connecting the visible layer ν and the visible encoding, is the weight connecting the visible code and the hidden layer; The weights for connecting past motion sequences with past encodings; The past encoding and hidden layer weights.

[0055] During training, given the transition states of the neurons in the visible layer, the past motion sequence P and the parameter θ, the neurons in the hidden layer are calculated as follows:

[0056]

[0057] Among them, σ(·) is the activation function designed by this method, using sigmoid. The parameters of the motion feature extraction module combined with deep autoencoder can be obtained through the pre-training model described in formula (4). Through this innovatively designed training mechanism, the model can converge quickly and obtain a more accurate feature extraction effect.

[0058] Step 2.3 uses a feature extraction module to extract features and map high-dimensional motion data to a low-dimensional feature space;

[0059] Step 2.4 Use the obtained motion features to establish constraints:

[0060] In the encoding and decoding stages of the network, the original motion data is first mapped to the hidden layer features, and then restored to the original motion data to generate reconstructed motion data. By establishing a constraint relationship between the reconstructed data and the input data, the loss function of the network is constructed, which is defined as follows:

[0061]

[0062] Among them, l(x,θ) is the loss function, which represents the loss of the network; θ represents the parameters of the network; x is the input motion; For output motion; is the mean square error term between the input motion and the reconstructed motion, which represents the difference between the input and the output; λ||θ||1 is the regularization term, which represents the constraint on the network parameters and is used to prevent overfitting. The first term of this loss function is the mean square error between the input motion and the reconstructed motion, which requires that the input and output of the network are the same. Then, the hidden layer features of the network can be represented as the feature space of the original motion, and the past motion sequence is added during feature extraction, so that the extracted features have not only spatial structure but also temporal association.

[0063] Step 3: Construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video, improve the coherence of the generated image, and enhance the overall quality.

[0064] See also Figure 3 ,Temporal excitation introduces an attention mechanism in the time dimension, and dynamically assigns weights to important features in each time period to predict behaviors and actions in future frames. The temporal excitation module is constructed by a series of temporal convolutional layers with RELU activation functions. This invention innovatively introduces the time dimension to process video input, ensuring that dynamic features of different time frames are captured and stimulated.

[0065] First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W), and the global spatial information of the input features is represented by spatial average pooling, which can be expressed as formula (6):

[0066]

[0067] Among them, X represents the input feature map, Represents the compressed feature map, c and t represent the number of channels and time dimension of the input feature map, respectively, and i and j represent the spatial height and width of the input feature map, respectively.

[0068] Then, after obtaining the compressed feature map, two 1×1 convolutional layers are innovatively designed to learn the compressed activation values. The first convolutional layer is used to reduce the number of channels from c to c / b, where b is a hyperparameter that needs to be set manually. The second convolutional layer restores the number of channels from c / b to c. The significance of using two 1×1 convolutional layers is to reduce the complexity of the model and make the network have stronger nonlinear fitting capabilities.

[0069] Finally, the Sigmoid activation function is used to obtain the weight distribution on different time frames, and the dot multiplication is performed with the original feature map to obtain the feature map after time series excitation. The formula (7) of time series excitation is as follows:

[0070]

[0071] Among them, ⊙ represents the dot multiplication operation, which distributes the weights after time series excitation to the original feature map, and Γ represents the learning of activation values, which is completed using two fully connected layers.

[0072] For the video understanding framework, the general input usually uses a tensor of (N, C, T, H, W), where N is the batch size, C is the number of channels, T is the time dimension, and H and W are the spatial resolution of the image. Assuming a convolution operation with a convolution kernel of 3, the input is X, and the weights are W = (w1, w2, w3), the convolution operation can be written as: Y = w1X i-1 +w2X i +w3X i+1 It can be seen that the convolution operation can be performed in two steps: shift and multiply-add. That is, first shift X by -1, 0, +1 respectively and multiply it by the sum of weights W to form Y. The specific operation can be expressed as:

[0073]

[0074] Y=w1X -1 +w2X 0 +w3X +1 (9)

[0075] The shift operation does not require any computation and only requires offsetting the address pointer during training. The multiplication and addition operation requires a lot of computation, but the temporal shift module does not perform this step. Instead, it merges the multiplication and accumulation into the subsequent 2D convolution. Therefore, compared with the 2D-CNN-based model, it does not introduce additional parameters and computational costs.

[0076] Traditional 2D-CNN operates between channels, that is, it operates separately along each row, without time fusion. The time shift module shifts one quarter of the channels by -1, another quarter by +1, and the remaining half along the time dimension. The setting of the number of shifted channels is a hyperparameter.

[0077] Step 4: Introduce joint loss function constraints in the generation process: According to the optimized StyleGAN-V network, adjust the corresponding loss function to ensure the adaptability of the generation task and the model training effect.

[0078] Step 5. Calculate the improved loss function: After completing the forward propagation, use the stochastic gradient descent method to backpropagate the model, continuously adjust the model parameters, and gradually minimize the loss function.

[0079] Step 6: See Figure 4 , update the video generation model parameters, continue training, and determine whether the number of iterations has been reached and converged. If it has been reached, the training is completed to generate the model; if it has not been reached, return to step 1 to continue training.

[0080] Input the sample into the model generated by the above method to generate a video of a severe weather scene and output the generated result. Figure 5 , we can see that the generated video has good effect and effectively improves the authenticity of driving scenes in bad weather.

[0081] As shown in Table 1, DIGAN and LongVideoGAN are classic models in the current video generation field. Compared with the indicators of the original network and other models, the improved algorithm of the present invention has a significant reduction in FVD16 and FVD128 indicators, indicating that the algorithm proposed in the present invention is advanced.

[0082] Table 1

[0083]

[0084] The above is an explanation of the specific implementation of the present invention, rather than a limitation of the present invention. Those skilled in the relevant technical field can also make various equivalent technical solutions without departing from the scope of the present invention, so all equivalent technical solutions should be included in the patent protection scope of the present invention.

Claims

1. A method for generating unmanned driving scenes based on the StyleGAN-V algorithm, characterized in that: The following steps are involved: Step 1: Collect real videos of driving scenes in severe weather, preprocess them, and build a dataset; Step 2: Preliminary construction of the video generation model, the steps include: 2.1 Construct a motion feature extraction module based on a deep autoencoder structure; 2.2 Use the pre-trained parameters to initialize the motion feature extraction module; 2.3 The motion features of the driving scene videos in the dataset are extracted using the motion feature extraction module, and the high-dimensional motion data is mapped to a low-dimensional feature space; 2.4 Use the obtained motion features to constrain the generated image and complete the preliminary construction of the video generation model; Step 3: construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video; Step 4: Introduce joint loss function constraints in the generation process; Step 5: Calculate the improved loss function: After completing the forward propagation, use the stochastic gradient descent method to back-propagate the model, continuously adjust the model parameters, and gradually minimize the loss function; Step 6: Update the video generation model parameters and continue training to determine whether the number of iterations has been reached and converged. If so, the training is completed to generate the model; if not, return to step 1 to continue training.

2. The method for generating unmanned driving scenes based on the StyleGAN-V algorithm according to claim 1, characterized in that: In the step 2.1, the input of the motion feature extraction module combined with the deep autoencoding structure consists of two parts: the current motion T and the previous t frame motion; the motion feature extraction module combined with the deep autoencoding is deformed by the deep autoencoding structure, and the past motion sequence p is added as input in the deep autoencoding structure; the current motion T is the motion data of a single frame, which serves as the x input of the network model, and the previous t frame motion is the motion data of the previous t frames of the current motion T, which serves as the input of the past motion sequence p in the network model.

3. The method for generating unmanned driving scenes based on the StyleGAN-V algorithm according to claim 2, characterized in that: In step 2.4, in the encoding and decoding stage of the deep self-encoding structure, the original motion data is first mapped to the hidden layer features, and then restored to the original motion data to generate reconstructed motion data; by establishing a constraint relationship between the reconstructed data and the input data, the loss function of the network is constructed, which is defined as follows:

4. The method for generating unmanned driving scenes based on the StyleGAN-V algorithm according to claim 3, characterized in that: In step 3, the steps of constructing the timing excitation module are: First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W), and the global spatial information of the input features is represented by spatial average pooling, which can be expressed as formula (6): Among them, X represents the input feature map, represents the compressed feature map, c and t represent the number of channels and time dimension of the input feature map, respectively, and i and j represent the spatial height and width of the input feature map, respectively; Then, the Sigmoid activation function is used to obtain the weight distribution on different time frames, and the weight distribution is multiplied with the original feature map to obtain the feature map after time series excitation. The formula (7) of time series excitation is as follows:

Images