An unmanned driving scene generation method based on a StyleGAN-V algorithm

Abstract

This invention discloses a method for generating autonomous driving scenarios based on the StyleGAN-V algorithm. The implementation steps are as follows: First, collect real-world driving scenario videos in adverse weather conditions and preprocess them to construct a dataset; then, initially construct a video generation model; next, construct a temporal feature extraction module combining temporal excitation and temporal displacement to extract the temporal features of the video; then, introduce a joint loss function constraint during the generation process; finally, calculate the improved loss function, update the parameters of the video generation model, and continue training until the generation model is complete. This invention can realistically simulate driving scenarios in adverse weather conditions. Experimental verification shows that this method can effectively improve the coherence and dynamic feature capture capability of the generated video, not only enhancing the realism of driving simulation scenarios in adverse weather conditions and improving the perception of subtle changes in movement within the video, but also achieving better performance in terms of video dynamic features and image continuity.

Description

Technical Field

[0001] This invention relates to the field of image generation, and more specifically to a method for generating autonomous driving scenes based on the StyleGAN-V algorithm. Background Technology

[0002] With the rapid growth of the global car fleet and the increasing frequency of traffic accidents, autonomous driving technology has become a crucial solution to address traffic safety challenges. Autonomous driving intelligently controls vehicle movement, thereby minimizing traffic accidents, ensuring traffic safety, and improving traffic efficiency. Testing of autonomous driving technology typically employs virtual driving scenarios for verification and training. This virtual scenario testing provides a safe and cost-effective testing environment, improving the generalization ability of its performance. However, driving in adverse weather conditions remains a significant challenge; therefore, simulating severe weather such as rain, snow, and fog in virtual scenarios is crucial for testing the performance of autonomous driving technology in various complex environments.

[0003] Existing methods mostly utilize simulation platforms to create virtual scenes. For example, application number "202410788219.6" discloses "A method, device, and computer program product for generating virtual simulation scene materials for autonomous driving," which creates simulation scenes by inputting relevant keywords and parameters into a pre-trained image generation model. Its drawback is that the generated scenes lack detail and diversity, affecting the realism and applicability of test results. Another application, application number "202210172378.4," discloses "A method and device for generating unmanned driving simulation scenes." This method acquires local environmental images and breaks them down into various environmental element images, then virtually generates lane element images based on a local map, and finally synthesizes these element images into an unmanned driving simulation scene. Its drawback is that there are significant differences between the generated scenes and real-world scenes, especially in generating complex and harsh scenes. A common problem with these solutions is the difficulty in simulating severe weather conditions and the inability to realistically reproduce complex environments and details with high quality. Summary of the Invention

[0004] This invention proposes an autonomous driving scene generation method based on the StyleGAN-V algorithm to solve the problems in existing technologies, such as difficulty in simulating severe weather conditions and inability to reproduce complex environments and details with high quality.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A method for generating autonomous driving scenes based on the StyleGAN-V algorithm, comprising the following steps:

[0006] Step 1: Collect real-world driving scenarios in severe weather, preprocess them, and build a dataset;

[0007] Step 2: Initially construct the video generation model, which includes the following steps:

[0008] 2.1 Construct a motion feature extraction module based on a deep autoencoder structure;

[0009] 2.2 The parameters obtained from pre-training are used to initialize the motion feature extraction module;

[0010] 2.3 The motion features of driving scene videos in the dataset are extracted using the motion feature extraction module, mapping the high-dimensional motion data to a low-dimensional feature space;

[0011] 2.4 The obtained motion features are used to constrain the generated image, completing the initial construction of the video generation model;

[0012] Step 3: Construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video;

[0013] Step 4: Introduce joint loss function constraints during the generation process;

[0014] Step 5: Calculate the improved loss function: After completing the forward propagation, use the stochastic gradient descent method to backpropagate the model, continuously adjust the model parameters, and gradually minimize the loss function;

[0015] Step 6: Update the video generation model parameters, continue training, and determine whether the number of iterations has been reached and convergence has occurred. If it has, the training of the generation model is complete; otherwise, return to Step 1 to continue training.

[0016] Furthermore, in step 2.1 above, the input of the motion feature extraction module combined with the deep autoencoder structure consists of two parts: the current motion T and the motion of the previous t frames; the motion feature extraction module combined with the deep autoencoder structure is a modified deep autoencoder structure, in which the past motion sequence p is added as input; the current motion T is the motion data of a single frame, which serves as the x input of the network model, and the motion of the previous t frames is the motion data of the previous t frames of the current motion T, which serves as the input of the past motion sequence p in the network model.

[0017] Furthermore, in step 2.4 above, during the encoding and decoding stages of the deep autoencoder structure, the original motion data is first mapped to hidden layer features, and then restored to the original motion data to generate reconstructed motion data. By establishing constraints between the reconstructed data and the input data, the network's loss function is constructed and defined as follows:

[0018]

[0019] Furthermore, in step three above, the steps for constructing the timing stimulus module are as follows:

[0020] First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W). The global spatial information of the input features is represented by spatial average pooling, which can be expressed as Equation (6):

[0021]

[0022] Where X represents the input feature map, This represents the compressed feature map, where c and t represent the number of channels and the time dimension of the input feature map, respectively, and i and j represent the spatial height and width of the input feature map, respectively.

[0023] Then, the Sigmoid activation function is used to obtain the weight distribution at different time frames, and the dot product is performed with the original feature map to obtain the feature map after temporal activation. The formula (7) for temporal activation is shown below:

[0024]

[0025] Compared with existing technologies, the beneficial effects of this invention are:

[0026] (1) This invention uses videos of driving scenarios in severe weather for training and learning, which can realistically simulate driving scenarios in severe weather. It can be used for virtual simulation testing of unmanned driving, avoiding the problems of difficulty in simulating severe weather conditions and inability to generate severe weather environments caused by existing simulation platforms.

[0027] (2) To address the limited ability of video generation models to capture motion features, this invention constructs a motion feature extraction module based on deep autoencoders. The method of this invention utilizes motion data from both the current and previous frames for feature extraction. Due to the introduction of temporal information and autoencoders, it provides more reliable feature representation capabilities. By learning the features of the original motion data, the quality of video generation and the continuity of motion in the frame are improved.

[0028] (3) To address issues such as texture adhesion and visual anomalies in video generation models, this invention constructs a temporal feature extraction module that combines temporal excitation and temporal displacement. The temporal feature extraction module in this invention can dynamically assign weights to features from different time frames and introduce temporal displacement operations. By combining temporal excitation and temporal displacement operations, the fusion and interaction of temporal information can be promoted, thereby better capturing temporal changes and dynamic features in the video and improving the quality and realism of the generated video. In step three of this invention, the feature map after temporal excitation expands the original feature map along the time and channel dimensions. Because the temporal excitation module does not act on specific spatial locations but on different temporal locations, it does not display specific spatial dimensions. Through this innovative design, the temporal excitation module can effectively improve the coherence of the generated video and the ability to capture dynamic features. The core idea of ​​the temporal displacement structure constructed in step three is to divide the time dimension into multiple segments and perform displacement operations within each segment. This allows the model to have better temporal modeling capabilities and innovatively improves the perception of subtle changes in movement within the video.

[0029] (4) The channel-based feature shifting method designed in this invention mixes the feature information of adjacent time frames together, which is equivalent to implementing a convolution operation in the time dimension. The fusion of time information is achieved through the inter-channel fusion capability of subsequent 2D convolution. After combining temporal excitation and temporal displacement, the final temporal feature extraction module is obtained. The fusion module assigns different activation weights to different times, and then through temporal shifting, adjacent frames can directly obtain some information from each other. This design not only preserves features with large changes, but also allows different time features to interact directly and learn temporal feature information. This method combines video training for severe weather scenarios, a deep autoencoder motion feature extraction module, and a temporal feature extraction module based on temporal excitation and temporal displacement, thus showing significant advantages in generating autonomous driving severe weather scenarios. This invention not only improves the realism of driving simulation scenarios under severe weather conditions, but also achieves better performance in terms of video dynamic features and image continuity. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the present invention;

[0031] Figure 2 This is a diagram of a deep autoencoder structure;

[0032] Figure 3 This is a structural diagram of the timing excitation module;

[0033] Figure 4 Network training flowchart;

[0034] Figure 5This is a schematic diagram of an autonomous driving scenario in adverse weather conditions, generated in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying embodiments. Obviously, the described embodiments are only some embodiments of this invention and are used only to illustrate the invention, but are not intended to limit the scope of the invention.

[0036] See Figure 1 The basic idea of ​​this invention is as follows: First, collect videos of various severe weather conditions and preprocess these videos to establish a dataset; second, construct a motion feature extraction module combined with deep autoencoders to improve the original motion feature extraction method and enhance feature representation capabilities; next, construct a temporal feature extraction module combining temporal excitation and temporal displacement to constrain the generation and improve the coherence of the generated images; finally, guide the training of the generator network by fine-tuning the model and constraining the generation with a joint loss function to obtain better generation results.

[0037] Based on the above basic ideas, this invention provides a method for generating severe weather scenes for autonomous driving based on an improved StyleGAN-V algorithm. The StyleGAN-V algorithm performs excellently in generating videos and can produce high-quality traffic scenes. The specific implementation steps are as follows.

[0038] Step 1: Collect real-world videos of driving in severe weather, preprocess them, and construct a dataset. Details are as follows:

[0039] Step 1.1 The real-world driving scenario videos in severe weather should include video clips from different cities and various weather conditions to ensure the diversity and richness of the training data.

[0040] Step 1.2 Video preprocessing employs inter-frame interpolation. Inter-frame interpolation generates more video frames by inserting new frames between existing frames, making the video appear smoother. This not only increases the video's frame rate and improves its smoothness but also enhances the richness and diversity of the video dataset, especially when the amount of data is limited.

[0041] Step 2: Initially construct the video generation model, which includes the following steps:

[0042] Step 2.1 Constructs a motion feature extraction module based on a deep autoencoder structure, and uses it as a component of generative adversarial network (GAN) motion feature capture and motion transfer. The motion transfer network establishes motion synthesis constraints and uses the motion features obtained from the motion extraction module for motion transfer synthesis to improve the quality of motion synthesis. The specific construction process is as follows:

[0043] Well-known deep autoencoder structures, such as Figure 2 As seen, the input to the motion feature extraction module of this invention, which incorporates a known deep autoencoder structure, consists of two parts: the current motion T and the motion of the previous t frames; the main network is formed by a transformation of the autoencoder, within the deep autoencoder structure.

[0044] This method innovatively adds past motion sequences p as input. The current motion T is the motion data of a single frame, which serves as the x input to the network model designed in this method. The motion of the previous t frames is the motion data of the previous t frames of the current motion T, which serves as the input to the past motion sequence p in the network model, so that the network can extract the temporal features of the motion sequence. The entire network model can be divided into an encoding stage and a decoding stage.

[0045] In the encoding phase of the deep autoencoder structure, the network maps the raw motion data into the feature space, based on the current motion x∈R. n×d and past motion sequences p∈R n×(d×t) Let t be the motion sequence of the previous t frames before the current motion T. Finally, the function definition of the network in the encoding stage is obtained as:

[0046]

[0047] Where, h∈R n×c This represents the hidden layer features of the original motion data; W xf ∈R d×k The weights connecting the current motion x and the current encoding f; W1 fh ∈R k×c The weights connecting the current encoding f with the hidden feature h: W pf ∈R (d×t)×k The weights connecting the past motion sequence p to the current encoding f; W2 fh ∈R k×c The weights connecting the past encoding f and the hidden feature h; b1∈R c is the bias vector; ReLU is the nonlinear activation function.

[0048] During the decoding stage of the deep autoencoder structure, the network reconstructs the original motion space from the obtained hidden layer motion features h, thus obtaining a motion capture reconstruction. The function for its decoding stage is:

[0049]

[0050] in, The reconstructed output obtained by combining the motion feature extraction module of the deep autoencoder structure; W xf '∈R c×k The weights connecting the hidden layer feature h to the current decoded feature f'; To connect the current decoding f′ with the reconstruction motion Weights; W pf '∈R (d×t)×k The weights that connect the past motion sequence to the past decoded f′; To connect the past decoding f′ with the reconstructed motion The weights; b2∈R d This is the bias vector; the selection of k, n, d, and t is the same as in the encoding stage.

[0051] Step 2.2 The parameters obtained from pre-training are used to initialize the feature extraction module;

[0052] The parameters of the feature extraction model are pre-trained using a network to enable rapid convergence. Past motion sequences are added to the network structure, where ν represents the visible layer, h represents the hidden layer, and p represents the past motion sequence. The energy function is defined as:

[0053]

[0054] Where θ represents the parameters of the model; ν i For the data of the i-th visible layer node; h j For the j-th hidden layer node data; a i b j The bias is set for the i-th visible layer and the j-th hidden layer; To connect the visible layer ν with the weights of the visible encoding, The weights that connect the visible encoding to the hidden layer; Weights that connect past motion sequences to past encodings; The weights of the past encoding and hidden layers.

[0055] During training, given the state transitions of neurons in the visible layer, past motion sequences P, and parameters θ, the calculations for neurons in the hidden layer are as follows:

[0056]

[0057] Wherein, σ(·) is the activation function designed in this method, using sigmoid; the parameters of the motion feature extraction module combined with deep autoencoder can be obtained through the pre-trained model described in the above formula (4). Through this innovative training mechanism, the model can converge quickly and obtain more accurate feature extraction results.

[0058] Step 2.3 Use the feature extraction module to extract features, mapping the high-dimensional motion data to a low-dimensional feature space;

[0059] Step 2.4 Establish constraints using the obtained motion features:

[0060] In the encoding and decoding stages of the network, the original motion data is first mapped to hidden layer features, and then restored to the original motion data to generate reconstructed motion data. By establishing constraints between the reconstructed data and the input data, the network's loss function is constructed and defined as follows:

[0061]

[0062] Where l(x,θ) is the loss function, representing the network loss; θ represents the network parameters; and x is the input motion. To output motion; The first term of the loss function is the mean squared error of the input and reconstructed motions, representing the difference between the input and output. λ||θ||1 is the regularization term, representing a constraint on the network parameters to prevent overfitting. This first term forces the network's input and output to be identical, allowing the hidden features to be represented as the feature space of the original motion. Furthermore, by incorporating past motion sequences during feature extraction, the extracted features not only possess spatial structure but also temporal correlation.

[0063] Step 3: Construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video, improve the coherence of the generated images, and enhance the overall quality.

[0064] See Figure 3 Temporal activation introduces an attention mechanism in the temporal dimension, dynamically assigning weights to important features within each time frame to predict behaviors and actions in future frames. The temporal activation module is constructed from a series of temporal convolutional layers with ReLU activation functions. This invention innovatively introduces the temporal dimension to process video input, ensuring that dynamic features from different time frames are captured and activated.

[0065] First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W). The global spatial information of the input features is represented by spatial average pooling, which can be expressed as Equation (6):

[0066]

[0067] Where X represents the input feature map, This represents the compressed feature map, where c and t represent the number of channels and the time dimension of the input feature map, respectively, and i and j represent the spatial height and width of the input feature map, respectively.

[0068] Then, after obtaining the compressed feature map, two 1×1 convolutional layers are innovatively designed to learn the compressed activation values. The first convolutional layer is used to reduce the number of channels from c to c / b, where b is a hyperparameter that needs to be set manually. The second convolutional layer restores the number of channels from c / b back to c. The significance of using two 1×1 convolutional layers is to reduce the model complexity and enable the network to have a stronger non-linear fitting ability.

[0069] Finally, the Sigmoid activation function is used to obtain the weight distribution at different time frames, and then multiplied by the original feature map to obtain the feature map after temporal activation. The formula (7) for temporal activation is shown below:

[0070]

[0071] Here, ⊙ represents the dot product operation, which distributes the weights after temporal activation onto the original feature map, and Γ represents the learning of activation values, which is accomplished using two fully connected layers.

[0072] For video understanding frameworks, the input typically takes the form of a tensor of (N, C, T, H, W), where N is the batch size, C is the number of channels, T is the time dimension, and H and W are the spatial resolution of the image. Assuming a convolution operation with a 3-kernel designation, input X, and weights W = (w1, w2, w3), the convolution operation can be written as: Y = w1X i-1 +w2X i +w3X i+1 It can be seen that the convolution operation can be divided into two steps: shifting and multiplying / adding. That is, X is first shifted by -1, 0, and +1 respectively, and then multiplied by the sum of the weights W to form Y. The specific operation can be represented as:

[0073]

[0074] Y = w1X -1 +w2X 0 +w3X +1 (9)

[0075] Shift operations require no computation; they only require shifting the address pointer during training. Multiply-accumulate operations require a lot of computation, but the time shift module does not perform this step. Instead, it combines multiply-accumulate into the subsequent 2D convolution. Therefore, compared with 2D-CNN-based models, it does not introduce additional parameters and computational costs.

[0076] Traditional 2D-CNNs operate between channels, that is, they run separately along each row, without temporal fusion. The temporal shifting module shifts one-quarter of the channels by -1 along the time dimension, shifts another quarter by +1, and leaves the remaining half unshifted. The setting of the number of shifted channels is a hyperparameter.

[0077] Step 4: Introduce joint loss function constraints during the generation process: Adjust the corresponding loss function according to the optimized StyleGAN-V network to ensure the adaptability of the generation task and the model training effect.

[0078] Step 5: Calculate the improved loss function: After completing the forward propagation, the model is backpropagated using the stochastic gradient descent method. The model parameters are continuously adjusted to gradually minimize the loss function.

[0079] Step Six, see Figure 4 Update the video generation model parameters, continue training, and determine whether the number of iterations has been reached and converged. If it has, the training of the generation model is complete; otherwise, return to step one to continue training.

[0080] Input the samples into the model generated using the above method to produce severe weather scene videos, and output the generated results. See [link / reference]. Figure 5 As can be seen, the generated video has good quality and effectively improves the realism of driving scenarios in bad weather.

[0081] As shown in Table 1, DIGAN and LongVideoGAN are classic models in the current video generation field. Compared with the original network and other models, the improved algorithm of this invention has a significant reduction in FVD16 and FVD128, indicating that the algorithm proposed in this invention is advanced.

[0082] Table 1

[0083]

[0084] The above description is a specific illustration of the present invention, and not a limitation thereof. Those skilled in the art can make various equivalent technical solutions without departing from the scope of the present invention; therefore, all equivalent technical solutions should fall within the patent protection scope of the present invention.

Claims

1. A method for generating autonomous driving scenarios based on the StyleGAN-V algorithm, characterized in that: Includes the following steps: Step 1: Collect real-world driving scenarios in severe weather, preprocess them, and build a dataset; Step 2: Initially construct the video generation model, which includes the following steps: 2.1 Construct a motion feature extraction module based on a deep autoencoder structure; In step 2.1, the input of the motion feature extraction module combined with the deep autoencoder structure consists of two parts: the current motion T and the motion of the previous t frames; the motion feature extraction module combined with the deep autoencoder structure is a modified deep autoencoder structure, in which a past motion sequence p is added as input; the current motion T is the motion data of a single frame, which is used as the x input of the network model, and the motion of the previous t frames is the motion data of the previous t frames of the current motion T, which is used as the input of the past motion sequence p in the network model; 2.2 The parameters obtained from pre-training are used to initialize the motion feature extraction module; 2.3 The motion features of driving scene videos in the dataset are extracted using the motion feature extraction module, mapping the high-dimensional motion data to a low-dimensional feature space; 2.4 The obtained motion features are used to constrain the generated image, completing the initial construction of the video generation model; In step 2.4, during the encoding and decoding stages of the deep autoencoder structure, the original motion data is first mapped to hidden layer features, and then restored to the original motion data to generate reconstructed motion data. By establishing constraints between the reconstructed data and the input data, the network's loss function is constructed, defined as follows: Where l(x,θ) is the loss function, representing the network loss; θ represents the network parameters; and x is the input motion. To output motion; λ||θ||1 is the mean squared error term for the input motion and the reconstructed motion, representing the difference between the input and the output; λ||θ||1 is the regularization term, representing the constraint on the network parameters, used to prevent overfitting. Step 3: Construct a temporal feature extraction module that combines temporal excitation and temporal displacement to extract the temporal features of the video; In step three, the steps for constructing the timing stimulus module are as follows: First, the two-dimensional input (N*T,C,H,W) is reshaped into (N,C,T,H,W). The global spatial information of the input features is represented by spatial average pooling, which can be expressed as: Where X represents the input feature map, This represents the compressed feature map, where c and t represent the number of channels and the time dimension of the input feature map, respectively; i and j represent the spatial height and width of the input feature map, respectively; H and W are the spatial resolution of the image; N is the batch size; C is the number of channels; and T is the time dimension. Then, the Sigmoid activation function is used to obtain the weight distribution at different time frames, and a dot product is performed with the original feature map to obtain the temporally activated feature map. The formula for temporal activation is shown below: Where ⊙ represents the dot product operation, which distributes the weights after temporal activation to the original feature map, and Γ represents the learning of activation values, which is accomplished using two fully connected layers; Step 4: Introduce joint loss function constraints during the generation process; Step 5: Calculate the improved loss function: After completing the forward propagation, use the stochastic gradient descent method to backpropagate the model, continuously adjust the model parameters, and gradually minimize the loss function; Step 6: Update the video generation model parameters, continue training, and determine whether the number of iterations has been reached and convergence has occurred. If it has, the training of the generation model is complete; otherwise, return to Step 1 to continue training.

Images