A high-risk driving scene recognition method based on GAN data enhancement

Abstract

The application relates to a high-risk driving scene recognition method based on GAN data enhancement, which comprises the following steps: generating a first motion contour graph based on vehicle forward video data, dividing the first motion contour graph into a training set part, a verification set part and a test set part; performing enhancement processing on the first motion contour graph of the training set part; obtaining a second motion contour graph; inputting the second motion contour graph into the GAN model to output a third motion contour graph; inputting the first motion contour graph, the second motion contour graph and the third motion contour graph into the high-risk driving scene recognition model for training; and inputting the first motion contour graph into the high-risk driving scene recognition model for identification. Compared with the prior art, the application expands high-risk driving scene samples through GAN data enhancement, improves the richness and diversity of modeling data, and further improves the generalization ability of the model, and has the advantages of universality and the like.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving algorithms, and in particular to a method for identifying high-risk driving scenarios based on GAN data augmentation. Background Technology

[0002] Extracting high-risk driving scenarios from natural driving data has become an important method for constructing test scenario libraries for autonomous driving systems. Existing research proposes using deep learning algorithms to build a driving scenario risk identification model based on in-vehicle forward-facing video to extract high-risk driving scenarios. The training samples of the model consist of non-conflict driving scenario samples and high-risk driving scenario samples. However, due to the sporadic and low-probability nature of high-risk traffic events, the number of high-risk driving scenarios collected in a short period is small, the scenario coverage is low, and the training samples lack sufficiency and diversity. These data deficiencies greatly affect the model's generalization ability and robustness.

[0003] In existing technologies, the lack of sufficient and diverse samples also leads to the problem of limited model complexity. Compared with lightweight CNNs, complex DNN models are more prone to overfitting. This has led to the frequent use of lightweight models in tasks that identify high-risk driving scenarios. For example, the published paper "Predicting Hazardous Driving Events Using Multimodal Deep Learning Based on Video Motion Profile and Kinematics Data" proposes using AlexNet to identify high risks, while "A Lightweight VK-Net Based on Motion Profiles for Hazardous Driving Scenario Identification" proposes a lightweight CNN with an attention mechanism that can reduce the cost of video analysis while maintaining the accuracy of high-risk scene identification. The limited complexity of the models published in these two papers alleviates the overfitting phenomenon to some extent, but at the same time, it sacrifices the powerful feature extraction capabilities and nonlinear relationship expression capabilities of complex models.

[0004] Generative Adversarial Networks (GANs) are commonly used data augmentation techniques. However, due to the limited number of samples in dangerous driving scenarios, directly using GANs to generate new samples results in poor performance and a large distribution distance from the original samples. How to use GANs to generate new samples with the same distribution as the original samples under limited sample conditions is a challenge, especially in the case of small samples. It is necessary to study how to comprehensively utilize these small samples to enhance the quality of the generated data and use the enhanced samples as training data to improve the accurate identification of dangerous driving scenarios.

[0005] Therefore, in order to address the problem of balancing the powerful feature extraction capabilities and nonlinear relationship expression capabilities of complex models, and the problem of how to enhance the quality of GAN-generated images with limited samples, this invention provides a high-risk driving scene identification method based on GAN data augmentation. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a high-risk driving scenario identification method based on GAN data augmentation.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides a method for identifying high-risk driving scenarios based on GAN data augmentation, comprising the following steps:

[0009] Step S1: Obtain vehicle forward video data, generate a first motion contour map based on the vehicle forward video data, and divide the first motion contour map into a training set part, a validation set part and a test set part.

[0010] Step S2: Enhance the first motion contour map of the training set; to obtain the second motion contour map;

[0011] Step S3: Obtain the GAN model, input the second motion contour map into the GAN model, and output the third motion contour map;

[0012] Step S4: Based on the deep convolutional neural network architecture, construct a high-risk driving scene identification model; input the training set portion of the first motion contour map, the second motion contour map and the third motion contour map into the high-risk driving scene identification model for training to obtain the high-risk driving scene identification model;

[0013] Step S5: Input the first motion contour map into the high-risk driving scenario identification model. After calculation by the high-risk driving scenario identification model, a continuous risk value is output. The driving risk is evaluated based on the relationship between the continuous risk value and the optimal classification threshold determined by the Youden index method.

[0014] If the continuous risk value is greater than the optimal classification threshold, then the first motion profile is a high-risk driving scenario; otherwise, it is a non-conflict driving scenario.

[0015] Furthermore, obtaining the GAN model includes constructing the GAN model, training the GAN model, validating the GAN model, and applying the GAN model.

[0016] Furthermore, the construction of the GAN model specifically includes:

[0017] Step S301: Construct a deconvolution module, which is used to generate an image;

[0018] Step S302: Stack multiple deconvolution modules to form a generator network;

[0019] Step S303: Construct a convolutional module, which is used to distinguish between real and fake images;

[0020] Step S304: Stack multiple convolutional modules to form a discriminator network.

[0021] Furthermore, training the GAN model includes training the generator network and the discriminator network; during the training of the generator network and the discriminator network, the iteration period and batch size of the training process are set.

[0022] One iteration cycle specifically includes the following steps:

[0023] Step S311: Sample the second motion contour map with the batch size to obtain the real image of the current batch;

[0024] Step S312: Based on the real image, obtain the corresponding generated image through the generator network;

[0025] Step S313: Introduce the overfitting factor γ to obtain the pseudo-enhancement probability p;

[0026] Step S314: Based on the real image, the generated image, and the pseudo-enhancement probability p, construct a pseudo-enhancement dataset, as shown in the following formula:

[0027] x real_apa =M apa ×x fake +(1-M apa 0×x real

[0028] Where, x real_apa M is a pseudo-augmented dataset. apa Introduce a matrix, x, for pseudo-enhancement. real For a real image, x fake To generate an image;

[0029] Step S315: Fix the parameters of the discriminator network, train the generator network, design the loss function of the generator, substitute the generated image into the loss function, and optimize the parameters of the generator network. The loss function of the generator is shown in the following formula:

[0030]

[0031] Among them, L G Let be the loss function of the generator. The expected output value of the discriminator for the generated image;

[0032] Step S316: Fix the parameters of the generator network gen, train the discriminator network, design the loss function of the discriminator, input the generated image and the pseudo-data augmented image into the loss function, and optimize the parameters of the discriminator network;

[0033] Step S317: When the iteration period is greater than the preset iteration period, save the generator network and the discriminator network to complete the training of the GAN model.

[0034] Furthermore, the overfitting factor γ is also used to determine whether the discriminator network has an overfitting problem. The formula for calculating the overfitting factor γ is as follows:

[0035]

[0036] Furthermore, the loss function of the discriminator network also includes a gradient penalty term, as shown in the following formula:

[0037]

[0038] in, The discriminator's expected output value for the generated image, L represents the expected output value of the discriminator for the real image. gp This is the gradient penalty term.

[0039] Furthermore, the verification of the GAN model includes: using the image quality assessment index FID to measure the difference between the third motion contour map and the first motion contour map output by the GAN model.

[0040] Furthermore, the application of the GAN model includes the following steps:

[0041] Step S321: Based on the second motion contour map, randomly generate a noise vector with the same size as the training set of the second motion contour map;

[0042] Step S322: Input the noise vector into the generator network and output the third motion contour map.

[0043] Furthermore, the construction of the high-risk driving scenario identification model includes the following steps:

[0044] Step S401: Construct an initial convolutional layer, which includes: a convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer. The initial convolutional layer is used for input.

[0045] Step S402: Construct a residual block, wherein the residual block includes: the main path and the shortcut;

[0046] Step S403: Establish a multi-layer residual network based on the residual block;

[0047] Step S404: Construct a global average pooling layer and a fully connected layer, which are used for output.

[0048] Furthermore, the loss of the high-risk driving scenario identification model on the validation set is continuously increasing. If the number of consecutive increases exceeds the preset threshold of the early stop mechanism, training is stopped and the high-risk driving scenario identification model is saved.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) This invention uses GAN technology to augment driving scenario samples and uses the augmented samples as training data to improve the generalization ability of the model.

[0051] (2) By introducing gradient penalty into the loss function of the discriminator network in the GAN model, this invention can balance the gradient descent speed of the discriminator, prevent the discriminator from experiencing gradient explosion during training and failing to converge. As a constraint, it can improve the original GAN ​​model and distinguish the linear interpolation points between real samples and generated samples when updating the discriminator network.

[0052] (3) This invention effectively enhances the quality of generated data by integrating DIG (Discriminator Gradient Norm Difference) and APA (Adaptive Pseudo-Enhancement) techniques into the original GAN ​​model.

[0053] (4) By using the image quality assessment index FID, the present invention can effectively measure the distribution difference between the third motion contour map and the first motion contour map.

[0054] (5) This invention uses ResNet network to construct a high-risk driving scenario identification model, which makes full use of the high-level feature extraction capability and nonlinear relationship expression capability of complex models, and introduces enhanced samples into the training set to improve the accuracy of high-risk driving scenario assessment.

[0055] (6) The present invention uses video information extraction and compression technology to compress the vehicle-mounted forward video to generate a motion contour map, thus solving the problem of high computing cost. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0057] Figure 2This is a schematic diagram of the forward video data acquisition of the vehicle in this embodiment;

[0058] Figure 3 This is a schematic diagram of the first motion contour map generation process in this embodiment;

[0059] Figure 4 This is a schematic diagram of a sample generated using a GAN model in this embodiment;

[0060] Figure 5 This is a comparison chart of FID changes for different GAN models in this embodiment;

[0061] Figure 6 This is a diagram of the enhanced GAN network architecture proposed in this embodiment;

[0062] Figure 7 This is a graph showing the change in the AUC metric of the data augmentation model in this embodiment on the test set. Detailed Implementation

[0063] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0064] Example

[0065] This embodiment provides a method for identifying high-risk driving scenarios based on GAN data augmentation, such as... Figure 1 The method includes the following steps:

[0066] Step 1: Based on the vehicle-mounted forward video data collected during natural driving, a rectangular area is sampled for each frame in the driving area. Then, the pixel average of different channels within the rectangular area is calculated and superimposed on the channels in the vertical direction. Finally, the pixel matrix of each frame is stitched together in chronological order to form a motion contour matrix, called the first motion contour map. The map contains the trajectory information of various objects in the driving scene for a certain period of time, including the relative motion trajectories of traffic facilities, roadside buildings, traffic participants, etc.

[0067] Specifically, the detailed process of step 1 is as follows:

[0068] Step 1.1: Based on the principle of perspective, the scenery within the driver's line of sight eventually converges to a single point during driving. Therefore, selecting the scenery below the horizon at that point can encompass all events, including those related to people, involved in the basic driving process. For each recorded forward-facing video segment, the area below the horizon can be divided into three distinct regions—far, middle, and near—based on its distance from the horizon, to analyze the temporal trajectory of changes at different locations. Each region is defined by two boundaries (upper boundary y...). lf y l m and y l c The lower boundary y u f y u m and y u c ) definition;

[0069] Step 1.2: Based on the driver's rapid acceleration and deceleration behaviors in the driving video data, select the time interval [t] near the time of occurrence. a ,t b The video segment is analyzed. Within this time range, the video data from the three different areas at varying distances in step 1.1 are analyzed at fps (frame rate), w (sampling width), and y (data depth). l i With y u i Data sampling is performed using the upper and lower boundaries (i∈{f,m,c}) as the sampling height. Using this method, fps×(t) can be obtained. b -t a ) Near and far rectangular regions I i f I i m I i c (i∈[0,fps×(t b -t a In this embodiment, fps is set to 4, w is 368 pixels, [t] a ,t b The data collection process is based on the high deceleration behavior of autonomous driving (acceleration less than -0.3g), such as... Figure 2 As shown, during this process, data is collected from 8 seconds before the current time to 4 seconds after the current time. (t) b -t a The specific method for generating a motion contour map using the sampled rectangular region, with a time span of 12 seconds, is as follows:

[0070] Step 1.2.1: For the rectangular region I at near and far distances... i f I i m I i c Separate based on different RGB channels to form matrix representations of different channels for rectangular regions at different distances (y u i -y l i ,w,3);

[0071] Step 1.2.2: For each channel matrix representation (y) in each near-far rectangular region. u i -y l i By calculating the average vertical pixel value under this matrix, pixel lines L formed by each near and far region are generated. i f L i m L i c (i∈[1,fps×(t b -t a )]);

[0072] Step 1.2.3: Stitch together the pixel lines formed by these near and far rectangular regions in chronological order to generate a compressed near-far image (MP) from different channels. f MP m MP c Based on this, a colored first motion profile is created, for example, the generated first motion profile. Figure 3 As shown:

[0073]

[0074] Step 1.3: Watch the in-vehicle forward-facing video and, based on the thresholds for rapid acceleration and deceleration and expert experience, calibrate the first motion profile data generated in Step 1.2. In this embodiment, after data sampling and calibration, a total of 1366 driving events were extracted, including 530 dangerous driving scenarios and 836 non-conflict driving scenarios. The calibrated first motion profile data was then split into a training set Θ in a 7:1:2 ratio. train , Validation set Θ valid With test set Θ test .

[0075] Step 2: Enhance the first motion contour map, which was divided into the training set in Step 1; to obtain the second motion contour map.

[0076] Each motion contour map corresponds to a driving scene. Image processing techniques are used to process the first motion contour map to simulate changes in lighting intensity and color changes of traffic participants in the driving scene, thereby enhancing the richness of the driving scene and achieving data augmentation. Specific image processing includes randomly transforming the brightness, contrast, saturation, and hue of the motion contour map; and flipping the motion contour map horizontally with a certain probability.

[0077] Specifically, the detailed process of step 2 is as follows:

[0078] Step 2.1: Select the first motion contour image, label it image im1, and then apply the brightness adjustment factor factor. b The image is modified. The adjusted image is labeled as im2, where im2 is the result of combining the original image im1 with the brightness coefficient factor. b The result of multiplication:

[0079] im2 = factor b ×im1

[0080] In this embodiment, factor b Set to [0.6, 1.4];

[0081] Step 2.2: For image im2, first generate its grayscale version, denoted as gray2. Then, use the saturation adjustment factor (factor)... S Image im2 is adjusted to generate a new image im3. This image is the result of combining the original color image im2 with its grayscale version gray2, where the coefficients are used to adjust the image. s To balance the ratio of color to grayscale:

[0082] im3 = factor s ×im2+(1-factor s )×gray2

[0083] In this embodiment, factor s Set to [0.8, 1.2];

[0084] Step 2.3: Convert image im3 to grayscale to calculate its average brightness value mean3. Then, use the contrast adjustment factor factor. c To generate image im4, which is generated by combining im3 with its brightness mean3, according to the factor... c The result of adjusting the contrast:

[0085] im4 = factor c ×im3+(1-factor c )×mean3

[0086] In this embodiment, factor c Set to [0.8, 1.2];

[0087] Step 2.4: First, convert the image im4 to HSV format to obtain its original hue H. origin Applying the hue adjustment factor h To modify the hue value and generate a new hue H newThen, the function HSV_to_RGB is used to convert the modified hue values ​​back to RGB format, resulting in the updated image im5:

[0088] H new =H origin +factor h ×255

[0089] im5 = HSV_to_RGB(H new )

[0090] In this embodiment, factor h Set to [-0.1, 0.1];

[0091] Step 2.5: Using a certain probability value flip_prob, horizontally flip image im5 using the function horizontal_flip to generate image im6.

[0092] im6=horizontal_flip(im5,flip_prob)

[0093] In this embodiment, flip_prob is set to 0.5.

[0094] Step 3: Obtain the GAN model. Input the second motion contour map into the GAN model and output the third motion contour map. The GAN model is obtained through the following steps: constructing the GAN model, training the GAN model, validating the GAN model, and applying the GAN model.

[0095] Based on the enhanced second motion contour map samples from step 2, a Generative Adversarial Nets (GAN) model is trained to generate a new sample—the third motion contour map—that is distributed in the same way as the first motion contour map collected from natural driving, in order to enhance the diversity of driving scenarios. In this process, spectral normalization, Wasserstein distance and gradient penalty, gradient norm constraint, and pseudo-data augmentation are used to improve the image quality of the generated third motion contour map.

[0096] Specifically, the detailed process of step 3 is as follows:

[0097] Step 3.1: Construct the network structure of the GAN model for data augmentation. The detailed process is shown below:

[0098] Step 3.1.1: Construct the deconvolution module gen_layer for generating images. This module includes a deconvolution layer, a batch normalization layer, and an activation layer. In this process, it is necessary to set the number of input channels, the number of output channels, the deconvolution kernel size, the deconvolution stride, whether to fill the boundary during deconvolution, and the selection of the activation function.

[0099] Step 3.1.2: Construct the convolutional module disc_layer for determining the authenticity of images. This module includes convolutional layers, batch normalization layers, and activation layers. In this process, it is necessary to set the number of input channels, the number of output channels, the deconvolution kernel size, the deconvolution stride, whether to fill the boundary during deconvolution, and the selection of activation functions.

[0100] Step 3.1.3: Construct a generator network gen based on the deconvolution module gen_layer. Specifically, the generator network is composed of multiple stacked deconvolution modules gen_layer. During this process, the number of deconvolution modules needs to be set, and the parameters of each layer's deconvolution module need to be set according to step 3.1.1. In this embodiment, gen constructs 5 layers of deconvolution modules gen_layer. The first layer has 102 input channels, 512 output channels, a kernel size of 4, a stride of 1, and does not fill boundaries during convolution. The activation function used is the ReLU function.

[0101] Step 3.1.4: Construct the discriminator network disc based on the convolutional modules disc_layer. Specifically, the discriminator network is composed of multiple stacked convolutional modules disc_layer. During this process, the number of convolutional modules needs to be set, and the parameters of each layer's convolutional modules need to be set according to Step 3.1.2. In this embodiment, the disc constructs 4 layers of convolutional modules disc_layer. The first layer has 3 input channels, 64 output channels, a kernel size of 4, a stride of 2, and a padding size of 1. The activation function used is the LeakyReLU function.

[0102] Step 3.2: Train the constructed GAN model, mainly including training the constructed generator network (gen) and discriminator network (disc). In this process, the training iteration period n needs to be set. epoch_GAN In this embodiment, n is related to the batch size (batch_size). epoch_GAN Set to 1000, batch_size to 64, such as Figure 6 The diagram shows the GAN network architecture under small sample data. The specific training steps for each epoch are as follows:

[0103] Step 3.2.1: Sample the second motion contour map with a batch size of batch_size to obtain the real image x of the current batch. real And the corresponding label y.

[0104] Step 3.2.2: Randomly generate a real image x that matches the current batch. real Each noise vector has the same size and dimension z. To apply the generated image—the third motion contour map—to augment training, images of the corresponding categories need to be generated based on the labels y of the current batch. One-hot encoding is used to encode the label information into feature vectors y. vector Then noise and y vector The noise vector of the generator is constructed by concatenating the noise vectors in sequence. ′ By splicing the noise ′ The input is fed into the generator network gen to obtain the generated image x for the current batch. fake ,like Figure 4 As shown, its corresponding label is also y. In this embodiment, z is 100. Since there are two types of data labels in this embodiment, the noise input to the generator gen is... ′ The size is 102.

[0105] Step 3.2.3: Employ Adaptive Pseudo Augmentation (APA) based on the real image x real With the generated image x fake Constructing pseudo-augmented data x real_apa To prevent the introduction of excessive generated images from negatively impacting the discriminator network training, the proportion of generated images (the intensity of pseudo-data augmentation) is actually calculated based on the degree of overfitting in the model. An overfitting factor γ is introduced to simply quantify whether the discriminator is overfitting. The calculation formula is as follows:

[0106]

[0107] The calculated overfitting factor γ is then compared with a preset threshold t for judgment. If γ is greater than t, the discriminator is considered to be overfitting; otherwise, it is not. The strength of pseudo-data augmentation is directly controlled by the probability p of introducing pseudo-augmentation. The strategy for adjusting the probability p of introducing pseudo-augmentation using γ is as follows: p is initialized to zero. If, after quantization by γ, it is determined that the discriminator model has / has not overfitted, the probability p will increase / decrease by a fixed step size. Using this step size allows p to dynamically slide within the range of 0 to 1 according to the degree of overfitting of the discriminator, thereby controlling the strength of introducing pseudo-augmentation. In this embodiment, t is set to 0.6.

[0108] Step 3.2.4: Construct the pseudo-augmentation dataset x based on the pseudo-augmentation probability p obtained in Step 3.2.3. real_apa The specific approach is to generate a random probability matrix P of the same size as the dataset (each element is in the range of 0 to 1), and compare it with p (if the element is less than p, the element at that position is set to 0; if it is not less than p, it is set to 1), thus obtaining the pseudo-enhancement introduction matrix M. apa Then calculate it as x real_apa :

[0109] x real_apa =M apa ×x fake +(1-M apa )×x real .

[0110] Step 3.2.5: The training process of the generator network (gen) and the discriminator network (disc) is an adversarial process. Therefore, first, the parameters of the discriminator network (disc) are kept constant, and the generator network (gen) is trained. Since Wasserstein distance is used as the loss function for the generator, the generator's goal is to make x... fake To make it more realistic, the loss function L of the generator is designed. G as follows:

[0111]

[0112] in, This represents the discriminator's expected output value for the generated image. Therefore, the generated image x... fake Substitute these parameters into the loss function described above, and optimize the generator network gen based on the results.

[0113] Step 3.2.6: Similarly, fix the generator network `gen` and train the discriminator network `disc`. First, calculate the Wasserstein distance as the loss function, and introduce a gradient penalty as a constraint to improve the original GAN ​​model. The gradient penalty acts as a linear interpolation point between the real and generated samples during each discriminator update. Perform the discrimination. Specifically, calculate the discriminant function pair. The gradient is calculated, and the L2 norm of the gradient is taken. This norm is then multiplied by a hyperparameter λ, which serves as the gradient penalty term L. gp In this embodiment, ∈ is set to 0.5 and λ is set to 10. The calculation method is as follows:

[0114]

[0115]

[0116] Then the gradient penalty term is added to the loss function L of the discriminator. D The calculation results are as follows:

[0117]

[0118] in, The discriminator's expected output value for the generated image, The expected output value of the discriminator for the real image;

[0119] When updating the parameters of the discriminator `disc`, the gradient norm difference between the real image and the generated image is added as a regularization term to improve the quality of the generated image. The gradient norm difference between the real sample and the generated sample is calculated as follows:

[0120]

[0121] Furthermore, considering that the data from different batches may have significantly different distributions, leading to large differences in gradient norms, a moving average is applied when calculating the gradient norm as loss to reduce variance. In this embodiment, a smoothing factor of α is set (g β ) batch In this embodiment, α is set to 1.0 to calculate the gradient norm of the samples in a batch. The specific calculation is as follows:

[0122]

[0123] The gradient norm difference mentioned above is incorporated into the loss function of the discriminator, and the hyperparameter λ is set accordingly. DIG If the gradient norm difference is a factor, then calculate the loss L for this term. DIG The method is as follows:

[0124] L DIG =λ DIG ·((g real_apa ) batch -(g fake ) batch ) 2

[0125] Among them, (g real ) batch Let g be the gradient norm value calculated by the discriminator for the real image in the current batch. fake ) batch This is the gradient norm value calculated by the discriminator for the generated images in the current batch.

[0126] Finally, based on this, the loss function of the discriminator is calculated as follows: In this embodiment, λ DIG Set to 1.0:

[0127] L ′ D =L D +L DIG

[0128] Therefore, the generated image x fake Image x with pseudo-data augmentation real_apa Substituting the parameters into the updated loss function L ′ D And based on this result, optimize the network parameters of the discriminator disc;

[0129] Step 3.2.7: Steps 3.2.1 to 3.2.6 involve iteratively updating the network parameters for each batch. If the current batch is not the last batch of the current iteration epoch, increment the current batch size by 1 and repeat steps 3.2.1 to 3.2.6 to train the GAN model. Otherwise, it is necessary to determine whether the conditions for terminating training are met (the current iteration epoch is greater than the preset iteration epoch n). epoch_GAN If the condition is met, training is terminated; otherwise, the current training epoch is incremented by 1, and the current batch is initialized to 0. Steps 3.2.1 to 3.2.6 are then repeated to train the GAN model.

[0130] Step 3.3: Following step 3.2, the GAN model training stops, and the generator network M of the trained GAN model is saved. G With discriminator network M D .

[0131] Step 3.4: Measure the generator network M of the GAN model using the image quality evaluation metric FID. G The difference in distribution between the generated sample and the real sample is calculated using the following formula:

[0132] FID = ||μ x -μ g || 2 +Tr(∑ x +∑ g -2(∑ x ∑ g ) 1 / 2 )

[0133] Where, μ x With μ g These represent the mean vectors of the real dataset and the generated dataset, respectively. Tr represents the trace operation of the matrix, and Σ... x With ∑ g These represent the covariance matrices of the real dataset and the generated dataset, respectively. In this embodiment, Figure 4This shows a comparison between the third motion profile image generated using GAN and the first motion profile image. Figure 5 To investigate the changes in FID using different improvement methods for GAN, the FID index was finally obtained as 90.43 based on the method of generating samples using the GAN model as described in step 3, which is a significant improvement compared to the FID index of 151.33 when the model was not improved.

[0134] Step 3.5: Apply the trained model M G The process is as follows: the data processed in step 2 to obtain the second motion contour map is then input into the generator network M of the GAN model, referring to step 3.2.2. G The third motion contour map data generated by the GAN model is obtained.

[0135] Step 4: Based on the deep convolutional neural network architecture, construct a high-risk driving scenario identification model; use the first motion contour map, the second motion contour map, and the third motion contour map, which are divided into training sets, as the training set of the high-risk driving scenario identification model, and input them into the high-risk driving scenario identification model for training to obtain the high-risk driving scenario identification model; use the test set and validation set of the first motion contour map as the test set and validation set of the high-risk driving scenario identification model.

[0136] Specifically, the detailed process of step 4 is as follows:

[0137] Step 4.1: Construct the ResNet network structure for the high-risk driving scenario identification model. The detailed process is as follows:

[0138] Step 4.1.1: First, construct the initial convolutional layer, which includes a convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer. During this process, it is necessary to set the number of input channels, kernel size, stride, and padding size of the convolutional layer. In addition, the kernel size, stride, and padding size of the max pooling layer are also set. In this implementation, the number of input channels of the convolutional layer is 3, the kernel size is 7*7, the stride is 2, and the padding is 3. The kernel size of the max pooling layer is 3*3, the stride is 2, and there is no padding during the pooling process.

[0139] Step 4.1.2: Construct residual blocks. Each residual block consists of two parts: the main path contains two convolutional layers, each followed by a batch normalization layer and an activation function layer; the shortcut is to provide a bypass that directly connects the input to the output of the residual block. This can be an identity mapping or include convolutional operations (when the input and output dimensions are inconsistent).

[0140] Step 4.1.3: Based on the residual blocks constructed in Step 4.1.2, a multi-layer residual network is established. In this process, it is necessary to determine the number of residual blocks and the parameters of each residual block. In this embodiment, 8 residual blocks are constructed. Specifically, for example, the first group consists of 2 residual blocks with 64 input channels. The kernel size of the first convolutional layer is 3*3, the stride is 1, and the padding during convolution is 1. The kernel size of the second convolutional layer is 3*3, the stride is 1, and the padding during convolution is 1. The shortcut adopts horizontal equal transformation.

[0141] Step 4.1.4: Construct a global average pooling layer and a fully connected layer for output. In this process, the global average pooling layer compresses each channel of the feature map into a single average value. The number of output nodes of the fully connected layer is equal to the number of categories of the classification task. Based on this, the Softmax function is applied so that the sum of different classification probability values ​​is 1. In this embodiment, the number of output nodes of the fully connected layer is equal to 2.

[0142] Step 4.2: Use the first motion contour map from Step 1, the enhanced second motion contour map from Step 2, and the third motion contour map generated in Step 3 as the training set, and input them into the high-risk driving scene recognition model constructed in Step 4.1 for model training. After n... epoch The iteration cycle continues until the model converges on the training set. Furthermore, due to the small amount of training data, to prevent overfitting on the training set, an early stopping mechanism is implemented based on the model's loss change on the validation set. Specifically, this involves determining whether the model's loss on the validation set Θ... valid If the loss increases continuously, and the number of consecutive increases exceeds the preset threshold `early_stopping` of the early stopping mechanism, then training should be stopped immediately, and the high-risk driving scenario identification model M should be saved. DA In this embodiment, n is set. epoch The value is 1000, and early_stopping is 15.

[0143] Step 4.3: In the test set Θ test Above, the set includes e n A driving scenario marked as non-conflict and e c A driving scenario marked as high-risk. In this embodiment, e c There are 98, e n There are 176 scenarios in the set. For each scenario in the set, the already trained M is used. DA The model performs classification prediction. Record M. DA The number of predictions the model makes for non-conflict driving scenarios and high-risk driving scenarios are respectively and Using this data, the following confusion matrix was constructed to demonstrate the model's predictive performance:

[0144] Table 1.M DA Confusion matrix of the model on the test set

[0145]

[0146] The sensitivity I of the high-risk driving scenario recognition model was calculated. sensitivity Specificity I specificity The formula is as follows:

[0147] I sensitivity =TP / (TP+FN)

[0148] I specificity =TN / (FP+TN)

[0149] And according to I sensitivity and I specificity ROC curves are generated to evaluate the predictive performance of the high-risk driving scenario recognition model. In this embodiment, I sensitivity The calculated result is 0.74, I specificity The calculated result is 0.82; Figure 7 As shown, the AUC value of the high-risk driving scenario identification model changes before and after augmentation using GAN-generated samples during training. The curves indicate that the model's performance improves after GAN-generated sample augmentation. In this embodiment, the model's AUC reaches 0.852, a 4% improvement compared to the unenhanced model's AUC of 0.819. This is also superior to the AUC of 0.842 of the AlexNet model in the published paper "Predicting Hazardous Driving Events Using Multimodal Deep Learning Based on Video MotionProfile and Kinematics Data" and the AUC of 0.814 of the VK-Net model in "A Lightweight VK-Net Based on Motion Profiles for Hazardous Driving Scenario Identification".

[0150] Step 5: Input the first motion contour map to be identified into the high-risk driving scenario identification model. After calculation by the high-risk driving scenario identification model, a continuous risk value is output. The driving risk is evaluated based on the relationship between the continuous risk value and the optimal classification threshold determined by the Youden index method. If the continuous risk value is greater than the optimal classification threshold, the first motion contour map is a high-risk driving scenario; otherwise, it is a non-conflict driving scenario.

[0151] Specifically, the detailed process of step 5 is as follows:

[0152] Step 5.1: Generate a motion contour map from the vehicle-mounted forward video data collected during a natural driving process using the algorithm in Step 1;

[0153] Step 5.2: Input the motion contour map data generated in Step 5.1 into the M trained in Step 4. DA In the model, the predicted value is calculated, and a classification threshold is determined based on the Youden index. If the continuous risk value is greater than the optimal classification threshold, the prediction result is a high-risk driving scenario, and an alert is issued. In this embodiment, the Youden index is calculated to be 0.6, meaning that when the model's predicted value is greater than 0.6, the prediction result is risky, and when the model's predicted value is less than 0.6, the prediction result is risk-free.

[0154] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for identifying high-risk driving scenarios based on GAN data augmentation, characterized in that, Includes the following steps: Step S1: Obtain vehicle forward video data, generate a first motion contour map based on the vehicle forward video data, and divide the first motion contour map into a training set part, a validation set part and a test set part. Step S2: Enhance the first motion contour map of the training set portion; The second motion profile diagram is obtained; Step S3: Obtain the GAN model by inputting the second motion contour map into the GAN model and outputting the third motion contour map; obtaining the GAN model includes constructing the GAN model, training the GAN model, validating the GAN model, and applying the GAN model. The specific implementation of constructing the GAN model is as follows: Step S301: Construct a deconvolution module, which is used to generate an image; Step S302: Stack multiple deconvolution modules to form a generator network; Step S303: Construct a convolutional module, which is used to distinguish between real and fake images; Step S304: Stack multiple convolutional modules to form a discriminator network; Training the GAN model includes training the generator network and the discriminator network; During the training of the generator network and the discriminator network, the iteration period and batch size of the training process are set; wherein, one iteration period specifically includes the following steps: Step S311: Sample the second motion contour map with the batch size to obtain the real image of the current batch; Step S312: Based on the real image, obtain the corresponding generated image through the generator network; Step S313: Introduce overfitting factor The pseudo-enhancement probability is obtained. ; Step S314: Based on the real image, the generated image, and the pseudo-enhancement probability Construct a pseudo-augmented dataset, as shown in the following formula: in, This is a pseudo-augmented dataset. Introduce a matrix for pseudo-enhancement. For real images, To generate an image; Step S315: Fix the parameters of the discriminator network, train the generator network, design the loss function of the generator, substitute the generated image into the loss function, and optimize the parameters of the generator network. The loss function of the generator is shown in the following formula: in, Let be the loss function of the generator. The expected output value of the discriminator for the generated image; Step S316: Fix the parameters of the generator network, train the discriminator network, design the loss function of the discriminator, input the generated image and the pseudo-data augmented image into the loss function, and optimize the parameters of the discriminator network; Step S317: When the iteration period is greater than the preset iteration period, save the generator network and the discriminator network to complete the training of the GAN model. Step S4: Based on the deep convolutional neural network architecture, construct a high-risk driving scene identification model; input the training set portion of the first motion contour map, the second motion contour map and the third motion contour map into the high-risk driving scene identification model for training to obtain the high-risk driving scene identification model; Step S5: Input the first motion contour map into the high-risk driving scenario identification model. After calculation by the high-risk driving scenario identification model, a continuous risk value is output. The driving risk is evaluated based on the relationship between the continuous risk value and the optimal classification threshold determined by the Youden index method. If the continuous risk value is greater than the optimal classification threshold, then the first motion profile is a high-risk driving scenario; otherwise, it is a non-conflict driving scenario.

2. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 1, characterized in that, Introducing overfitting factor It also includes determining whether the discriminator network has an overfitting problem, and the overfitting factor. The calculation formula is as follows: 。 3. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 1, characterized in that, The loss function of the discriminator network also includes a gradient penalty term, as shown in the following formula: ； in, The discriminator's expected output value for the generated image, The expected output value of the discriminator for the real image, This is the gradient penalty term.

4. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 1, characterized in that, The verification of the GAN model includes: using the image quality assessment index FID to measure the difference between the third motion contour map and the first motion contour map output by the GAN model.

5. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 1, characterized in that, The application of the GAN model includes the following steps: Step S321: Based on the second motion contour map, randomly generate a noise vector with the same size as the training set of the second motion contour map; Step S322: Input the noise vector into the generator network and output the third motion contour map.

6. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 1, characterized in that, The construction of the high-risk driving scenario identification model includes the following steps: Step S401: Construct an initial convolutional layer, which includes: a convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer. The initial convolutional layer is used for input. Step S402: Construct a residual block, wherein the residual block includes: the main path and the shortcut; Step S403: Establish a multi-layer residual network based on the residual block; Step S404: Construct a global average pooling layer and a fully connected layer, which are used for output.

7. The method for identifying high-risk driving scenarios based on GAN data augmentation according to claim 6, characterized in that, If the loss of the high-risk driving scenario identification model on the validation set increases continuously, and the number of consecutive increases exceeds the preset threshold of the early stop mechanism, then training is stopped and the high-risk driving scenario identification model is saved.

Images