A method for judging traffic congestion during power road construction based on video classification

Abstract

The invention discloses a method for judging traffic congestion during road construction for electric power occupation based on video classification, which relates to the technical field of image processing, and specifically comprises the following steps: generating classification labels; constructing an initial traffic congestion judgment model; screening pseudo-label data; acquiring a tertiary traffic congestion judgment model; generating a traffic congestion judgment model; judging road congestion: judging the road congestion situation according to the traffic congestion judgment model; the invention labels part of the video data captured by a camera, and then uses a semi-supervised training method to automatically label the remaining data, and completes algorithm model training; the invention can be applied to road congestion detection and recognition tasks in electric power construction scenes, has a high accuracy rate, and has the advantages of good stability, strong anti-interference ability, high versatility, etc., has good robustness, and can be applied to intelligent supervision systems on infrastructure sites.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a method for judging traffic congestion during power road construction based on video classification. Background Art

[0002] Road congestion can be defined as a phenomenon that occurs when the number of input vehicles exceeds the output capacity within a unit statistical time. Depending on the number and size of vehicles, congestion will cause varying degrees of loss of road usability. Since power infrastructure tasks often face engineering requirements for road construction, the probability of road congestion is greatly increased. According to the definition of road congestion, the specific task of the congestion prediction algorithm is to perform binary classification of the monitoring scene based on the acquired video data, that is, congestion or no congestion.

[0003] The most common variables for artificially measuring traffic conditions are density, speed, and flow. Traffic density is defined as the number of vehicles per unit length of a road section, traffic speed is the speed recorded by a vehicle at a given time and a specified location, and traffic flow is defined as the number of vehicles passing a reference point per unit time. It is difficult to measure the traffic density of the entire road section at a power construction site using traditional methods, because road construction is an accidental event, the construction section does not have sensor equipment for traffic flow monitoring, and the number of sensors used to estimate the presence of vehicles is limited. Currently, with the increase in the number of traffic cameras and the advancement of computer vision, it is a more feasible way to use cameras to fuse artificial intelligence algorithms to identify traffic congestion.

[0004] The development of deep learning provides a more robust way to identify road congestion. A common method is to use image classification neural networks to perform binary classification of the collected road traffic images to determine whether they are congested. However, due to the lack of depth information in the image and the lack of time dimension information, obtaining road congestion information only from a single frame of image is greatly affected by factors such as the distance of the camera deployment and the speed of the traffic flow. In addition to the inherent limitations of single-frame images in solving the task of distinguishing traffic congestion scenes during road construction, there are two main problems in the generalization application of actual algorithms to achieve the optimization and deployment of road construction congestion detection algorithms. On the one hand, the environment in the power construction scenario is complex, and complex construction facilities are added to the road traffic. The amount of data is less than that in the general traffic scenario, which cannot meet the model training requirements. However, the model trained with the data set in the general traffic scenario cannot directly achieve good generalization effects in the power construction environment. Figure 1 and Figure 2Examples of the environment in a general traffic scenario and a power line construction scenario are shown respectively. The two scenario data have domain differences, and it is necessary to use appropriate transfer learning methods to fully utilize the pre-trained model trained in the general traffic scenario data set. On the other hand, manually marking traffic images as congested and non-congested labels is a time-consuming task that requires a lot of human and material resources. Therefore, how to solve the above problems is an urgent problem for technicians in this field. Summary of the invention

[0005] In view of this, the present invention provides a method for judging traffic congestion during power road construction based on video classification, which overcomes the above-mentioned defects.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for judging traffic congestion during power road construction based on video classification, the specific steps are as follows:

[0008] Generation of classification labels: classify the source domain data in the source domain dataset and generate labels at the same time;

[0009] Construction of the initial traffic congestion judgment model: For the classified source domain data, the initial traffic congestion judgment model is constructed using the ResNet3D model backbone network and the BERT classification model;

[0010] Screening of pseudo-label data: Use the classified source domain data to train the primary traffic congestion judgment model to obtain the secondary traffic congestion judgment model; and use the secondary traffic congestion judgment model to predict the labels of the target domain data in the target domain data set, and generate pseudo labels for a preset number of target domain data according to the confidence data;

[0011] Acquisition of the tertiary traffic congestion judgment model: Based on the labels of the source domain data and the pseudo labels of the target domain data, the secondary traffic congestion judgment model is trained using the training method of the domain adversarial transfer network DANN to obtain the tertiary traffic congestion judgment model;

[0012] Generation of traffic congestion judgment model: predicting the target domain data without predicted pseudo labels in the target domain data set according to the three-time traffic congestion judgment model, generating pseudo labels for a preset number of target domain data according to the confidence data, and repeating the three-time traffic congestion judgment model related steps, optimizing the three-time traffic congestion judgment model until the number of target domain data with pseudo labels generated in the target domain data set is greater than or equal to the preset threshold, and obtaining the traffic congestion judgment model;

[0013] Road congestion judgment: Judge the road congestion situation based on the traffic congestion judgment model.

[0014] Optionally, before the initial traffic congestion judgment model is constructed, the source domain data is also preprocessed, specifically: 64 frames are sampled for each source domain data, and each frame of data is scaled to a size of 224*224 pixels.

[0015] Optionally, the initial traffic congestion judgment model is specifically as follows: the ResNet3D50 model backbone network extracts features from the preprocessed source domain data, inputs the extracted feature data into the BERT classification model for prediction, and outputs the prediction results.

[0016] An optional training method based on the domain adversarial transfer network DANN is as follows: source domain data and target domain data with different distributions are mapped into a specified feature space, and by finding a suitable measurement criterion, the "distance" between the data of the two domain distributions in the feature space is made infinitely close, and then, the classifier trained with the source domain data is used to complete the video classification of the target domain data.

[0017] Optionally, the domain adversarial transfer network DANN includes a feature extractor, a label predictor and a domain discriminator; wherein the feature extractor is used to map the video data to a specific feature space; the label predictor is used to distinguish whether the video data belongs to a congested state; and the domain discriminator is used to distinguish the domain of the video data.

[0018] Optionally, the loss function for the label predictor is:

[0019]

[0020] In the formula, x i and i is the coordinate of any given data point; G y (G f (·)) is the output value when the feature data is input into the label predictor branch.

[0021] Optionally, the loss function of the domain discriminator is:

[0022]

[0023] Where, d i is the binary label of the i-th sample.

[0024] Optionally, the secondary traffic congestion judgment model is trained using a training method of a domain adversarial transfer network DANN, which also includes optimization of the secondary traffic congestion judgment model. The overall optimization goal is:

[0025]

[0026] Where n is the number of training data samples, γ is the hyperparameter loss weight coefficient; w and V are the corresponding layer weight parameters, and b, z, and c are the corresponding layer bias parameters.

[0027] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a method for judging traffic congestion during power road construction based on video classification, performs partial data annotation on the video data captured by the camera, and then uses a semi-supervised training method to automatically annotate the remaining data, and completes the algorithm model training, and uses the DANN adaptive learning method to weaken the difference in the performance of the two data domains in the feature extraction network, solves the problem of less data in the application scenario and makes full use of the data set in the general traffic scenario to improve the generalization of the network. In addition, the present invention can be applied to the task of road congestion detection and identification in power construction scenarios, has a high accuracy rate, and has the advantages of good stability, strong anti-interference ability, high versatility, etc., has good robustness, and can be applied to the intelligent supervision system of infrastructure site. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0029] Figure 1 Sample video images for general traffic scene training;

[0030] Figure 2 Sample video images for training scenes of power line construction;

[0031] Figure 3 This is a schematic diagram of the network architecture of the present invention;

[0032] Figure 4 Schematic diagram of semi-supervised training strategy;

[0033] Figure 5 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The embodiment of the present invention discloses a method for judging traffic congestion during power road construction based on video classification, such as Figure 5 As shown, the specific steps are:

[0036] Step 1: Classify the source domain data in the source domain dataset and generate labels;

[0037] Specifically: Use public general-scenario road traffic video surveillance datasets and divide the data into congested and non-congested categories.

[0038] The label file contains the video data file name, data ID, frame number, frame pixel information, and whether it is congested (0 for non-congested and 1 for congested).

[0039] Step 2: For the classified source domain data, the ResNet3D model backbone network and the BERT classification model are used to build the initial traffic congestion judgment model;

[0040] Specifically, each public general scene road traffic video data is preprocessed, including sampling 64 frames of each video data and scaling each frame data to 224*224 pixels. The ResNet3D model backbone network is used to extract features from the preprocessed video data, and the extracted feature data is input into the BERT classification model to generate video classification predictions and establish a deep learning network model;

[0041] The specific deep learning network model is as follows: the image is sent to the network after unified preprocessing. The backbone network of the network uses ResNet3D50, which extracts feature video data and predicts the features using the BERT structure, fully perceives the feature information of the time latitude and spatial dimension of the video data, and completes the prediction of whether the video is a traffic congestion scene.

[0042] Step 3: Use the classified source domain data to train the preliminary traffic congestion judgment model to obtain the secondary traffic congestion judgment model; and use the secondary traffic congestion judgment model to predict the labels of the target domain data in the target domain data set, and generate pseudo labels for a preset number of target domain data according to the confidence data;

[0043] Specifically, all road traffic monitoring sample video data obtained in the power road construction scene are preprocessed, including sampling 64 frames of each video data, and scaling each frame of data to 224*224 pixels, and all data are used as training set data;

[0044] The deep learning network model is trained using public general-scenario road traffic video data, and the label prediction is performed on the monitoring video training set data in the power road construction scenario. The top 20% of the training set confidence ranking from high to low is used to generate pseudo labels.

[0045] Step 4: Based on the labels of the source domain data and the pseudo labels of the target domain data, the secondary traffic congestion judgment model is trained using the training method of the domain adversarial transfer network DANN to obtain a tertiary traffic congestion judgment model;

[0046] Specifically, the deep learning network model is trained using the labels of public general-scenario road traffic video data and the pseudo-label training set of road monitoring video training sets in power road construction scenarios. Based on the training method of the domain adversarial transfer network DANN, the domain adaptive transfer training of the network model is realized;

[0047] The domain adversarial transfer training of the DANN involved specifically refers to mapping the source domain data (labeled data in general traffic scenarios) and target domain data (road construction video data in power scenarios) with different distributions into a feature space that is as consistent as possible during the training process, and by finding a suitable metric, making the "distance" of the data distributed in the two domains as close as possible in this feature space. Subsequently, the classifier trained in the source domain can be migrated to the target domain data to complete the video classification task.

[0048] like Figure 3 As shown in Figure 1, the network structure of DANN mainly consists of three parts: feature extractor, label predictor and domain discriminator. The feature extractor is used to map data to a specific feature space, the label predictor determines whether the data is in a congested state, and the domain discriminator is used to distinguish which domain the data comes from.

[0049] The network loss used to solve the domain adaptation problem in this embodiment consists of two parts: training loss (label predictor loss) and domain discriminant loss. The label discriminator classifies the data from the source domain, and the domain discriminator classifies the data in the feature space into domain categories, trying to distinguish which domain the data comes from. The feature extractor and the domain discriminator are connected by a gradient reversal layer. During the training process, for the labeled data from the source domain, the network continuously minimizes the loss of the label predictor, and for all data from the source domain and the target domain, the network continuously minimizes the loss of the domain discriminator. The ideal feature extractor is expected to make it impossible for the domain discriminator to distinguish which domain the data comes from.

[0050] Among them, the label discrimination loss is specifically:

[0051] For the feature extraction network, sigmoid is used as the activation function, and its output is:

[0052] G f (x;w,b)=sigm(w*x+b) (1);

[0053] Among them, x is the feature data input to this layer of the network, w is the weight parameter of this layer, and b is the bias parameter of this layer.

[0054] When the feature data is input into the label predictor branch, softmax is used as the activation function, and its output is:

[0055] G y (G f (x); V,c) = softmax(V*G f (x)+c) (2);

[0056] Among them, G f (x) is the feature data input to the network layer, V is the weight parameter of this layer, and c is the bias parameter of this layer.

[0057] For a given data point (x i ,y i ), using the negative likelihood log as the loss function, the loss function of the label predictor is:

[0058]

[0059] Domain discrimination loss, specifically:

[0060] For the domain discriminator, sigmoid is also used as the activation function, and its output is:

[0061] G d (G f (x);u,z)=sigm(u T *G f (x)+z) (4);

[0062] Among them, G f (x) is the characteristic data input to the network layer, u T is the weight parameter of this layer, and z is the bias parameter of this layer.

[0063] Similar to the label discrimination loss function, the loss of the domain discriminator is a binary cross entropy loss:

[0064]

[0065] d i Represents the binary label of the i-th sample, which is used to indicate whether the sample belongs to the source domain or the target domain.

[0066] According to the definition of label discrimination loss and domain discriminator loss, the total loss can be used to perform reverse error propagation to optimize the network model during network training. The overall optimization goal is as follows:

[0067]

[0068] Wherein, n is the number of training data samples, γ is the hyperparameter loss weight coefficient, and in the experiment conducted in this embodiment, the weight coefficient is set to 1.0. Due to the existence of the gradient reversal layer, the process of DANN minimizing the total objective function is to update the parameters of the label predictor and the domain discriminator.

[0069] Step 5: predict the target domain data without predicted pseudo labels in the target domain data set according to the three-time traffic congestion judgment model, generate pseudo labels for a preset number of target domain data according to the confidence data, and repeat the relevant steps of obtaining the three-time traffic congestion judgment model, optimize the three-time traffic congestion judgment model, until the number of target domain data with pseudo labels generated in the target domain data set is greater than or equal to the preset threshold, and obtain the traffic congestion judgment model;

[0070] Specifically:

[0071] The model after preliminary training is used to predict the surveillance video training set data without predicted pseudo labels in the power road construction scenario. The top 20% of the data ranked from high to low in confidence are used to generate pseudo labels. Repeat step 4 to optimize the model until the number of pseudo labels of road surveillance video data in the power road construction scenario is not less than 80% of the total.

[0072] The training set target domain data labeling pseudo labeling method can be summarized as follows: Figure 4 In the steps shown, the model is first trained on labeled data from the source domain and used to generate predictions for the unlabeled set of the target domain. All prediction results are filtered by setting a confidence threshold and pseudo labels and data with higher credibility are selected to be included in the data for the next round of model training. This process is repeated n times. Finally, the model trained for the nth time is used for prediction and testing.

[0073] Step 6: Road congestion judgment: judge the road congestion situation according to the traffic congestion judgment model;

[0074] Specifically: save the optimized model and apply the model to road congestion judgment in actual power road construction scenarios.

[0075] Example 2

[0076] The data set used in this embodiment is divided into source domain data and target domain data. Specifically, the source domain data adopts 2610 video data of the BrnoCompSpeed ​​data set in public traffic scenarios, of which 1080 video data are congested label data and 1530 video data are non-congested label data. The target domain data set was produced by actual collection at the Guangzhou power construction site, and a total of 1547 video data from 11 construction sections were collected. Among them, 1047 target domain video data were randomly selected without labeling for semi-supervised model training, and 500 video data were manually calibrated for model performance testing.

[0077] During the training process, the feature extraction network uses ResNet3D-50, the pre-trained model uses the model trained on the kinetics400 dataset to reduce the difficulty of model training, and the discriminator is initialized by random initialization. During the semi-supervised training process, 64 frames of images are selected from each video data as input data by random sampling, the batch-size is set to 8, and each frame of image data is uniformly scaled to 112×112. The model training optimizer uses stochastic gradient descent (SGD) as the optimizer, the maximum number of iterations is 50, the first 5 iterations are slow start training, the initial learning rate is 0.01, the momentum term is 0.9, the weight decay coefficient is 1×10e-4, and the learning rate is reduced to 0.1 times the current learning rate when training 5000 and 10000 batch data respectively. The first 10 epochs use the source domain data for semi-supervised training, and then use the pseudo-label labeling strategy to label the unlabeled target domain data with pseudo labels. The initial confidence threshold is set to 0.9 and the confidence threshold is increased by 0.01 every two epochs. The maximum confidence threshold is set to 0.99. The experimental hardware conditions for model semi-supervised learning and prediction are NVIDIA RTX3090 GPU, the system environment is Ubuntu16.04, and the training framework is Pytorch.

[0078] This embodiment uses the average accuracy as an evaluation indicator when performing a personnel positioning algorithm on the target domain test data. The calculation method is as follows:

[0079]

[0080] Among them, k represents the total number of video classifications. For congestion and non-congestion recognition, the number of classifications is 2. m represents the number of videos correctly identified by the algorithm, and n represents the total number of this class in the test dataset.

[0081] Table 1 shows the results of the congestion judgment ablation experiment and accuracy test on the traffic test dataset of the power road construction scenario of the method proposed in this embodiment. ResNet3D-50 Supervised means that the source domain data is used for labeled full supervision training, and the model with the best performance on the source domain data set is tested on the target domain test set. It can be seen that due to the difference in data distribution between the source domain data set and the target domain data set, the model trained with the source domain data set only achieved an average accuracy of 77.3% on the target domain data set. ResNet3D-50Supervised+DANN means that the source domain data is used for labeled full supervision training, and the target domain training data set is input to use the DANN method for domain discriminator loss back propagation. Compared with the ResNet3D-50 Supervised method, the use of DANN weakens the specific performance of the feature extraction network for the difference in data distribution between the two domains, and an experimental result of 88.3% is achieved on the target domain data test set, which is an 11.0% performance improvement compared to ResNet3D-50 Supervised. ResNet3D-50 Semi-supervised means that model learning is performed by semi-supervised training using source domain data and target domain data. Since the target domain data is used in the semi-supervised training process, this process has the effect of domain adaptation, so the experimental test results achieved an accuracy of 87.2%, which is a 9.9% performance improvement compared to ResNet3D-50 Supervised. ResNet3D-50 Semi-supervised+DANN is the complete method proposed in this embodiment. This method makes full use of the data in the source domain and the target domain, improves the performance robustness of the model, and can reduce the impact of data differences on model performance through domain adaptation methods. The experiment finally achieved the optimal experimental result of 93.2%. Table 2 is a detailed list of the experimental results of ResNet3D-50 Semi-supervised+DANN in this experiment.

[0082] Table 1 Comparison of the accuracy of traffic congestion experiments in power line construction scenarios

[0083]

[0084] Table 2 Details of the traffic congestion test results in the power line construction scenario

[0085]

[0086] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0087] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for judging traffic congestion during power road construction based on video classification, characterized in that: The specific steps are: Generation of classification labels: classify the source domain data in the source domain dataset and generate labels at the same time; Construction of the initial traffic congestion judgment model: For the classified source domain data, the initial traffic congestion judgment model is constructed using the ResNet3D model backbone network and the BERT classification model; Screening of pseudo-label data: Use the classified source domain data to train the primary traffic congestion judgment model to obtain the secondary traffic congestion judgment model; and use the secondary traffic congestion judgment model to predict the labels of the target domain data in the target domain data set, and generate pseudo labels for a preset number of target domain data according to the confidence data; Acquisition of the tertiary traffic congestion judgment model: Based on the labels of the source domain data and the pseudo labels of the target domain data, the secondary traffic congestion judgment model is trained using the training method of the domain adversarial transfer network DANN to obtain the tertiary traffic congestion judgment model; Generation of traffic congestion judgment model: predict the target domain data without predicted pseudo labels in the target domain data set according to the three-time traffic congestion judgment model, generate pseudo labels for a preset number of target domain data according to the confidence data, and repeat the relevant steps of obtaining the three-time traffic congestion judgment model, optimize the three-time traffic congestion judgment model, until the number of target domain data with pseudo labels generated in the target domain data set is greater than or equal to the preset threshold, and obtain the traffic congestion judgment model; Road congestion judgment: judge the road congestion situation based on the traffic congestion judgment model; The initial traffic congestion judgment model is as follows: the ResNet3D-50 model backbone network extracts features from the preprocessed source domain data, inputs the extracted feature data into the BERT classification model for prediction, and outputs the prediction results; The training method based on the domain adversarial transfer network DANN is as follows: the source domain data and the target domain data with different distributions are mapped into the specified feature space, and the distance between the data of the two domain distributions in the feature space is made infinitely close by finding a suitable metric criterion. Then, the classifier trained by the source domain data is used to complete the video classification of the target domain data. The domain adversarial transfer network DANN includes a feature extractor, a label predictor and a domain discriminator; wherein the feature extractor is used to map the video data to a specific feature space; the label predictor is used to distinguish whether the video data belongs to a congested state; and the domain discriminator is used to distinguish the domain of the video data; The loss function of the label predictor is: ； In the formula, and is the coordinate of any given data point; The output value when the feature data is input into the label predictor branch; The loss function of the domain discriminator is: ； In the formula, d i For the i Binary labels for samples.

2. According to the method of judging traffic congestion during power road construction based on video classification in claim 1, it is characterized in that: Before the initial traffic congestion judgment model was constructed, the source domain data was also preprocessed. Specifically, 64 frames of each source domain data were sampled and each frame of data was scaled to 224*224 pixels.

Images