Traffic network congestion identification method in complex scenarios based on drones and data fusion

Abstract

The present invention discloses a complex scene traffic network congestion identification method based on unmanned aerial vehicles and data fusion, and belongs to the technical field of traffic network congestion identification. It solves the problem that the traditional traffic congestion identification method in the prior art has low recognition efficiency and is difficult to apply to complex traffic scenes; the present invention collects visible light images and infrared images in complex traffic scenes, extracts and matches image features, and obtains fused images; constructs a data set, inputs the data set into the established basic model, auxiliary model and enhanced model for training, integrates the loss functions of the three models, and uses the overall loss function to train the three models to obtain the optimal model; by adjusting the enhancement coefficient, the optimal model is optimized, and the optimal model with the highest accuracy is obtained, that is, the final optimal model, which outputs the congested area positioning. The present invention effectively improves the recognition performance of the traffic network congestion identification model, and can be applied to complex scene traffic network congestion identification.

Description

Technical Field

[0001] The present invention relates to a method for identifying traffic network congestion in complex scenarios, and in particular to a method for identifying traffic network congestion in complex scenarios based on unmanned aerial vehicles and data fusion, and belongs to the technical field of traffic network congestion identification. Background Art

[0002] Traffic congestion can be divided into traffic network congestion, periodic congestion, occasional congestion, and bottleneck congestion. Traffic network congestion is usually caused by the superposition of multiple factors, such as multiple accidents occurring at the same time, multiple large-scale events, or the inefficiency of the traffic management system of the entire city, which leads to congestion at multiple nodes or sections in the traffic network at the same time, which may affect the traffic flow in the entire area; periodic congestion often occurs at fixed times and places, mainly including congestion during rush hours, weekends and holidays; occasional congestion is congestion caused by emergencies, such as traffic accidents, emergencies, special events, abnormal weather conditions, etc.; bottleneck congestion occurs at places where traffic flow is limited by the physical conditions of the road, such as narrowed roads, bridges, tunnels or toll stations, and also includes traffic capacity that cannot meet the demand during peak hours due to unreasonable or aging road design. Traffic speed reduction and congestion; Among the four types of traffic congestion, traffic network congestion often has the most serious impact on traffic flow. This is because traffic network congestion not only has a wide impact, covering the entire traffic network, including multiple important nodes and sections, but also its complexity and unpredictability make management and resolution more difficult. In addition, network congestion often lasts for a long time and requires coordinated and comprehensive strategies to resolve, so its negative impact on economic and social life is more significant; the identification of traffic network congestion is crucial to urban management and planning, which can significantly improve traffic efficiency, reduce environmental pollution, and promote economic development. Through effective congestion management, it can not only ensure the rapid response of emergency services, but also improve the quality of daily life of residents and reduce psychological pressure. In addition, reducing congestion will also help promote the use of public transportation and support the city's sustainable development strategy.

[0003] The use of drones in traffic network congestion identification has multiple advantages. First, drones can quickly and efficiently collect a wide range of traffic flow data from the air, which is highly flexible and cost-effective. Second, the high-resolution cameras carried by drones can provide a detailed perspective to help accurately identify congestion points. In addition, the real-time data support of drones can be directly used to optimize traffic management decisions, such as dynamically adjusting traffic signals. Drones also reduce the risk of deploying personnel in dangerous areas and enhance the safety of collection operations.

[0004] On the other hand, in traffic network congestion identification, complex scenes often pose a challenge to traditional visible light-based image acquisition technology. Complex scenes include low-light environments, severe weather conditions (such as fog, haze, etc.), and highly reflective surfaces (wet roads or surfaces exposed to direct sunlight). The above factors will lead to problems such as low contrast, highlight / reflection problems, and color distortion, thereby affecting image clarity and recognition accuracy, making it difficult for traditional cameras to work effectively in complex environments. Infrared cameras provide an effective solution to this. Compared with traditional visible light cameras, the main advantage of infrared cameras is that they can still obtain clear images under extremely low light conditions, penetrate haze and precipitation, reduce the impact of scattering and absorption, and are insensitive to strong light reflections, thereby providing more stable image quality in various lighting environments. The fusion of traditional visible light images and infrared images can improve the accuracy of traffic congestion identification in complex scenes.

[0005] Prior art 1, a patent document with publication (announcement) number CN114627642B discloses a traffic congestion identification method and device, which determines the traffic congestion in the time period by comparing the driving speed in the time period to be identified with the speed subsequence of the traffic period to which it belongs; Prior art 2, a patent document with publication (announcement) number CN108847024A discloses a video-based traffic congestion identification method and system, which analyzes and processes the video data collected by the front-end device, and can confirm the road congestion by the number of identified vehicles; However, the above-mentioned method and system have low recognition efficiency and are difficult to apply to complex scenarios. Since the multi-source data fusion method has not been considered to improve the generalization ability of the algorithm for complex scenarios, there are few applicable scenarios and many restrictions.

[0006] Therefore, a complex scenario traffic network congestion identification method based on drones and data fusion is needed. Summary of the invention

[0007] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0008] In view of this, in order to solve the problem that the traditional traffic congestion identification method in the prior art has low recognition efficiency and is difficult to apply to complex traffic scenes, the present invention provides a complex scene traffic network congestion identification method based on drones and data fusion.

[0009] The technical solution is as follows: A complex scene traffic network congestion identification method based on drones and data fusion includes the following steps:

[0010] S1. Collect visible light images and infrared images in complex traffic scenes, adjust the size and color of the two images to be consistent, extract and match image features, and obtain a fused image after image fusion;

[0011] S2. construct a data set based on the fused image, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models to obtain the fused loss function, define the overall loss function, use the overall loss function to train the three models, and obtain the optimal model;

[0012] S3. The optimal model is further optimized by adjusting the enhancement coefficient. The optimal model with the highest accuracy is the final optimal model, which outputs the congested area location.

[0013] Furthermore, the S1 specifically includes the following steps:

[0014] S11. Mount a visible light camera and an infrared camera on a drone to collect visible light images and infrared images under complex traffic scenes, respectively, adjust the sizes and colors of the two images to be consistent, and obtain a visible light image with updated pixel values ​​and an infrared image with updated pixel values;

[0015] S12. Performing Gaussian blur processing on the visible light image after pixel value update and the infrared image after pixel value update, and calculating the local extreme value points in the visible light image after Gaussian blur processing at each scale according to the obtained visible light image after Gaussian blur processing;

[0016] S13. Establish a quantitative characterization matrix for each local extreme point, obtain a visible light image feature length matrix and an infrared image feature length matrix, calculate the minimum distance between the two, and determine the best extreme point matching result between the visible light image and the infrared image;

[0017] S14. Based on the determination of the best extreme point matching result between the visible light image and the infrared image, linearly transform the infrared image after the pixel value is updated to obtain a transformed infrared image, and perform pixel fusion on the visible light image after the pixel value is updated and the transformed infrared image to obtain a fused image;

[0018] In S11, a two-dimensional coordinate system is established to compare the visible light images. and infrared images The size of is the length of the visible light image, is the width of the visible light image, is the length of the infrared image, and is the width of the infrared image;

[0019] when There is no need to adjust the two image sizes when When the infrared image size needs to be adjusted, the size of the visible light image is used as the target size of the infrared image to obtain the infrared image after the image size is adjusted. ;

[0020] For visible light images of the same size and infrared image after image resizing Perform color space conversion to convert the color image into a grayscale image to obtain a grayscale visible light image , after updating its pixel values, the visible light image with updated pixel values ​​is obtained Infrared image after pixel value update ;

[0021] In S12, Gaussian blur processing It is expressed as:

[0022] ;

[0023] in, is the pixel position of the image in the two-dimensional coordinate system, is the pixel horizontal coordinate, is the pixel ordinate, is the standard deviation, used to control the degree of blur;

[0024] According to Gaussian blur , and obtain the visible light image after Gaussian blur processing at each scale And the infrared image after Gaussian blur ;

[0025] Calculate the visible light image after Gaussian blur processing at each scale The local extreme points in , which correspond to the key points in the image;

[0026] Visible light images after Gaussian blur processing at various scales The calculation process of local extreme points in is expressed as:

[0027] ;

[0028] in, For scale The difference Gaussian value of For scale Visible light image after Gaussian blur processing;

[0029] For scale The difference Gaussian value of For each pixel in, traverse the 26 neighboring points around it , i=1, 2, ..., 26, the 26 neighborhood points include 8 neighbors in the same scale layer, i.e., the neighborhood points in the up, down, left, right, and four diagonal directions, 9 neighborhood points in the corresponding position of the previous scale layer and its surroundings, and 9 neighborhood points in the corresponding position of the next scale layer and its surroundings. If there are more than 26 neighboring points, is a local maximum when Less than 26 neighboring points, then is a local minimum when If the number of neighboring points is not greater than or less than 26, then It is not a local extreme point. The local extreme point is obtained by integrating the local maximum and local minimum.

[0030] In S13, the local maximum pixel point is recorded as , calculate the characteristic length between the local maximum pixel and the surrounding 26 neighborhood points ;

[0031] The characteristic length between the local maximum pixel and the surrounding 26 neighborhood points It is expressed as:

[0032] ;

[0033] in, is the horizontal coordinate of the neighborhood point, is the ordinate of the neighborhood point, is the horizontal coordinate of the local maximum pixel point, is the ordinate of the local maximum pixel;

[0034] Compare the characteristic length formed between the local maximum pixel and the surrounding 26 neighborhood points The numerical values ​​are sorted from large to small, and the top 4 feature lengths are selected to establish the feature length matrix corresponding to the local maximum pixel point. ;

[0035] The local minimum pixel is recorded as , calculate the characteristic length between the local minimum pixel and the surrounding 26 neighborhood points ;

[0036] The characteristic length between the local minimum pixel and the surrounding 26 neighboring points It is expressed as:

[0037] ;

[0038] Compare the characteristic length formed between the local minimum pixel and the surrounding 26 neighborhood points The numerical values ​​are sorted from large to small, and the top 4 feature lengths are selected to establish the feature length matrix corresponding to the local minimum pixel point. ;

[0039] The feature length matrix corresponding to the local maximum pixel The feature length matrix corresponding to the local minimum pixel They are all called characteristic length matrices and are recorded as , that is, the characteristic length matrix corresponding to the visible light image is obtained The characteristic length moment corresponding to the infrared image ;

[0040] Feature length matrix corresponding to visible light image It is expressed as:

[0041] ;

[0042] in, is the total number of local extreme points, For the The characteristic length corresponding to the local extreme point;

[0043] The characteristic length matrix corresponding to the infrared image It is expressed as:

[0044] ;

[0045] By randomly transforming the extreme point matching of the two images, the distance between the feature length matrices of all visible light images and infrared images is calculated. That is, the distance between the characteristic visible light image and infrared image characteristic length matrix corresponding to the im-th local extreme point matching result, where im=1, 2, 3…, , is the total number of local extreme point matching results, and the minimum value of the distance between the feature length matrix of the visible light image and the infrared image is screened out , the corresponding extreme point matching situation is the best extreme point matching result;

[0046] In S14, the visible light image after the pixel value is updated is kept unchanged;

[0047] The transformed infrared image is Expressed as

[0048] ;

[0049] in, is the transformation matrix;

[0050] By adjusting the transformation matrix , so that the transformed infrared image Infrared image after updating pixel values Match at the best local extreme point;

[0051] The fused image It is expressed as:

[0052] ;

[0053] in, is the visible light image weight coefficient.

[0054] Furthermore, the step S2 specifically includes the following steps:

[0055] S21. Based on the fused images, select all images including traffic network congestion, periodic congestion, occasional congestion and bottleneck congestion as input images;

[0056] S22. Through manual annotation, traffic network congestion and three other congestion types are selected in the input image, a data set is constructed, and the data set is divided into a training set and a test set in a ratio of 8:2;

[0057] S23. According to the data set, construct a basic model, an auxiliary model and an enhanced model, calculate the loss functions of the three models and fuse them to obtain the fused loss function, and define the overall loss function;

[0058] S24. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model;

[0059] In S22, the traffic network congestion is marked as Network, and the three types of congestion, namely periodic congestion, occasional congestion and bottleneck congestion, are marked as other1, other2 and other3 respectively;

[0060] In S23, a network structure of a basic model, i.e., a deep learning model A, is constructed. The deep learning model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 splicing layers, 4 convolutional upsampling layers, and 1 output layer;

[0061] Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ;

[0062] The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values ​​of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values ​​of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ;

[0063] The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model A , the output value of the first congestion type are the congestion output values ​​of the first traffic network respectively , the output value of the first periodic congestion , the first occasional congestion output value and the first bottleneck congestion output value ;

[0064] The first auxiliary model, deep learning model B, is constructed. Deep learning model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer.

[0065] Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ;

[0066] The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values ​​of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node Output value of the eighth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ;

[0067] The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model B , the output value of the second congestion type are the congestion output values ​​of the second traffic network respectively , the output value of the second periodic congestion , the second occasional congestion output value and the second bottleneck congestion output value ;

[0068] The second auxiliary model, namely the deep learning model C, is constructed. The deep learning model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers and 1 output layer.

[0069] Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ;

[0070] The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values ​​of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ;

[0071] The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model B , output value of the third congestion type are the congestion output values ​​of the third traffic network respectively , the output value of the third periodic congestion , the third occasional congestion output value and the third bottleneck congestion output value ;

[0072] Construct a reinforcement model, namely the deep learning model D, which consists of 1 input layer, 26 convolutional layers, 26 activation layers, 7 pooling layers, 6 concatenation layers, 6 convolutional upsampling layers and 1 output layer;

[0073] Input the data set into the deep learning model D, calculate the weights and biases of the neural network units of the deep learning model D through the error back propagation algorithm, complete the training of the deep learning model D, and obtain the trained deep learning model D and its loss function ;

[0074] The output value of the output unit node corresponding to the deep learning model D is , respectively, are the output values ​​of the thirteenth output unit node , the output value of the fourteenth output unit node , the output value of the fifteenth output unit node and the output value of the sixteenth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model D are initially converted into the fourth preliminary output value and the probability value of the fourth preliminary output value , and then further optimize and transform it to obtain the fourth optimized output value and the enhanced probability value of the fourth optimized output value ;

[0075] The enhanced probability value using the fourth optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model D. After the training is completed, the output value of the fourth congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model D , output value of the fourth congestion type are the congestion output values ​​of the fourth traffic network , the output value of the fourth periodic congestion , Fourth occasional congestion output value and the fourth bottleneck congestion output value ;

[0076] According to the second congestion type, the output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient;

[0077] Probability value based on the first preliminary output value The actual congestion type annotation corresponding to the node of the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient;

[0078] Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ;

[0079] According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. , the relationship loss function between the output layer of deep learning model C and deep learning model A , the relationship loss function between the output layer of deep learning model D and deep learning model A The relationship loss function between the output layers of deep learning model D and deep learning model C , define the corresponding overall loss function between deep learning model A, deep learning model B, deep learning model C and deep learning model D ;

[0080] Overall loss function It is expressed as:

[0081] .

[0082] Furthermore, in S3, based on the trained deep learning model A, i.e., the optimal model, the performance of the deep learning model A is further optimized by adjusting the enhancement coefficient G, and the enhancement coefficient G is set to 1, 3, 5, 7, 9, 11, 15, 20, 30, 40 or 50, for a total of 11 working conditions. 11 deep learning models A are trained, and their corresponding accuracy is expressed as , i=1, 2, 3, ..., 11, calculate the accuracy of deep learning model A The maximum value in , and its corresponding deep learning model A is the final optimal model. ,in, To obtain the maximum value operation.

[0083] Furthermore, in S23, the layers in the network structure of the deep learning model A are connected in the following order:

[0084] Input layer: image size 227×227×3;

[0085] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0086] Activation layer 1: LeakyReLU;

[0087] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0088] Activation layer 2: LeakyReLU;

[0089] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0090] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0091] Activation layer 3: LeakyReLU;

[0092] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0093] Activation layer 4: LeakyReLU;

[0094] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0095] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0096] Activation layer 5: LeakyReLU;

[0097] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0098] Activation layer 6: LeakyReLU;

[0099] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0100] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0101] Activation layer 7: LeakyReLU;

[0102] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0103] Activation layer 8: LeakyReLU;

[0104] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0105] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0106] Activation layer 9: LeakyReLU;

[0107] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0108] Activation layer 10: LeakyReLU;

[0109] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0110] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0111] Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0112] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0113] Activation layer 11: LeakyReLU;

[0114] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0115] Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0116] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0117] Activation layer 12: LeakyReLU;

[0118] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0119] Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0120] Convolution layer 13: kernel size 3×3, number of kernels 64, stride: 1;

[0121] Activation layer 13: LeakyReLU;

[0122] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0123] Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0124] Convolution layer 14: kernel size 3×3, number of kernels 32, stride: 1;

[0125] Activation layer 14: LeakyReLU;

[0126] Output layer: Type: Softmax, Number of neurons: 4.

[0127] Furthermore, in S23, the layers in the network structure of the deep learning model B are connected in the following order:

[0128] Input layer: image size 227×227×3;

[0129] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0130] Activation layer 1: LeakyReLU;

[0131] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0132] Activation layer 2: LeakyReLU;

[0133] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0134] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0135] Activation layer 3: LeakyReLU;

[0136] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0137] Activation layer 4: LeakyReLU;

[0138] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0139] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0140] Activation layer 5: LeakyReLU;

[0141] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0142] Activation layer 6: LeakyReLU;

[0143] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0144] Convolution layer 7: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0145] Activation layer 7: LeakyReLU;

[0146] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0147] Activation layer 8: LeakyReLU;

[0148] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0149] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0150] Activation layer 9: LeakyReLU;

[0151] Convolution layer 10: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0152] Activation layer 10: LeakyReLU;

[0153] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0154] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0155] Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 1024;

[0156] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0157] Activation layer 11: LeakyReLU;

[0158] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0159] Activation layer 12: LeakyReLU;

[0160] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0161] Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 512;

[0162] Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0163] Activation layer 13: LeakyReLU;

[0164] Convolution layer 14: kernel size 3×3, number of kernels 256, stride: 1;

[0165] Activation layer 14: LeakyReLU;

[0166] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0167] Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 256;

[0168] Convolution layer 15: kernel size 3×3, number of kernels 128, stride: 1;

[0169] Activation layer 15: LeakyReLU;

[0170] Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0171] Activation layer 16: LeakyReLU;

[0172] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0173] Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 128;

[0174] Convolution layer 17: kernel size 3×3, number of kernels 64, stride: 1;

[0175] Activation layer 17: LeakyReLU;

[0176] Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0177] Activation layer 18: LeakyReLU;

[0178] Output layer: Type: Softmax, Number of neurons: 4.

[0179] Furthermore, in S23, the layers in the network structure of the deep learning model C are connected in the following order:

[0180] Input layer: image size 227×227×3;

[0181] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0182] Activation layer 1: LeakyReLU;

[0183] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0184] Activation layer 2: LeakyReLU;

[0185] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0186] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0187] Activation layer 3: LeakyReLU;

[0188] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0189] Activation layer 4: LeakyReLU;

[0190] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0191] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0192] Activation layer 5: LeakyReLU;

[0193] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0194] Activation layer 6: LeakyReLU;

[0195] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0196] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0197] Activation layer 7: LeakyReLU;

[0198] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0199] Activation layer 8: LeakyReLU;

[0200] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0201] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0202] Activation layer 9: LeakyReLU;

[0203] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0204] Activation layer 10: LeakyReLU;

[0205] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0206] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0207] Activation layer 11: LeakyReLU;

[0208] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0209] Activation layer 12: LeakyReLU;

[0210] Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2;

[0211] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0212] Concatenation layer 1: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024;

[0213] Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0214] Activation layer 13: LeakyReLU;

[0215] Convolution layer 14: kernel size 3×3, number of kernels 512, stride: 1;

[0216] Activation layer 14: LeakyReLU;

[0217] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0218] Concatenation layer 2: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0219] Convolution layer 15: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0220] Activation layer 15: LeakyReLU;

[0221] Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0222] Activation layer 16: LeakyReLU;

[0223] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0224] Concatenation layer 3: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0225] Convolution layer 17: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0226] Activation layer 17: LeakyReLU;

[0227] Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0228] Activation layer 18: LeakyReLU;

[0229] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0230] Concatenation layer 4: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0231] Convolution layer 19: kernel size 3×3, number of kernels 64, stride: 1;

[0232] Activation layer 19: LeakyReLU;

[0233] Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0234] Activation layer 20: LeakyReLU;

[0235] Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0236] Concatenation layer 5: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0237] Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0238] Activation layer 21: LeakyReLU;

[0239] Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0240] Activation layer 22: LeakyReLU;

[0241] Output layer: Type: Softmax, Number of neurons: 4.

[0242] Furthermore, in S23, the layers in the network structure of the deep learning model D are connected in the following order:

[0243] Input layer: image size 227×227×3;

[0244] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0245] Activation layer 1: LeakyReLU;

[0246] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0247] Activation layer 2: LeakyReLU;

[0248] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0249] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0250] Activation layer 3: LeakyReLU;

[0251] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0252] Activation layer 4: LeakyReLU;

[0253] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0254] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0255] Activation layer 5: LeakyReLU;

[0256] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0257] Activation layer 6: LeakyReLU;

[0258] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0259] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0260] Activation layer 7: LeakyReLU;

[0261] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0262] Activation layer 8: LeakyReLU;

[0263] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0264] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0265] Activation layer 9: LeakyReLU;

[0266] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0267] Activation layer 10: LeakyReLU;

[0268] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0269] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0270] Activation layer 11: LeakyReLU;

[0271] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0272] Activation layer 12: LeakyReLU;

[0273] Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2;

[0274] Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1;

[0275] Activation layer 13: LeakyReLU;

[0276] Convolution layer 14: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1;

[0277] Activation layer 14: LeakyReLU;

[0278] Pooling layer 7: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0279] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 1024;

[0280] Concatenation layer 1: connects the feature map with the output of activation layer 12, and the number of channels increases to 2048;

[0281] Convolution layer 15: kernel size 3×3, number of kernels 1024, stride: 1;

[0282] Activation layer 15: LeakyReLU;

[0283] Convolution layer 16: kernel size 3×3, number of kernels 1024, stride: 1;

[0284] Activation layer 16: LeakyReLU;

[0285] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0286] Concatenation layer 2: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024;

[0287] Convolutional layer 17: kernel size 3×3, number of kernels 512, stride: 1;

[0288] Activation layer 17: LeakyReLU;

[0289] Convolution layer 18: kernel size 3×3, number of kernels 512, stride: 1;

[0290] Activation layer 18: LeakyReLU;

[0291] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0292] Concatenation layer 3: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0293] Convolution layer 19: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0294] Activation layer 19: LeakyReLU;

[0295] Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0296] Activation layer 20: LeakyReLU;

[0297] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0298] Concatenation layer 4: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0299] Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0300] Activation layer 21: LeakyReLU;

[0301] Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0302] Activation layer 22: LeakyReLU;

[0303] Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0304] Concatenation layer 5: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0305] Convolutional layer 23: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0306] Activation layer 23: LeakyReLU;

[0307] Convolutional layer 24: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0308] Activation layer 24: LeakyReLU;

[0309] Convolution upsampling 6: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0310] Concatenation layer 6: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0311] Convolutional layer 25: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0312] Activation layer 25: LeakyReLU;

[0313] Convolutional layer 26: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0314] Activation layer 26: LeakyReLU;

[0315] Output layer: Type: Softmax, Number of neurons: 4.

[0316] The beneficial effects of the present invention are as follows: the present invention effectively improves the quality of images collected by drones in complex scenes by performing multi-source image fusion on visible light images and infrared images collected by visible light cameras and infrared cameras onboard drones in traffic scenes; the present invention realizes intelligent identification of traffic network congestion in complex scenes, and reduces the complexity of the model and improves the recognition speed of the model while ensuring the accuracy of identifying traffic network congestion; the present invention further improves the model performance by improving the model by adjusting the enhancement coefficient of the optimal model, so that the output congested area positioning is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0317] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0318] Figure 1 The figure is a flowchart of the complex scenario traffic network congestion identification method based on drones and data fusion. DETAILED DESCRIPTION

[0319] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0320] refer to Figure 1 The present embodiment is described in detail. The complex scene traffic network congestion identification method based on drone and data fusion specifically includes the following steps:

[0321] S1. Collect visible light images and infrared images in complex traffic scenes, adjust the size and color of the two images to be consistent, extract and match image features, and obtain a fused image after image fusion;

[0322] S2. construct a data set based on the fused image, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models to obtain the fused loss function, define the overall loss function, use the overall loss function to train the three models, and obtain the optimal model;

[0323] S3. The optimal model is further optimized by adjusting the enhancement coefficient. The optimal model with the highest accuracy is the final optimal model, which outputs the congested area location.

[0324] Furthermore, the S1 specifically includes the following steps:

[0325] S11. Mount a visible light camera and an infrared camera on a drone to collect visible light images and infrared images under complex traffic scenes, respectively, adjust the sizes and colors of the two images to be consistent, and obtain a visible light image with updated pixel values ​​and an infrared image with updated pixel values;

[0326] S12. Performing Gaussian blur processing on the visible light image after pixel value update and the infrared image after pixel value update, and calculating the local extreme value points in the visible light image after Gaussian blur processing at each scale according to the obtained visible light image after Gaussian blur processing;

[0327] S13. Establish a quantitative characterization matrix for each local extreme point, obtain a visible light image feature length matrix and an infrared image feature length matrix, calculate the minimum distance between the two, and determine the best extreme point matching result between the visible light image and the infrared image;

[0328] S14. Based on the determination of the best extreme point matching result between the visible light image and the infrared image, linearly transform the infrared image after the pixel value is updated to obtain a transformed infrared image, and perform pixel fusion on the visible light image after the pixel value is updated and the transformed infrared image to obtain a fused image;

[0329] In the above S11, in order to ensure the image fusion effect of the visible light image and the infrared camera, it is necessary to ensure that the two images are compatible in size and color. Therefore, a two-dimensional coordinate system is established to compare the visible light image. and infrared images The size of is the length of the visible light image, is the width of the visible light image, is the length of the infrared image, and is the width of the infrared image;

[0330] when There is no need to adjust the two image sizes when When the infrared image size needs to be adjusted, the size of the visible light image is used as the target size of the infrared image to obtain the infrared image after the image size is adjusted. ;

[0331] Infrared image after image resizing It is expressed as:

[0332] ;

[0333] in, is the difference kernel, is the difference number of the infrared image along the length direction, is the difference number of the infrared image along the length and width directions;

[0334] For visible light images of the same size and infrared image after image resizing Perform color space conversion to convert the color image into a grayscale image to obtain a grayscale visible light image , after updating its pixel values, the visible light image with updated pixel values ​​is obtained Infrared image after pixel value update ;

[0335] The color space conversion process is expressed as:

[0336] ;

[0337] ;

[0338] in, is the pixel value of the red channel in the image, is the pixel value of the green channel in the image, is the pixel value of the blue channel in the image;

[0339] In S12, Gaussian blur processing It is expressed as:

[0340] ;

[0341] in, is the pixel position of the image in the two-dimensional coordinate system, is the pixel horizontal coordinate, Pixel ordinate, is the standard deviation, used to control the degree of blur;

[0342] According to Gaussian blur , and obtain the visible light image after Gaussian blur processing at each scale And the infrared image after Gaussian blur ;

[0343] Visible light image after Gaussian blur It is expressed as:

[0344] ;

[0345] Infrared image after Gaussian blur It is expressed as:

[0346] ;

[0347] Calculate the visible light image after Gaussian blur processing at each scale The local extreme points in , which correspond to the key points in the image;

[0348] Visible light images after Gaussian blur processing at various scales The calculation process of local extreme points in is expressed as:

[0349] ;

[0350] in, For scale The difference Gaussian value of For scale Visible light image after Gaussian blur processing;

[0351] For scale The difference Gaussian value of For each pixel in, traverse the 26 neighboring points around it , i=1, 2, ..., 26, the 26 neighborhood points include 8 neighbors in the same scale layer, i.e., the neighborhood points in the up, down, left, right, and four diagonal directions, 9 neighborhood points in the corresponding position of the previous scale layer and its surroundings, and 9 neighborhood points in the corresponding position of the next scale layer and its surroundings. If there are more than 26 neighboring points, is a local maximum when Less than 26 neighboring points, then is a local minimum when If the number of neighboring points is not greater than or less than 26, then It is not a local extreme point. The local extreme point is obtained by integrating the local maximum and local minimum.

[0352] Specifically, Gaussian blurring of images can simulate the multi-scale characteristics of images and help detect stable key points at multiple scales;

[0353] In S13, the quantization representation matrix is ​​composed of feature lengths. For the convenience of description, the local maximum pixel point is recorded as , calculate the characteristic length between the local maximum pixel and the surrounding 26 neighborhood points ;

[0354] The characteristic length between the local maximum pixel and the surrounding 26 neighborhood points It is expressed as:

[0355] ;

[0356] in, is the horizontal coordinate of the neighborhood point, is the ordinate of the neighborhood point, is the horizontal coordinate of the local maximum pixel point, is the ordinate of the local maximum pixel;

[0357] Compare the characteristic length formed between the local maximum pixel and the surrounding 26 neighborhood points The numerical values ​​are sorted from large to small, and the top 4 feature lengths are selected to establish the feature length matrix corresponding to the local maximum pixel point. , ;

[0358] The local minimum pixel is recorded as , calculate the characteristic length between the local minimum pixel and the surrounding 26 neighborhood points ;

[0359] The characteristic length between the local minimum pixel and the surrounding 26 neighboring points It is expressed as:

[0360] ;

[0361] Compare the characteristic length formed between the local minimum pixel and the surrounding 26 neighborhood points The numerical values ​​are sorted from large to small, and the top 4 feature lengths are selected to establish the feature length matrix corresponding to the local minimum pixel point. ;

[0362] The feature length matrix corresponding to the local maximum pixel The feature length matrix corresponding to the local minimum pixel They are all called characteristic length matrices and are recorded as , that is, the characteristic length matrix corresponding to the visible light image is obtained The characteristic length moment corresponding to the infrared image ;

[0363] Feature length matrix corresponding to visible light image It is expressed as:

[0364] ;

[0365] in, is the total number of local extreme points, For the The characteristic length corresponding to the local extreme point;

[0366] The characteristic length matrix corresponding to the infrared image It is expressed as:

[0367] ;

[0368] By randomly transforming the extreme point matching of the two images, the distance between the feature length matrices of all visible light images and infrared images is calculated. , that is, the distance between the characteristic visible light image and infrared image characteristic length matrix corresponding to the im-th local extreme point matching result, where im=1, 2, 3…, , is the total number of local extreme point matching results, and the minimum value of the distance between the feature length matrix of the visible light image and the infrared image is screened out , the corresponding extreme point matching situation is the best extreme point matching result;

[0369] The distance between the feature length matrices of visible light image and infrared image It is expressed as:

[0370] ;

[0371] Minimum value of the distance between the feature length matrices of the visible light image and the infrared image It is expressed as:

[0372] ;

[0373] In S14, the visible light image after the pixel value is updated is kept unchanged;

[0374] The transformed infrared image is Expressed as

[0375] ;

[0376] in, is the transformation matrix;

[0377] By adjusting the transformation matrix , so that the transformed infrared image Infrared image after updating pixel values Matching at the best local extreme point, so far, the matching of the visible light image and the infrared image is completed, that is, for the same coordinate position in the two images, the corresponding actual position is the same;

[0378] The fused image It is expressed as:

[0379] ;

[0380] in, is the visible light image weight coefficient.

[0381] Furthermore, the step S2 specifically includes the following steps:

[0382] S21. Based on the fused images, select all images including traffic network congestion, periodic congestion, occasional congestion and bottleneck congestion as input images;

[0383] S22. Through manual annotation, traffic network congestion and three other congestion types are selected in the input image, a data set is constructed, and the data set is divided into a training set and a test set in a ratio of 8:2;

[0384] S23. According to the data set, construct the basic model, auxiliary model and enhanced model, calculate the loss functions of the three models and fuse them to obtain the fused loss function, and define the overall loss function;

[0385] S24. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model;

[0386] In S22, the traffic network congestion is marked as Network, and the three types of congestion, namely periodic congestion, occasional congestion and bottleneck congestion, are marked as other1, other2 and other3 respectively;

[0387] In S23, a network structure of a basic model, i.e., a deep learning model A, is constructed. The deep learning model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 splicing layers, 4 convolutional upsampling layers, and 1 output layer;

[0388] Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ;

[0389] Loss function of the trained deep learning model A It is expressed as:

[0390] ;

[0391] in, is the total number of data samples in the dataset of deep learning model A, For the The true labels of samples, For deep learning model A, predict The probability that a sample is of category 1, is the natural logarithm;

[0392] The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values ​​of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values ​​of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ;

[0393] Probability value of the first preliminary output value It is expressed as:

[0394] ;

[0395] The enhanced probability value of the first optimized output value It is expressed as:

[0396] ;

[0397] in, G is the enhancement coefficient;

[0398] The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model A , the output value of the first congestion type are the congestion output values ​​of the first traffic network respectively , the output value of the first periodic congestion , the first occasional congestion output value and the first bottleneck congestion output value ;

[0399] The first auxiliary model, deep learning model B, is constructed. Deep learning model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer.

[0400] Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ;

[0401] Loss function of the trained deep learning model B It is expressed as:

[0402] ;

[0403] in, is the total number of data samples in the dataset of deep learning model B, For the The true labels of samples, For deep learning model B to predict The probability that a sample is class 1;

[0404] The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values ​​of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ;

[0405] Probability value of the second preliminary output value It is expressed as:

[0406] ;

[0407] The enhanced probability value of the second optimized output value It is expressed as:

[0408] ;

[0409] The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model B , the output value of the second congestion type are the congestion output values ​​of the second traffic network respectively , the output value of the second periodic congestion , the second occasional congestion output value and the second bottleneck congestion output value ;

[0410] The second auxiliary model, namely the deep learning model C, is constructed. The deep learning model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers and 1 output layer.

[0411] Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ;

[0412] Loss function of the trained deep learning model C It is expressed as:

[0413] ;

[0414] in, is the total number of data samples in the dataset of the deep learning model C, For the The true labels of samples, For the deep learning model C to predict The probability that a sample is class 1;

[0415] The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values ​​of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ;

[0416] The probability value of the third preliminary output value It is expressed as:

[0417] ;

[0418] The enhanced probability value of the third optimized output value It is expressed as:

[0419] ;

[0420] The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model B , output value of the third congestion type are the congestion output values ​​of the third traffic network respectively , the output value of the third periodic congestion , the third occasional congestion output value and the third bottleneck congestion output value ;

[0421] Construct a reinforcement model, namely the deep learning model D, which consists of 1 input layer, 26 convolutional layers, 26 activation layers, 7 pooling layers, 6 concatenation layers, 6 convolutional upsampling layers and 1 output layer;

[0422] Input the data set into the deep learning model D, calculate the weights and biases of the neural network units of the deep learning model D through the error back propagation algorithm, complete the training of the deep learning model D, and obtain the trained deep learning model D and its loss function ;

[0423] Loss function of the trained deep learning model D It is expressed as:

[0424] ;

[0425] in, is the total number of data samples in the dataset of the deep learning model D, For the The true labels of samples, For the deep learning model D to predict The probability that a sample is class 1;

[0426] The output value of the output unit node corresponding to the deep learning model D is , respectively, are the output values ​​of the thirteenth output unit node , the output value of the fourteenth output unit node , the output value of the fifteenth output unit node and the output value of the sixteenth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model D are initially converted into the fourth preliminary output value and the probability value of the fourth preliminary output value , and then further optimize and transform it to obtain the fourth optimized output value and the enhanced probability value of the fourth optimized output value ;

[0427] Probability value of the fourth preliminary output value It is expressed as:

[0428] ;

[0429] Enhanced probability value of the fourth optimized output value It is expressed as:

[0430] ;

[0431] The enhanced probability value using the fourth optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model D. After the training is completed, the output value of the fourth congestion type is obtained The actual congestion type annotation corresponding to the node of the output layer of the deep learning model D , output value of the fourth congestion type are the congestion output values ​​of the fourth traffic network , the output value of the fourth periodic congestion , Fourth occasional congestion output value and the fourth bottleneck congestion output value ;

[0432] According to the second congestion type, the output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient;

[0433] Loss function between deep learning model B and deep learning model A It is expressed as:

[0434] ;

[0435] Probability value based on the first preliminary output value The actual congestion type annotation corresponding to the node of the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient;

[0436] Consider the probability value of the first preliminary output value The loss function of the deep learning model A is It is expressed as:

[0437] ;

[0438] Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ;

[0439] The relationship loss function between the output layer of deep learning model B and deep learning model A It is expressed as:

[0440] ;

[0441] in, is the loss function between deep learning model B and deep learning model A The weight coefficient of is the probability value considering the first preliminary output value The loss function of the deep learning model A is The weight coefficient of

[0442] According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. , the relationship loss function between the output layer of deep learning model C and deep learning model A , the relationship loss function between the output layer of deep learning model D and deep learning model A The relationship loss function between the output layers of deep learning model D and deep learning model C , define the corresponding overall loss function between deep learning model A, deep learning model B, deep learning model C and deep learning model D ;

[0443] Overall loss function It is expressed as:

[0444] ;

[0445] Loss function between deep learning model C and deep learning model B It is expressed as:

[0446] ;

[0447] Consider the probability value of the second preliminary output value The loss function of the deep learning model B is It is expressed as:

[0448] ;

[0449] The relationship loss function between the output layers of deep learning model C and deep learning model B It is expressed as:

[0450] ;

[0451] in, is the loss function between deep learning model C and deep learning model B The weight coefficient of is the probability value considering the second preliminary output value The loss function of the deep learning model B is The weight coefficient of

[0452] Loss function between deep learning model C and deep learning model A It is expressed as:

[0453] ;

[0454] The relationship loss function between the output layer of deep learning model C and deep learning model A It is expressed as:

[0455] ,

[0456] in, is the loss function between deep learning model C and deep learning model A The weight coefficient of

[0457] Loss function between deep learning model D and deep learning model A It is expressed as:

[0458] ;

[0459] The relationship loss function between the output layer of deep learning model D and deep learning model A It is expressed as:

[0460] ;

[0461] in, is the loss function between deep learning model D and deep learning model A The weight coefficient of

[0462] Loss function between deep learning model D and deep learning model C It is expressed as:

[0463] ;

[0464] Consider the probability value of the third preliminary output value The loss function of the deep learning model C It is expressed as:

[0465] ;

[0466] The relationship loss function between the output layers of deep learning model D and deep learning model C It is expressed as:

[0467] ;

[0468] in, is the loss function between deep learning model D and deep learning model C The weight coefficient of

[0469] Specifically, traffic network congestion is not limited to specific bottlenecks or accident points, but is widely distributed throughout the entire traffic network. Through monitoring images, we can observe the widespread distribution of congestion and the continuous stagnation of vehicles. Traffic network congestion is usually caused by multiple factors, including a large number of vehicles, traffic accidents, poor road conditions, etc. The image may show multiple factors causing congestion, such as the coexistence of traffic accidents and high traffic volume;

[0470] In view of the congestion characteristics of traffic networks, the model structure is designed to include input layer, convolution layer, activation layer, pooling layer, splicing layer, convolution upsampling layer and output layer. The functions of each structural layer are as follows: The convolution layer can recognize various patterns in the image, such as vehicles, road markings, congested areas, etc., and capture features ranging from basic edges and textures to complex objects such as vehicle configurations and accident scenes; the activation layer introduces nonlinear processing, allowing the model to learn more complex image patterns, which helps to accelerate network training while maintaining the sparsity of the output and improving the generalization ability of the model; the pooling layer can help the model reduce sensitivity to small position changes while maintaining key information; the splicing layer can be used to combine features from different convolution paths, which helps to capture local details and global context at the same time; the convolution upsampling layer can increase the spatial resolution of the feature map, which is crucial in recovering detail information from deep networks and is especially useful for accurately locating congested areas; the output layer provides the final output, such as the classification or positioning of congested areas.

[0471] Furthermore, in S3, based on the trained deep learning model A, the performance of the deep learning model A is further optimized by adjusting the enhancement coefficient G, and the enhancement coefficient G is 1, 3, 5, 7, 9, 11, 15, 20, 30, 40 or 50, for a total of 11 working conditions. 11 deep learning models A are trained, and their corresponding accuracy is expressed as , i=1, 2, 3, ..., 11, calculate the accuracy of deep learning model A The maximum value in , and its corresponding deep learning model A is the final optimal model. ,in, To maximize the operation, the model has the highest level of lightweight and can be easily deployed in drone-mounted devices to achieve rapid and accurate identification of traffic network congestion;

[0472] Specifically, intelligent identification of traffic network congestion faces many challenges, including complex scenes, dynamically changing environments, and image occlusions. By improving the optimal model and optimizing the basic model's ability to capture image details, we can better cope with various complex and changing traffic scenarios.

[0473] Furthermore, in S23, the layers in the network structure of the deep learning model A are connected in the following order:

[0474] Input layer: image size 227×227×3;

[0475] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0476] Activation layer 1: LeakyReLU;

[0477] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0478] Activation layer 2: LeakyReLU;

[0479] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0480] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0481] Activation layer 3: LeakyReLU;

[0482] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0483] Activation layer 4: LeakyReLU;

[0484] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0485] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0486] Activation layer 5: LeakyReLU;

[0487] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0488] Activation layer 6: LeakyReLU;

[0489] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0490] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0491] Activation layer 7: LeakyReLU;

[0492] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0493] Activation layer 8: LeakyReLU;

[0494] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0495] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0496] Activation layer 9: LeakyReLU;

[0497] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0498] Activation layer 10: LeakyReLU;

[0499] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0500] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0501] Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0502] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0503] Activation layer 11: LeakyReLU;

[0504] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0505] Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0506] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0507] Activation layer 12: LeakyReLU;

[0508] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0509] Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0510] Convolution layer 13: kernel size 3×3, number of kernels 64, stride: 1;

[0511] Activation layer 13: LeakyReLU;

[0512] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0513] Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0514] Convolution layer 14: kernel size 3×3, number of kernels 32, stride: 1;

[0515] Activation layer 14: LeakyReLU;

[0516] Output layer: Type: Softmax, Number of neurons: 4.

[0517] Furthermore, in S23, the layers in the network structure of the deep learning model B are connected in the following order:

[0518] Input layer: image size 227×227×3;

[0519] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0520] Activation layer 1: LeakyReLU;

[0521] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0522] Activation layer 2: LeakyReLU;

[0523] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0524] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0525] Activation layer 3: LeakyReLU;

[0526] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0527] Activation layer 4: LeakyReLU;

[0528] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0529] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0530] Activation layer 5: LeakyReLU;

[0531] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0532] Activation layer 6: LeakyReLU;

[0533] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0534] Convolution layer 7: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0535] Activation layer 7: LeakyReLU;

[0536] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0537] Activation layer 8: LeakyReLU;

[0538] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0539] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0540] Activation layer 9: LeakyReLU;

[0541] Convolution layer 10: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0542] Activation layer 10: LeakyReLU;

[0543] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0544] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0545] Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 1024;

[0546] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0547] Activation layer 11: LeakyReLU;

[0548] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0549] Activation layer 12: LeakyReLU;

[0550] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0551] Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 512;

[0552] Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0553] Activation layer 13: LeakyReLU;

[0554] Convolution layer 14: kernel size 3×3, number of kernels 256, stride: 1;

[0555] Activation layer 14: LeakyReLU;

[0556] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0557] Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 256;

[0558] Convolution layer 15: kernel size 3×3, number of kernels 128, stride: 1;

[0559] Activation layer 15: LeakyReLU;

[0560] Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0561] Activation layer 16: LeakyReLU;

[0562] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0563] Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 128;

[0564] Convolution layer 17: kernel size 3×3, number of kernels 64, stride: 1;

[0565] Activation layer 17: LeakyReLU;

[0566] Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0567] Activation layer 18: LeakyReLU;

[0568] Output layer: Type: Softmax, Number of neurons: 4.

[0569] Furthermore, in S23, the layers in the network structure of the deep learning model C are connected in the following order:

[0570] Input layer: image size 227×227×3;

[0571] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0572] Activation layer 1: LeakyReLU;

[0573] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0574] Activation layer 2: LeakyReLU;

[0575] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0576] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0577] Activation layer 3: LeakyReLU;

[0578] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0579] Activation layer 4: LeakyReLU;

[0580] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0581] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0582] Activation layer 5: LeakyReLU;

[0583] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0584] Activation layer 6: LeakyReLU;

[0585] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0586] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0587] Activation layer 7: LeakyReLU;

[0588] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0589] Activation layer 8: LeakyReLU;

[0590] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0591] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0592] Activation layer 9: LeakyReLU;

[0593] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0594] Activation layer 10: LeakyReLU;

[0595] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0596] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0597] Activation layer 11: LeakyReLU;

[0598] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0599] Activation layer 12: LeakyReLU;

[0600] Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2;

[0601] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0602] Concatenation layer 1: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024;

[0603] Convolution layer 13: kernel size 3×3, number of kernels 512, stride: 1;

[0604] Activation layer 13: LeakyReLU;

[0605] Convolution layer 14: kernel size 3×3, number of kernels 512, stride: 1;

[0606] Activation layer 14: LeakyReLU;

[0607] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0608] Concatenation layer 2: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0609] Convolution layer 15: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0610] Activation layer 15: LeakyReLU;

[0611] Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0612] Activation layer 16: LeakyReLU;

[0613] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0614] Concatenation layer 3: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0615] Convolution layer 17: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0616] Activation layer 17: LeakyReLU;

[0617] Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0618] Activation layer 18: LeakyReLU;

[0619] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0620] Concatenation layer 4: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0621] Convolution layer 19: kernel size 3×3, number of kernels 64, stride: 1;

[0622] Activation layer 19: LeakyReLU;

[0623] Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0624] Activation layer 20: LeakyReLU;

[0625] Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0626] Concatenation layer 5: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0627] Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0628] Activation layer 21: LeakyReLU;

[0629] Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0630] Activation layer 22: LeakyReLU;

[0631] Output layer: Type: Softmax, Number of neurons: 4.

[0632] Furthermore, in S23, the layers in the network structure of the deep learning model D are connected in the following order:

[0633] Input layer: image size 227×227×3;

[0634] Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0635] Activation layer 1: LeakyReLU;

[0636] Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0637] Activation layer 2: LeakyReLU;

[0638] Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0639] Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0640] Activation layer 3: LeakyReLU;

[0641] Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0642] Activation layer 4: LeakyReLU;

[0643] Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2;

[0644] Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0645] Activation layer 5: LeakyReLU;

[0646] Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0647] Activation layer 6: LeakyReLU;

[0648] Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0649] Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0650] Activation layer 7: LeakyReLU;

[0651] Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0652] Activation layer 8: LeakyReLU;

[0653] Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0654] Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1;

[0655] Activation layer 9: LeakyReLU;

[0656] Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1;

[0657] Activation layer 10: LeakyReLU;

[0658] Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0659] Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0660] Activation layer 11: LeakyReLU;

[0661] Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1;

[0662] Activation layer 12: LeakyReLU;

[0663] Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2;

[0664] Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1;

[0665] Activation layer 13: LeakyReLU;

[0666] Convolution layer 14: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1;

[0667] Activation layer 14: LeakyReLU;

[0668] Pooling layer 7: Type: Max pooling, pooling size: 2×2, Stride: 2;

[0669] Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 1024;

[0670] Concatenation layer 1: connects the feature map with the output of activation layer 12, and the number of channels increases to 2048;

[0671] Convolution layer 15: kernel size 3×3, number of kernels 1024, stride: 1;

[0672] Activation layer 15: LeakyReLU;

[0673] Convolution layer 16: kernel size 3×3, number of kernels 1024, stride: 1;

[0674] Activation layer 16: LeakyReLU;

[0675] Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512;

[0676] Concatenation layer 2: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024;

[0677] Convolutional layer 17: kernel size 3×3, number of kernels 512, stride: 1;

[0678] Activation layer 17: LeakyReLU;

[0679] Convolution layer 18: kernel size 3×3, number of kernels 512, stride: 1;

[0680] Activation layer 18: LeakyReLU;

[0681] Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256;

[0682] Concatenation layer 3: connects the feature map with the output of activation layer 8, and the number of channels increases to 512;

[0683] Convolution layer 19: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0684] Activation layer 19: LeakyReLU;

[0685] Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 256, stride: 1;

[0686] Activation layer 20: LeakyReLU;

[0687] Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128;

[0688] Concatenation layer 4: connects the feature map with the output of activation layer 6, and the number of channels increases to 256;

[0689] Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0690] Activation layer 21: LeakyReLU;

[0691] Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 128, stride: 1;

[0692] Activation layer 22: LeakyReLU;

[0693] Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64;

[0694] Concatenation layer 5: connects the feature map with the output of activation layer 4, and the number of channels increases to 128;

[0695] Convolutional layer 23: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0696] Activation layer 23: LeakyReLU;

[0697] Convolutional layer 24: convolution kernel size 3×3, number of convolution kernels 64, stride: 1;

[0698] Activation layer 24: LeakyReLU;

[0699] Convolution upsampling 6: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32;

[0700] Concatenation layer 6: connects the feature map with the output of activation layer 2, and the number of channels increases to 64;

[0701] Convolutional layer 25: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0702] Activation layer 25: LeakyReLU;

[0703] Convolutional layer 26: convolution kernel size 3×3, number of convolution kernels 32, stride: 1;

[0704] Activation layer 26: LeakyReLU;

[0705] Output layer: Type: Softmax, Number of neurons: 4.

[0706] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A complex scene traffic network congestion identification method based on drones and data fusion, characterized by: The following steps are involved: S1. Collect visible light images and infrared images in complex traffic scenes, adjust the size and color of the two images to be consistent, extract and match image features, and obtain a fused image after image fusion; S2. construct a data set based on the fused image, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models to obtain the fused loss function, define the overall loss function, use the overall loss function to train the three models, and obtain the optimal model; S3. Further optimize the optimal model by adjusting the enhancement coefficient. The optimal model with the highest accuracy is the final optimal model, which outputs the location of the congested area; The S2 specifically includes the following steps: S21. Based on the fused images, select all images including traffic network congestion, periodic congestion, occasional congestion and bottleneck congestion as input images; S22. Through manual annotation, traffic network congestion and three other congestion types are selected in the input image, a data set is constructed, and the data set is divided into a training set and a test set in a ratio of 8:2; S23. According to the data set, construct the basic model, auxiliary model and enhanced model, calculate the loss functions of the three models and fuse them to obtain the fused loss function, and define the overall loss function; S24. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In S22, the traffic network congestion is marked as Network, and the three types of congestion, namely periodic congestion, occasional congestion and bottleneck congestion, are marked as other1, other2 and other3 respectively; In S23, a network structure of a basic model, i.e., a deep learning model A, is constructed. The deep learning model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 splicing layers, 4 convolutional upsampling layers, and 1 output layer; The data set is input into the deep learning model A, and the weights and biases of the neural network units of the deep learning model A are calculated through the error back propagation algorithm to complete the training of the deep learning model A and obtain the trained deep learning model A and its loss function La; The output value of the output unit node corresponding to the deep learning model A is a i , respectively, the output value a1 of the first output unit node, the output value a2 of the second output unit node, the output value a3 of the third output unit node, and the output value a4 of the fourth output unit node. According to the exponential function exp, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability value f of the first preliminary output value. ai , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value g of the first optimized output value ai ; The enhanced probability value g of the first optimized output value is adopted ai And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value j of the first congestion type is obtained ai The actual congestion type label h corresponding to the node of the output layer of the deep learning model A ai , the output value j of the first congestion type ai are the first traffic network congestion output value j a1 , the output value j of the first periodic congestion a2 , the first occasional congestion output value j a3 and the first bottleneck congestion output value j a4 ; The first auxiliary model, deep learning model B, is constructed. Deep learning model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. The data set is input into the deep learning model B, and the weights and biases of the neural network units of the deep learning model B are calculated through the error back propagation algorithm, the training of the deep learning model B is completed, and the trained deep learning model B and its loss function Lb are obtained; The output value of the output unit node corresponding to the deep learning model B is b i , respectively, the output value b1 of the fifth output unit node, the output value b2 of the sixth output unit node, the output value b3 of the seventh output unit node, and the output value b4 of the eighth output unit node. According to the exponential function exp, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value f of the second preliminary output value. bi , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value g of the second optimized output value bi ; The enhanced probability value g of the second optimized output value is adopted bi And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value j of the second congestion type is obtained bi The actual congestion type label h corresponding to the node of the output layer of the deep learning model B bi , the output value j of the second congestion type bi are the congestion output value j of the second traffic network respectively b1 , the output value j of the second periodic congestion b2 , the second occasional congestion output value j b3 and the second bottleneck congestion output value j b4 ; The second auxiliary model, namely the deep learning model C, is constructed. The deep learning model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers and 1 output layer. The data set is input into the deep learning model C, and the weights and biases of the neural network units of the deep learning model C are calculated through the error back propagation algorithm to complete the training of the deep learning model C and obtain the trained deep learning model C and its loss function Lc; The output value of the output unit node corresponding to the deep learning model C is c i , are the output value c1 of the ninth output unit node, the output value c2 of the tenth output unit node, the output value c3 of the eleventh output unit node, and the output value c4 of the twelfth output unit node, respectively. According to the exponential function exp, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value f of the third preliminary output value. ci , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value g of the third optimized output value ci ; The enhanced probability value g of the third optimized output value is adopted ci And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value j of the third congestion type is obtained ci The actual congestion type label h corresponding to the node of the output layer of the deep learning model B bi , the third congestion type output value j ci are the congestion output value j of the third traffic network respectively c1 , the output value j of the third periodic congestion c2 , the third occasional congestion output value j c3 and the third bottleneck congestion output value j c4 ; Construct a reinforcement model, namely the deep learning model D, which consists of 1 input layer, 26 convolutional layers, 26 activation layers, 7 pooling layers, 6 concatenation layers, 6 convolutional upsampling layers and 1 output layer; The data set is input into the deep learning model D, and the weights and biases of the neural network units of the deep learning model D are calculated through the error back propagation algorithm to complete the training of the deep learning model D and obtain the trained deep learning model D and its loss function Ld; The output value of the output unit node corresponding to the deep learning model D is d i , respectively, the output value d1 of the thirteenth output unit node, the output value d2 of the fourteenth output unit node, the output value d3 of the fifteenth output unit node, and the output value d4 of the sixteenth output unit node. According to the exponential function exp, the output values ​​of the four output unit nodes corresponding to the deep learning model D are initially converted into the fourth preliminary output value and the probability value f of the fourth preliminary output value. di , and then further optimize and transform it to obtain the fourth optimized output value and the enhanced probability value g of the fourth optimized output value di ; The enhanced probability value g of the fourth optimized output value is adopted di And the data set, combined with the error chain derivative rule, continue to train the deep learning model D. After the training is completed, the output value j of the fourth congestion type is obtained di The actual congestion type label h corresponding to the node of the output layer of the deep learning model D di , the fourth congestion type output value j di are the fourth traffic network congestion output value j d1 , the output value j of the fourth periodic congestion d2 , the fourth occasional congestion output value j d3 and the fourth bottleneck congestion output value j d4 ; According to the second congestion type, the output value is j bi and the enhanced probability value g of the first optimized output value ai , construct the loss function L between deep learning model B and deep learning model A BA , get the corresponding weight coefficient; The probability value f based on the first preliminary output value ai The actual congestion type label h corresponding to the node of the output layer of the deep learning model A ai , construct the probability value f considering the first preliminary output value ai The loss function L of the deep learning model A is AA , get the corresponding weight coefficient; Based on the loss function L between deep learning model B and deep learning model A BA and considering the probability value f of the first preliminary output value ai The loss function L of the deep learning model A is AA , establish the relationship loss function L between the output layer of deep learning model B and deep learning model A BAA ; According to the above operations, the relationship loss function L between the output layers of the deep learning model C and the deep learning model B is established. CBB , the relationship loss function L between the output layer of the deep learning model C and the deep learning model A CAA , the relationship loss function L between the output layer of the deep learning model D and the deep learning model A DAA The relationship loss function L between the output layers of the deep learning model D and the deep learning model C DCC , define the overall loss function L corresponding to deep learning model A, deep learning model B, deep learning model C and deep learning model D total ; The overall loss function L total It is expressed as: L total =L DCC +L CBB +L BAA +L DAA +L CAA .

2. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 1 is characterized in that: The S1 specifically includes the following steps: S11. Mount a visible light camera and an infrared camera on a drone to collect visible light images and infrared images under complex traffic scenes, respectively, adjust the sizes and colors of the two images to be consistent, and obtain a visible light image with updated pixel values ​​and an infrared image with updated pixel values; S12. Performing Gaussian blur processing on the visible light image after pixel value update and the infrared image after pixel value update, and calculating the local extreme value points in the visible light image after Gaussian blur processing at each scale according to the obtained visible light image after Gaussian blur processing; S13. Establish a quantitative characterization matrix for each local extreme point, obtain a visible light image feature length matrix and an infrared image feature length matrix, calculate the minimum distance between the two, and determine the best extreme point matching result between the visible light image and the infrared image; S14. Based on the determination of the best extreme point matching result between the visible light image and the infrared image, linearly transform the infrared image after the pixel value is updated to obtain a transformed infrared image, and perform pixel fusion on the visible light image after the pixel value is updated and the transformed infrared image to obtain a fused image; In S11, a two-dimensional coordinate system is established to compare the visible light image KJ(x KJ ,y KJ ) and infrared image HW(x HW ,y HW ), where x KJ is the length of the visible light image, y KJ is the width of the visible light image, x HW is the length of the infrared image, and y HW is the width of the infrared image; When x KJ =x HW When x KJ ≠x HW When the infrared image size needs to be adjusted, the size of the visible light image is used as the target size of the infrared image to obtain the infrared image HW after the image size is adjusted. N (x KJ ,y KJ ); For visible light images of the same size KJ(x KJ ,y KJ ) and infrared image HW after image size adjustment N (x KJ ,y KJ ) to perform color space conversion, converting it from a color image to a grayscale image, and obtaining a grayscale visible light image KJ gray , after updating its pixel values, the visible light image KJ after pixel value update is obtained n (x KJ ,y KJ ) and the infrared image HW after pixel value update n (x KJ ,y KJ ); In S12, Gaussian blur processing GS(x gs ,y gs ,σ) is expressed as: Among them, (x gs ,y gs ) is the pixel position of the image in the two-dimensional coordinate system, x gs is the pixel horizontal coordinate, y gs is the pixel ordinate, σ is the standard deviation, which is used to control the degree of blur; According to Gaussian blur processing GS(x gs ,y gs ,σ), and obtain the visible light image KJ after Gaussian blur processing at each scale gs n (x KJ ,y KJ ,σ) and Gaussian blurred infrared image HW gs n (x KJ ,y KJ ,σ); The visible light image KJ after Gaussian blur processing at each scale is calculated gs n (x KJ ,y KJ ,σ), which correspond to the key points in the image; Visible light image KJ after Gaussian blur processing at various scales gs n (x KJ ,y KJ ,σ) in the local extreme point calculation process is expressed as: Among them, DJ gs n (x KJ ,y KJ ,σ) is the differential Gaussian value of scale σ, For scale Visible light image after Gaussian blur processing; For the difference Gaussian value DJ of scale σ gs n (x KJ ,y KJ ,σ), traverses the 26 neighboring points JB around it i (x jb i ,y jb i ), i = 1, 2, ..., 26, the 26 neighborhood points include 8 neighbors in the same scale layer, i.e., the neighborhood points in the upper, lower, left, right, and four diagonal directions, 9 neighborhood points in the corresponding position and its surroundings in the previous scale layer, and 9 neighborhood points in the corresponding position and its surroundings in the next scale layer. When DJ gs n (x KJ ,y KJ ,σ) is greater than 26 neighboring points, then DJ gs n (x KJ ,y KJ ,σ) is a local maximum, when DJ gs n (x KJ ,y KJ ,σ) is less than 26 neighboring points, then DJ gs n (x KJ ,y KJ ,σ) is a local minimum, when DJ gs n (x KJ ,y KJ ,σ) does not satisfy the requirement of being greater than or less than 26 neighboring points, then DJ gs n (x KJ ,y KJ ,σ) is not a local extreme point, and the local extreme point is obtained by integrating the local maximum and local minimum; In S13, the local maximum pixel is recorded as JB max (x jb max ,y jb max ), calculate the characteristic length D between the local maximum pixel and the surrounding 26 neighborhood points JB i ; The characteristic length D formed between the local maximum pixel and the surrounding 26 neighborhood points JB i It is expressed as: Among them, x jb i is the horizontal coordinate of the neighborhood point, y jb i is the ordinate of the neighborhood point, x jbmax is the horizontal coordinate of the local maximum pixel, y jbmax is the ordinate of the local maximum pixel; Compare the characteristic length D formed between the local maximum pixel and the surrounding 26 neighborhood points JB i The numerical values ​​are sorted from large to small, and the top 4 feature lengths are selected to establish the feature length matrix TD corresponding to the local maximum pixel point. JB max , TD JB max =[D JB 1 D JB 2 D JB 3 D JB 4 ]; The local minimum pixel is recorded as JB min (x jb min ,y jb min ), calculate the characteristic length D between the local minimum pixel and the surrounding 26 neighborhood points JB i '; The characteristic length D formed between the local minimum pixel and the surrounding 26 neighborhood points JB i ' is expressed as: Compare the characteristic length D formed between the local minimum pixel and the surrounding 26 neighborhood points JB i ', and sort them in descending order, select the top 4 feature lengths, and establish the feature length matrix TD corresponding to the local minimum pixel point JB min' , TD JB min' =[D JB 1' D JB 2' D JB 3' D JB 4' ]; The feature length matrix TD corresponding to the local maximum pixel JB max The feature length matrix TD corresponding to the local minimum pixel JB min ' are all called characteristic length matrices and are denoted as TD JB , that is, the characteristic length matrix TD corresponding to the visible light image is obtained JB KJ The characteristic length moment TD corresponding to the infrared image JB HW ; The characteristic length matrix TD corresponding to the visible light image JB KJ It is expressed as: TD JB KJ =[TD JB 1 TD JB 2 ... TD JB SM ] Among them, SM is the total number of local extreme points, TD JB SM is the characteristic length corresponding to the SMth local extreme point; The characteristic length matrix TD corresponding to the infrared image JB HW It is expressed as: TD JB HW =[TD JB 1 TD JB 2 ... TD JB SM ] By randomly transforming the extreme point matching of the two images, the distance JL between the feature length matrices of all visible light images and infrared images is calculated. im , that is, the distance between the characteristic visible light image and the infrared image characteristic length matrix corresponding to the im-th local extreme point matching result, where im = 1, 2, 3..., Tom, Tom is the total number of local extreme point matching results, and the minimum value JL of the distance between the characteristic length matrix of the visible light image and the infrared image is screened out min , the corresponding extreme point matching situation is the best extreme point matching result; In S14, the visible light image after the pixel value is updated is kept unchanged; The transformed infrared image is HW bh n Expressed as HW bh n =HW n (x KJ ,y KJ )·BH Among them, BH is the transformation matrix; By adjusting the transformation matrix BH, the transformed infrared image HW bh n Infrared image HW after pixel value update n (x KJ ,y KJ ) matches at the best local extreme point; The fused image RH n It is expressed as: RH n =α·KJ n (x KJ ,y KJ )+(1-a)·HW bh n Among them, α is the visible light image weight coefficient.

3. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 2 is characterized in that: In S3, based on the trained deep learning model A, i.e., the optimal model, the performance of the deep learning model A is further optimized by adjusting the enhancement coefficient G, where the enhancement coefficient G is 1, 3, 5, 7, 9, 11, 15, 20, 30, 40, or 50, for a total of 11 working conditions. 11 deep learning models A are trained, and their corresponding accuracy is expressed as ACa. ni , i = 1, 2, 3, ..., 11, calculate the accuracy ACa of the deep learning model A i The maximum value ACa in max , and its corresponding deep learning model A is the final optimal model, ACa max =max[ACa i ], where max is the maximum value operation.

4. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 3 is characterized in that: In S23, the layers in the network structure of the deep learning model A are connected in the following order: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 1: LeakyReLU; Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 2: LeakyReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 4: LeakyReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 5: LeakyReLU; Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 6: LeakyReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 7: LeakyReLU; Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 8: LeakyReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 10: LeakyReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 512; Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 11: LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 256; Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 12: LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 128; Convolution layer 13: kernel size 3×3, number of kernels 64, stride: 1; Activation layer 13: LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32; Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 64; Convolution layer 14: kernel size 3×3, number of kernels 32, stride: 1; Activation layer 14: LeakyReLU; Output layer: Type: Softmax, Number of neurons:

4.

5. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 4 is characterized in that: In S23, the layers in the network structure of the deep learning model B are connected in the following order: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 1: LeakyReLU; Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 2: LeakyReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 4: LeakyReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 5: LeakyReLU; Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 6: LeakyReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 7: LeakyReLU; Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 8: LeakyReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 9: LeakyReLU; Convolution layer 10: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 10: LeakyReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512; Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 1024; Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 11: LeakyReLU; Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 12: LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 512; Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 13: LeakyReLU; Convolution layer 14: kernel size 3×3, number of kernels 256, stride: 1; Activation layer 14: LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 256; Convolution layer 15: kernel size 3×3, number of kernels 128, stride: 1; Activation layer 15: LeakyReLU; Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 16: LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 128; Convolution layer 17: kernel size 3×3, number of kernels 64, stride: 1; Activation layer 17: LeakyReLU; Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 18: LeakyReLU; Output layer: Type: Softmax, Number of neurons:

4.

6. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 5 is characterized in that: In S23, the layers in the network structure of the deep learning model C are connected in the following order: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 1: LeakyReLU; Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 2: LeakyReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 4: LeakyReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 5: LeakyReLU; Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 6: LeakyReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 7: LeakyReLU; Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 8: LeakyReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 10: LeakyReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 11: LeakyReLU; Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 12: LeakyReLU; Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512; Concatenation layer 1: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024; Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 13: LeakyReLU; Convolution layer 14: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 14: LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 2: connects the feature map with the output of activation layer 8, and the number of channels increases to 512; Convolution layer 15: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 15: LeakyReLU; Convolution layer 16: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 16: LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 3: connects the feature map with the output of activation layer 6, and the number of channels increases to 256; Convolution layer 17: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 17: LeakyReLU; Convolution layer 18: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 18: LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 4: connects the feature map with the output of activation layer 4, and the number of channels increases to 128; Convolution layer 19: kernel size 3×3, number of kernels 64, stride: 1; Activation layer 19: LeakyReLU; Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 20: LeakyReLU; Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32; Concatenation layer 5: connects the feature map with the output of activation layer 2, and the number of channels increases to 64; Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 21: LeakyReLU; Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 22: LeakyReLU; Output layer: Type: Softmax, Number of neurons:

4.

7. The complex scene traffic network congestion identification method based on drone and data fusion according to claim 6 is characterized in that: In S23, the layers in the network structure of the deep learning model D are connected in the following order: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 1: LeakyReLU; Convolutional layer 2: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 2: LeakyReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 4: LeakyReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 5: LeakyReLU; Convolution layer 6: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 6: LeakyReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 7: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 7: LeakyReLU; Convolution layer 8: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 8: LeakyReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 9: convolution kernel size 3×3, number of convolution kernels 512, stride: 1; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 10: LeakyReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 11: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 11: LeakyReLU; Convolution layer 12: convolution kernel size 3×3, number of convolution kernels 1024, stride: 1; Activation layer 12: LeakyReLU; Pooling layer 6: Type: Max Pooling, Pooling size: 2×2, Stride: 2; Convolution layer 13: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1; Activation layer 13: LeakyReLU; Convolution layer 14: convolution kernel size 3×3, number of convolution kernels 2048, stride: 1; Activation layer 14: LeakyReLU; Pooling layer 7: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 1024; Concatenation layer 1: connects the feature map with the output of activation layer 12, and the number of channels increases to 2048; Convolution layer 15: kernel size 3×3, number of kernels 1024, stride: 1; Activation layer 15: LeakyReLU; Convolution layer 16: kernel size 3×3, number of kernels 1024, stride: 1; Activation layer 16: LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512; Concatenation layer 2: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024; Convolutional layer 17: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 17: LeakyReLU; Convolution layer 18: kernel size 3×3, number of kernels 512, stride: 1; Activation layer 18: LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 3: connects the feature map with the output of activation layer 8, and the number of channels increases to 512; Convolution layer 19: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 19: LeakyReLU; Convolutional layer 20: convolution kernel size 3×3, number of convolution kernels 256, stride: 1; Activation layer 20: LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 4: connects the feature map with the output of activation layer 6, and the number of channels increases to 256; Convolutional layer 21: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 21: LeakyReLU; Convolutional layer 22: convolution kernel size 3×3, number of convolution kernels 128, stride: 1; Activation layer 22: LeakyReLU; Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 5: connects the feature map with the output of activation layer 4, and the number of channels increases to 128; Convolutional layer 23: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 23: LeakyReLU; Convolutional layer 24: convolution kernel size 3×3, number of convolution kernels 64, stride: 1; Activation layer 24: LeakyReLU; Convolution upsampling 6: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32; Concatenation layer 6: connects the feature map with the output of activation layer 2, and the number of channels increases to 64; Convolutional layer 25: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 25: LeakyReLU; Convolutional layer 26: convolution kernel size 3×3, number of convolution kernels 32, stride: 1; Activation layer 26: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.

Images