Traffic jam monitoring method and device, equipment and storage medium

Abstract

The invention relates to a traffic jam monitoring method, device and equipment and a storage medium, and is applied to the field of deep learning, and the method comprises the steps: connecting a traffic monitoring camera, and obtaining traffic video data through the traffic monitoring camera; extracting traffic frame images in the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval; converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into a trained RSAN model; and calculating and outputting a traffic condition result of the current traffic RGB image. The technical effects of the invention are that the identification precision and the response speed of the system are improved, and the urban traffic congestion condition is efficiently and accurately identified.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a traffic congestion monitoring method, device, equipment and storage medium. Background Art

[0002] As urbanization accelerates, urban transportation systems are facing unprecedented challenges. According to World Bank data, about 20% of the world's urban population experiences severe traffic congestion every day, which not only wastes a lot of time and energy, but also aggravates air pollution and carbon emissions, affecting the sustainable development of cities and the quality of life of residents.

[0003] Traditional traffic congestion monitoring methods mainly rely on fixed-point sensor monitoring, such as geomagnetic sensors, infrared counters, etc. The monitoring method is to directly measure the number, speed and time interval of vehicles; it can also capture real-time images through video monitoring equipment, supplemented by manual observation and analysis to identify traffic congestion; in addition, there are also methods that indirectly calculate vehicle movement speed and traffic flow by analyzing the switching frequency and delay of mobile phone signals between base stations, thereby judging traffic conditions.

[0004] However, the current traffic congestion monitoring methods have limited recognition accuracy and it is difficult to accurately identify traffic congestion status in a complex and changeable traffic environment, which may cause users to misjudge traffic conditions, spend too much time on commuting, and affect residents' lives. Summary of the invention

[0005] In order to help solve the problem that current traffic congestion monitoring methods have limited recognition accuracy and are difficult to accurately identify traffic congestion status in complex and changeable traffic environments, thereby causing users to misjudge traffic conditions, spend too much time on commuting, and affect residents' lives, the present application provides a traffic congestion monitoring method, device, equipment and storage medium.

[0006] In a first aspect, the present application provides a traffic congestion monitoring method, which adopts the following technical solution: the method comprises:

[0007] Connecting to a traffic monitoring camera and acquiring traffic video data through the traffic monitoring camera;

[0008] Extracting a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval;

[0009] Converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into the trained RSAN model;

[0010] Calculate and output the traffic condition results of the current traffic RGB image.

[0011] In a specific implementation scheme, the connecting to a traffic monitoring camera and obtaining traffic video data through the traffic monitoring camera includes:

[0012] Obtaining identification information of the traffic monitoring camera from a preset camera database;

[0013] Create multiple threads, establish a communication connection between each thread and the traffic monitoring camera through the RTSP protocol, and obtain traffic video data.

[0014] In a specific implementation manner, before converting the traffic frame image into a traffic RGB image, the method further includes:

[0015] Using a bilateral filtering algorithm to filter the traffic frame image and generate a traffic filtered image;

[0016] Performing sharpening processing on the traffic filtered image and generating a traffic sharpened image;

[0017] The converting the traffic frame image into a traffic RGB image comprises:

[0018] The traffic sharpened image is converted into a traffic RGB image.

[0019] In a specific implementation scheme, the filtering process is calculated as follows:

[0020]

[0021] Among them, f p Represents the pixel value of the pixel p after filtering, Ω p represents the neighborhood of pixel p, W p represents the normalization factor, W p The calculation method is:

[0022]

[0023] Among them, G s (p,q) represents the spatial Gaussian weight function, G r (f p ,f q ) represents the range Gaussian weight function, G s (p,q) and G r (f p ,f q ) is calculated as:

[0024]

[0025] Where ||pq|| represents the Euclidean distance between pixel p and pixel q, σ srepresents the standard deviation of the spatial Gaussian function, σ r represents the standard deviation of the range Gaussian function;

[0026] The calculation method of the sharpening process is:

[0027] I shape =I origin +α×(I origin -I blurred )

[0028] Among them, I shape represents the traffic sharpened image, I origin represents the traffic filter image, I blurred represents an image obtained by performing Gaussian blur processing on the traffic filter image, and α represents a factor for controlling the sharpening intensity.

[0029] In a specific implementation scheme, the training method of the RSAN model includes:

[0030] Acquire traffic video training set data, extract frame image training data from the traffic video training set data, and filter and sharpen the frame image training data to generate clear image training data;

[0031] Converting the clear image training data into RGB image training data, and setting corresponding traffic condition labels for the RGB image training data, wherein the traffic condition labels include congested, general, and non-congested;

[0032] Constructing a spatial attention mechanism, and constructing a RSAN pre-training model based on the spatial attention mechanism;

[0033] Input the RGB image training data and the corresponding traffic condition labels into the RSAN pre-training model for processing, construct a loss function and perform a preset round of training;

[0034] The loss function is a cross entropy loss function, and the loss function is calculated as follows:

[0035]

[0036] Among them, y represents the probability distribution of the true label, represents the probability distribution predicted by the RSAN model, and c represents the type of the traffic condition label.

[0037] In a specific implementation scheme, the inputting the RGB image training data and the corresponding traffic condition labels into the RSAN pre-training model for processing includes:

[0038] Performing convolution processing on the RGB image training data to generate image convolution training data;

[0039] Performing batch normalization processing on the image convolution training data and generating image normalization training data;

[0040] Using an activation function to perform nonlinear transformation on the image normalized training data, and obtaining traffic feature data by calculating a residual block;

[0041] Performing spatial attention mechanism processing on the traffic feature data and generating traffic feature enhanced data;

[0042] Use multi-classification softmax layer to predict traffic congestion;

[0043] The calculation method of the residual block includes:

[0044] F(x)=H(x)+x

[0045] H(x)=W2×ReLU(W1×x+b1)+b2

[0046] Wherein, x represents the normalized image training data, H(x) represents the nonlinear transformation of the normalized image training data, F(x) represents the output of the residual block, W1 represents the weight matrix of the first convolutional layer, W2 represents the weight matrix of the second convolutional layer, b1 represents the bias vector of the first convolutional layer, b2 represents the bias vector of the second convolutional layer, and ReLU(·) represents the activation function.

[0047] In a specific implementation scheme, the processing of the traffic feature data by a spatial attention mechanism and generating traffic feature enhanced data includes:

[0048] The traffic characteristic data is subjected to global average pooling processing and global maximum pooling processing, and the calculation methods of the global average pooling processing and the global maximum pooling processing include:

[0049]

[0050] Wherein, x represents the input traffic feature data, W and H represent the height and width of the traffic feature image respectively, GAP(x) represents the result after global average pooling processing, and GMP(x) represents the result after global maximum pooling processing;

[0051] Aggregate the results of global average pooling and global maximum pooling to generate a pooling aggregation result;

[0052] The pooling aggregation result is converted into a channel importance vector through convolution calculation;

[0053] The channel importance vector is subjected to sigmoid function calculation to obtain an attention weight, and the calculation method of the attention weight includes:

[0054] AttentionMap=σ(Conv(concatenate(GAP(x),GMP(x))))

[0055] Among them, σ represents the sigmoid function, and AttentionMap represents the calculated attention weight;

[0056] The traffic feature data is multiplied by the corresponding attention weight to generate the traffic feature enhancement data, and the traffic feature enhancement data is calculated as follows:

[0057] AttendedFeatureMap=x⊙AttentionMap

[0058] Among them, ⊙ represents an element-by-element multiplication operation, and AttendedFeatureMap represents the traffic feature enhancement data.

[0059] In a second aspect, the present application provides a traffic congestion monitoring device, which adopts the following technical solution: the device comprises:

[0060] A video data acquisition module, used to connect to a traffic monitoring camera and acquire traffic video data through the traffic monitoring camera;

[0061] An image data generating module, used for extracting a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval;

[0062] An image format conversion module, used for converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into the trained RSAN model;

[0063] The traffic condition monitoring module is used to calculate and output the traffic condition results of the current traffic RGB image.

[0064] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the above-mentioned traffic congestion monitoring methods.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned traffic congestion monitoring methods.

[0066] In summary, this application has the following beneficial technical effects:

[0067] 1. High recognition accuracy: The RSAN model can automatically extract subtle features from video images, accurately quantify key indicators such as vehicle density and lane occupancy, and accurately identify traffic congestion even in complex and changing traffic environments;

[0068] 2. Fast response speed: Although the model has a large amount of calculation, through algorithm optimization and hardware acceleration, the RSAN model can achieve a relatively fast reasoning speed while ensuring high detection accuracy, which is particularly important for real-time traffic monitoring and immediate congestion response;

[0069] 3. Wide applicability: The RSAN model can be flexibly adjusted and is applicable to traffic networks of different sizes and types. Whether it is urban streets, highways or complex intersections, the detection effect can be optimized by adjusting the model parameters and structure;

[0070] 4. Strong robustness: Since the combination of attention mechanism and deep learning can capture the characteristics of traffic scenes from multiple scales and angles, the RSAN model can cope with complex road environments and lighting changes, reducing the impact of external interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a flow chart of a traffic congestion monitoring method in an embodiment of the present application;

[0072] Figure 2 Schematic diagram of a real-time monitoring system for urban traffic congestion in an embodiment of the present application;

[0073] Figure 3 It is a schematic diagram of the structure of the RSAN model integrating the attention mechanism in the embodiment of the present application;

[0074] Figure 4 is a schematic diagram of a traffic congestion monitoring device in an embodiment of the present application;

[0075] Figure 5 It is a schematic diagram used to embody a computer device in an embodiment of the present application.

[0076] Figure numerals: 401, video data acquisition module; 402, image data generation module; 403, image format conversion module; 404, traffic condition monitoring module. DETAILED DESCRIPTION

[0077] The following combination Figure 1-Figure 5 This application is described in further detail.

[0078] The embodiment of the present application discloses a traffic congestion monitoring method, which is applied to a real-time monitoring system for urban traffic congestion. The traffic congestion monitoring method can monitor on-site traffic conditions in real time, effectively distinguish different degrees of traffic congestion, and deeply analyze the traffic pressure distribution in various areas of the city, extract and make congestion warnings, thereby objectively reflecting the comprehensive operating status of the urban traffic system.

[0079] With the acceleration of urbanization, urban transportation systems are facing unprecedented challenges. According to World Bank data, about 20% of the world's urban population experiences serious traffic congestion every day, which may lead to a lot of time and energy waste, aggravate air pollution and carbon emissions, and affect the sustainable development of cities and the quality of life of residents. There are already some ways to monitor traffic congestion conditions. However, the current monitoring methods have limitations such as incomplete data coverage, insufficient real-time performance, limited recognition accuracy, and lack of prediction capabilities, making it difficult for users to obtain accurate traffic information, which may misjudge traffic conditions and choose to travel on congested roads, affecting residents' lives. In order to help improve the recognition accuracy of traffic congestion conditions and provide users with accurate traffic information, the present application provides a traffic congestion monitoring method.

[0080] Reference Figure 1 , the method comprises the following steps:

[0081] S10, connecting to a traffic monitoring camera and obtaining traffic video data through the traffic monitoring camera.

[0082] Specifically, by connecting to high-definition traffic monitoring cameras throughout the city's main roads and traffic nodes, real-time traffic video stream data is continuously collected to ensure data continuity. Among them, the methods for obtaining traffic video stream data may include: Figure 2 First, obtain the identification information of the traffic monitoring camera from the preset camera database. The identification information of the camera includes ID (camera unique identifier), URL (camera video stream address) information, etc., which is the basic information of the monitoring camera; the camera database is a database that stores the basic information of all traffic monitoring cameras, which can be used to manage and query the configuration information of the camera to ensure that the system can dynamically obtain and manage camera resources. After that, create multiple threads, establish a communication connection between each thread and the traffic monitoring camera through the RTSP protocol, pass in the parameters of the traffic monitoring camera, and obtain traffic video data.

[0083] It should be noted that the number of created threads is not necessarily the same as the number of traffic monitoring cameras. Usually, each thread is responsible for communicating with one or more cameras to obtain video stream data. The number of threads may be determined according to the system's concurrency capability, the number of cameras, and the actual hardware resources. If system resources are limited, the number of threads may be limited by using a thread pool, and the connection requests of multiple cameras may be processed through a scheduling mechanism. During the process of traffic congestion monitoring, if traffic monitoring cameras are added or reduced, the corresponding number of threads can be automatically increased or decreased.

[0084] S20, extracting a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval.

[0085] Specifically, the raw data acquired by the system is video stream data, and image data is used in the actual data processing process. Therefore, frame image data is extracted to facilitate the subsequent image processing process. The camera is connected to the communication interface for communication, the frame rate (fps) of the connected traffic camera is detected, and the camera information is saved. The saved camera information includes ID, URL, frame rate, connection parameters, and information about the location or area of ​​the camera. Then, the traffic frame image is extracted according to the frame rate of the traffic monitoring camera and the preset frame interval. In the implementation mode of the present application, OpenCV is used to evenly extract frame images from the video stream data. The extraction method is to perform timed extraction through the frame rate and frame interval. The calculation method of the number of frames extracted per second can be expressed as:

[0086]

[0087] Among them, F s Indicates the frame rate of the camera, D f Indicates the set frame interval, N s Indicates the number of frames of the image extracted per second.

[0088] It should be noted that the frame interval is usually set according to the frame rate (fps) of the camera and the requirements of the system; for example, if the frame rate of the camera is 30fps and the system only needs to obtain 1 frame of image per second, the frame interval can be set to 30 frames. The actual implementation method can be to implement the frame interval through counting or timestamp; for example, every time 30 frames of video stream data are received, 1 frame of image is extracted for processing.

[0089] S30, converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into the trained RSAN model.

[0090] Specifically, the image format that can be processed by the deep learning model is RGB format, so the image format of the extracted traffic frame image is converted into RGB format, and the generated traffic RGB image is input into the trained RSAN model to facilitate the model to process the image.

[0091] The trained RSAN model used in the embodiment of the present application is a deep residual network model (Residual Network with Spatial Attention, RSAN) that integrates the attention mechanism. The RSAN model uses ResNet18 as the basic model. ResNet18 has high efficiency and accuracy in image classification tasks. Through the deep residual network, RSAN can effectively extract key features in traffic videos, reduce computational redundancy, and maintain high-precision congestion detection capabilities. Figure 2 and 3 The RSAN model introduces a spatial attention mechanism that can adaptively highlight the most critical parts of the image for judging traffic congestion; by focusing on important areas, the model reduces the amount of computation on irrelevant areas, thereby improving the reasoning speed.

[0092] S40, calculating and outputting the traffic condition result of the current traffic RGB image.

[0093] Specifically, the deep residual network RSAN model calculates and outputs the traffic condition results of the current frame image. The output of the traffic condition results includes three conditions: no congestion, general and congestion. With the help of the RSAN model, the frame image in RGB format after data preprocessing is analyzed to accurately locate the position of the vehicle in each frame image and accurately identify the degree of traffic congestion. Once a possible traffic congestion is predicted, the system can immediately issue an early warning to the traffic management department to effectively guide traffic flow and reduce congestion pressure.

[0094] It should be noted that after the RSAN model predicts traffic conditions, the system will store the captured traffic views, congestion analysis data and other information in real time, and support flexible query, historical comparison and other functions, so as to facilitate the formulation of response measures and long-term plans based on traffic condition data in the later stage.

[0095] In the present application scheme, real-time traffic video data is obtained by connecting to urban traffic monitoring cameras, and frame images are extracted by processing the obtained traffic video stream data to obtain traffic frame images that can be used for model analysis. The RSAN (deep residual network) model that integrates the attention mechanism is used to analyze and process the frame images, which can improve the recognition accuracy of traffic congestion. The RSAN model can automatically extract subtle features in video images and accurately quantify key indicators such as vehicle density and lane occupancy, so that traffic congestion status can be accurately identified even in complex and changeable traffic environments. In addition, the RSAN model can effectively extract key features in traffic videos and reduce computational redundancy, and the spatial attention mechanism reduces the amount of computation on irrelevant areas by focusing on important areas, improves the reasoning speed, and thus improves the response speed of the system.

[0096] In one embodiment, considering that the original acquired video data may contain noise after being processed by frame image extraction, which may affect the recognition accuracy of the model for vehicles and lanes, the following steps are further performed before converting the traffic frame image into a traffic RGB image:

[0097] First, the resolution of the frame image is uniformly adjusted so that the frame image meets the image size requirements of the model input. After adjusting the resolution, the traffic frame image is filtered using the bilateral filtering algorithm to generate a traffic filtered image. The bilateral filtering algorithm can remove the noise generated by video compression and some unnecessary details. The calculation method of the bilateral filtering process can be expressed as:

[0098]

[0099] Among them, f p Represents the pixel value of the pixel p after filtering, Ω p represents the neighborhood of pixel p, which can also be understood as the set of pixels involved in the calculation, W p Represents the normalization factor to ensure the correct range of output pixel values, W p The calculation method can be expressed as:

[0100]

[0101] Among them, G s (p,q) represents the spatial Gaussian weight function, which depends on the spatial distance between pixel p and pixel q, G r (f p ,f q ) represents the range Gaussian weight function, which depends on the pixel value f p and pixel value f q The difference between G s (p,q) and G r(f p ,f q ) can be calculated as:

[0102]

[0103] Where ||pq|| represents the Euclidean distance between pixel p and pixel q, σ s represents the standard deviation of the spatial Gaussian function, controlling the influence range of spatial proximity, σ r Represents the standard deviation of the range Gaussian function, controlling the influence of pixel value similarity.

[0104] After the filtering process is completed, the obtained traffic filtered image is sharpened to generate a traffic sharpened image. Specifically, the traffic filtered image is sharpened by first using Gaussian blur to create a blurred image, then subtracting its blurred version from the original image, and then adding the result to the original image to enhance the edges and details of the image and achieve a sharpening effect; the calculation method of the sharpening process can be expressed as:

[0105] I shape =I origin +α×(I origin -I blurred )

[0106] Among them, I shape represents the traffic sharpened image, I origin represents the traffic filter image, I blurred It represents an image obtained by Gaussian blurring the traffic filtered image. α represents a factor that controls the sharpening intensity, which is usually between 0 and 1. It determines the influence of the difference between the blurred image and the original image. It can also be understood that α determines the influence of the difference between the blurred image and the traffic filtered image.

[0107] After filtering and sharpening, converting the traffic frame image into a traffic RGB image includes: converting the traffic sharpened image into a traffic RGB image.

[0108] In the present application scheme, considering that there may be some noise in the process of video compression and frame image extraction, and there may also be some unnecessary details in the original data, the noise and unnecessary details can be removed through filtering processing to improve the image quality; the sharpening processing can enhance the edges and details in the image, making the processed image look clearer, improving the visual effect, and further improving the image quality.

[0109] In one embodiment, the training method of the RSAN model may be specifically performed as follows:

[0110] First, traffic video training set data is obtained for model training; the training set data can be obtained by directly obtaining existing video data, or by connecting with traffic monitoring cameras to obtain actual data for training. The actual traffic data can be obtained by creating multiple threads through a computer, establishing a communication connection between each thread and a traffic monitoring camera, and obtaining video stream data; then, the frame rate of the camera is detected, the frame interval is set, and frame image training data is extracted from the traffic video training set data. The extracted frame images are used for model training. The number of image frames extracted per second can be determined according to the needs of the system and users, and there is no restriction here. Considering that the extracted frame images may have noise and unclear images during video compression and other processing, before training the model, the frame image training data is first filtered and sharpened to generate clear image training data to improve image quality, thereby contributing to the accuracy of model training. The filtering adopts the bilateral filtering algorithm, and the filtering and sharpening processing methods are consistent with the filtering and sharpening processing methods used in the previous article, which will not be repeated here.

[0111] Afterwards, the clear image training data is converted into RGB image training data. Considering that the format of the image data extracted from the video stream data may not conform to the image format of the model input, RGB conversion is performed to generate images in RGB format for model training. At the same time, corresponding traffic condition labels are set for the RGB image training data. The traffic condition labels include congested, general, and non-congested. The label can be set by mapping the average number of vehicles per N frames to the congestion degree label.

[0112] After the image data is preprocessed, a spatial attention mechanism is constructed, and an RSAN pre-training model is constructed based on the spatial attention mechanism. The structural diagram of the constructed RSAN model integrating the attention mechanism is shown in Figure 3 . Afterwards, the RGB image training data and the corresponding traffic condition labels are input into the RSAN pre-training model for processing. In the implementation of the present application, the RSAN model includes 12 residual blocks, which extract features in traffic monitoring video frames through residual blocks of different scales. Each convolution layer in the residual block is accompanied by a ReLU activation function and a pooling layer to reduce the spatial dimension of the feature map and retain important information.

[0113] Finally, a loss function is constructed and a preset number of rounds of training is performed. For example, the number of training rounds can be set to 50. After 50 rounds of training, the model training is completed and the trained model is saved. The loss function is the cross entropy loss function, and the loss function is calculated as follows:

[0114]

[0115] Among them, y represents the probability distribution of the true label, represents the probability distribution predicted by the RSAN model, and c represents the type of traffic condition label, which can also be understood as the type of predicted congestion condition.

[0116] Specifically, the RSAN pre-training model trains data in the following ways: first, convolution is performed on the RGB image training data to generate image convolution training data; batch normalization is performed on the image convolution training data to generate image normalization training data; the goal of batch normalization is to convert the input x in each mini-batch into i Normalize them so that they have zero mean and unit variance. The layer is usually applied after each convolutional layer to speed up the training process and improve the stability and generalization ability of the model. Batch normalization reduces internal covariate shift by standardizing the input of each layer. The calculation method of batch normalization can be expressed as:

[0117]

[0118] Among them, x i represents the i-th sample of the input, μ B and Respectively represent the mean and variance of the data in the mini-batch, γ and β are learnable scaling and translation parameters, ∈ represents a constant, which is a very small constant used to avoid the situation where the divisor is zero. represents the normalized sample, y i Represents the predicted value.

[0119] Among them, mini-batch refers to a smaller subset of data selected from the training data set. When training a model, the entire training data set is usually divided into multiple mini-batches, each mini-batch contains a certain number of training samples, and the number of these samples is usually a power of 2, such as 32, 64, 128, etc.

[0120] Afterwards, the activation function is used to perform nonlinear transformation on the image normalization training data, and the traffic feature data is obtained by calculating the residual block; the calculation method of the residual block includes:

[0121] F(x)=H(x)+x

[0122] H(x)=W2×ReLU(W1×x+b1)+b2

[0123] Among them, x represents the image normalized training data, H(x) represents the nonlinear transformation of the image normalized training data, F(x) represents the output of the residual block, W1 represents the weight matrix of the first convolutional layer, W2 represents the weight matrix of the second convolutional layer, b1 represents the bias vector of the first convolutional layer, b2 represents the bias vector of the second convolutional layer, and ReLU() represents the activation function.

[0124] The traffic feature data is processed by the spatial attention mechanism and traffic feature enhanced data is generated. The spatial attention mechanism module processes the feature map of the previous layer. The fully connected layer is used as the output layer at the end of the network to convert the learned features into outputs for specific tasks. In traffic congestion identification, a multi-classification softmax layer is used to realize multi-level traffic congestion condition prediction.

[0125] Among them, the method of processing the traffic feature data with the spatial attention mechanism can be specifically implemented as follows:

[0126] Perform global average pooling and global maximum pooling on the traffic feature data to obtain the global average pooling and maximum pooling results of the feature channel. The calculation methods of the global average pooling and global maximum pooling include:

[0127]

[0128]

[0129] Where x represents the input traffic feature data, W and H represent the height and width of the traffic feature image, respectively, GAP(x) represents the result after global average pooling, and GMP(x) represents the result after global maximum pooling;

[0130] The results of global average pooling and global maximum pooling are aggregated (Concatenate) to generate pooling aggregation results; the pooling aggregation results are converted into channel importance vectors through convolution calculation, which can also be understood as converting them into a vector with the same number of feature map channels through a convolution layer. This vector represents the importance of each channel; then, the channel importance vector is calculated by sigmoid function to obtain attention weights. The sigmoid function converts the channel importance vector into an attention weight between 0 and 1, which represents the contribution of each channel to the current task. The calculation method of attention weights includes:

[0131] AttentionMap=σ(Conv(concatenate(GAP(x),GMP(x))))

[0132] Among them, σ represents the sigmoid function, and AttentionMap represents the calculated attention weight;

[0133] Finally, the traffic feature data is multiplied by the corresponding attention weight to generate traffic feature enhancement data. It can also be understood that each channel of the original feature map is multiplied by the corresponding attention weight to achieve spatial weighting of the feature map, enhance the feature expression of important areas, and suppress the interference of irrelevant areas. The calculation method of traffic feature enhancement data can be expressed as:

[0134] AttendedFeatureMap=x⊙AttentionMap

[0135] Among them, ⊙ represents the element-by-element multiplication operation, and AttendedFeatureMap represents the traffic feature enhancement data.

[0136] The core of the spatial attention mechanism is to extract global context information through pooling operations, then learn attention weights through convolutional layers, and finally use these weights to adjust the original feature map so that the model can focus more on the key parts of the input data.

[0137] In this application, by constructing an RSAN model with an attention mechanism, subtle features in video images are extracted, key indicators such as vehicle density and lane occupancy are accurately quantified, and traffic congestion can be accurately identified even in complex and changing traffic environments. The deep residual network model (RSAN) with a spatial attention mechanism can focus more on image features related to traffic congestion. In addition, the attention mechanism combined with deep learning can capture the characteristics of traffic scenes from multiple scales and angles, thereby improving the accuracy and robustness of traffic congestion detection, coping with complex road environments and lighting changes, and reducing the impact of external interference.

[0138] Figure 1 FIG. 1 is a flow chart of a traffic congestion monitoring method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0139] Based on the above method, an embodiment of the present application also discloses a traffic congestion monitoring device.

[0140] Reference Figure 4 , the device includes the following modules:

[0141] The video data acquisition module 401 is used to connect to the traffic monitoring camera and acquire traffic video data through the traffic monitoring camera;

[0142] An image data generating module 402 is used to extract a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval;

[0143] An image format conversion module 403 is used to convert the traffic frame image into a traffic RGB image, and input the traffic RGB image into the trained RSAN model;

[0144] The traffic condition monitoring module 404 is used to calculate and output the traffic condition result of the current traffic RGB image.

[0145] In one embodiment, the video data acquisition module 401 is specifically used to obtain identification information of a traffic monitoring camera from a preset camera database; create multiple threads, establish a communication connection between each thread and the traffic monitoring camera through the RTSP protocol, and obtain traffic video data.

[0146] In one embodiment, the image format conversion module 403 is further used to filter the traffic frame image using a bilateral filtering algorithm and generate a traffic filtered image; sharpen the traffic filtered image and generate a traffic sharpened image; converting the traffic frame image into a traffic RGB image includes: converting the traffic sharpened image into a traffic RGB image.

[0147] In one embodiment, the calculation method of the filtering process in the image format conversion module 403 is:

[0148]

[0149] Among them, f p Represents the pixel value of the pixel p after filtering, Ω p represents the neighborhood of pixel p, W p represents the normalization factor, W p The calculation method is:

[0150]

[0151] Among them, G s (p,q) represents the spatial Gaussian weight function, G r (f p ,f q ) represents the range Gaussian weight function, G s (p,q) and G r (fp ,f q ) is calculated as:

[0152]

[0153] Where ||pq|| represents the Euclidean distance between pixel p and pixel q, σ s represents the standard deviation of the spatial Gaussian function, σ r represents the standard deviation of the range Gaussian function;

[0154] The sharpening process is calculated as:

[0155] I shape =I origin +α×(I origin -I blurred )

[0156] Among them, I shape represents the traffic sharpened image, I origin represents the traffic filter image, I blurred represents the image after Gaussian blurring of the traffic filter image, and α represents the factor that controls the sharpening strength.

[0157] In one embodiment, the image format conversion module 403 is further used to obtain traffic video training set data, extract frame image training data from the traffic video training set data, and filter and sharpen the frame image training data to generate clear image training data; convert the clear image training data into RGB image training data, and set corresponding traffic condition labels for the RGB image training data, and the traffic condition labels include congestion, general, and non-congestion; construct a spatial attention mechanism, and construct an RSAN pre-training model based on the spatial attention mechanism; input the RGB image training data and the corresponding traffic condition labels into the RSAN pre-training model for processing, construct a loss function and perform a preset round of training; the loss function is a cross entropy loss function, and the loss function is calculated as follows:

[0158]

[0159] Among them, y represents the probability distribution of the true label, represents the probability distribution predicted by the RSAN model, and c represents the type of traffic condition label.

[0160] In one embodiment, the image format conversion module 403 is further used to perform convolution processing on the RGB image training data and generate image convolution training data; perform batch normalization processing on the image convolution training data and generate image normalization training data; use an activation function to perform nonlinear transformation on the image normalization training data and obtain traffic feature data by calculating the residual block; perform spatial attention mechanism processing on the traffic feature data and generate traffic feature enhancement data; use a multi-classification softmax layer to realize traffic congestion prediction; the calculation method of the residual block includes:

[0161] F(x)=H(x)+x

[0162] H(x)=W2×ReLU(W1×x+b1)+b2

[0163] Where x represents the normalized image training data, H(x) represents the nonlinear transformation of the normalized image training data, F(x) represents the output of the residual block, W1 represents the weight matrix of the first convolutional layer, W2 represents the weight matrix of the second convolutional layer, b1 represents the bias vector of the first convolutional layer, b2 represents the bias vector of the second convolutional layer, and ReLU(·) represents the activation function.

[0164] In one embodiment, the image format conversion module 403 is further used to perform global average pooling processing and global maximum pooling processing on the traffic feature data. The calculation methods of the global average pooling processing and the global maximum pooling processing include:

[0165]

[0166] Where x represents the input traffic feature data, W and H represent the height and width of the traffic feature image, respectively, GAP(x) represents the result after global average pooling, and GMP(x) represents the result after global maximum pooling;

[0167] The results of global average pooling and global maximum pooling are aggregated to generate pooling aggregation results; the pooling aggregation results are converted into channel importance vectors through convolution calculation; the channel importance vectors are calculated by sigmoid function to obtain attention weights. The calculation methods of attention weights include:

[0168] AttentionMap=σ(Conv(concatenate(GAP(x),GMP(x))))

[0169] Among them, σ represents the sigmoid function, and AttentionMap represents the calculated attention weight;

[0170] The traffic feature data is multiplied by the corresponding attention weight to generate traffic feature enhancement data. The calculation method of traffic feature enhancement data is:

[0171] AttendedFeatureMap=x⊙AttentionMap

[0172] Among them, ⊙ represents the element-by-element multiplication operation, and AttendedFeatureMap represents the traffic feature enhancement data.

[0173] The traffic congestion monitoring device provided in the embodiment of the present application can be applied to the traffic congestion monitoring method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0174] It should be noted that: the traffic congestion monitoring device provided in the embodiment of the present application only uses the division of the above-mentioned functional modules / functional units as an example when performing traffic congestion monitoring. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the traffic congestion monitoring device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the traffic congestion monitoring method provided in the above method embodiment and the implementation method of the traffic congestion monitoring device provided in this embodiment belong to the same concept. The specific implementation process of the traffic congestion monitoring device provided in this embodiment is detailed in the above method embodiment, which will not be repeated here.

[0175] The embodiment of the present application also discloses a computer device.

[0176] Specifically, if Figure 5As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0177] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0178] The embodiment of the present application also discloses a computer-readable storage medium.

[0179] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the process of the implementation of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

[0180] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A traffic congestion monitoring method, characterized in that: The method comprises: Connecting to a traffic monitoring camera and acquiring traffic video data through the traffic monitoring camera; Extracting a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval; Converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into the trained RSAN model; Calculate and output the traffic condition results of the current traffic RGB image.

2. The method according to claim 1, characterized in that: The connecting to the traffic monitoring camera and obtaining the traffic video data through the traffic monitoring camera includes: Obtaining identification information of the traffic monitoring camera from a preset camera database; Create multiple threads, establish a communication connection between each thread and the traffic monitoring camera through the RTSP protocol, and obtain traffic video data.

3. The method according to claim 1, characterized in that: Before converting the traffic frame image into a traffic RGB image, the method further includes: Using a bilateral filtering algorithm to filter the traffic frame image and generate a traffic filtered image; Performing sharpening processing on the traffic filtered image and generating a traffic sharpened image; The converting the traffic frame image into a traffic RGB image comprises: The traffic sharpened image is converted into a traffic RGB image.

4. The method according to claim 3, characterized in that: The calculation method of the filtering process is: Among them, f p Represents the pixel value of the pixel p after filtering, Ω p represents the neighborhood of pixel p, W p represents the normalization factor, W p The calculation method is: Among them, G s (p,q) represents the spatial Gaussian weight function, G r (f p ,f q ) represents the range Gaussian weight function, G s (p,q) and G r (f p ,f q ) is calculated as: Where ||pq|| represents the Euclidean distance between pixel p and pixel q, σ s represents the standard deviation of the spatial Gaussian function, σ r represents the standard deviation of the range Gaussian function; The calculation method of the sharpening process is: I shape =I origin +α×(I origin -I blurred ) Among them, I shape represents the traffic sharpened image, I origin represents the traffic filter image, I blurred represents an image obtained by performing Gaussian blur processing on the traffic filter image, and α represents a factor for controlling the sharpening intensity.

5. The method according to claim 1, characterized in that: The training method of the RSAN model includes: Acquire traffic video training set data, extract frame image training data from the traffic video training set data, and perform filtering and sharpening processing on the frame image training data to generate clear image training data; Converting the clear image training data into RGB image training data, and setting corresponding traffic condition labels for the RGB image training data, wherein the traffic condition labels include congested, general, and non-congested; Constructing a spatial attention mechanism, and constructing a RSAN pre-training model based on the spatial attention mechanism; Input the RGB image training data and the corresponding traffic condition labels into the RSAN pre-training model for processing, construct a loss function and perform a preset round of training; The loss function is a cross entropy loss function, and the loss function is calculated as follows: Among them, y represents the probability distribution of the true label, represents the probability distribution predicted by the RSAN model, and c represents the type of the traffic condition label.

6. The method according to claim 5, characterized in that: The inputting the RGB image training data and the corresponding traffic condition label into the RSAN pre-training model process comprises: Performing convolution processing on the RGB image training data to generate image convolution training data; Performing batch normalization processing on the image convolution training data and generating image normalization training data; Using an activation function to perform nonlinear transformation on the image normalized training data, and obtaining traffic feature data by calculating a residual block; Performing spatial attention mechanism processing on the traffic feature data and generating traffic feature enhanced data; Use multi-classification softmax layer to predict traffic congestion; The calculation method of the residual block includes: F(x)=H(x)+x H(x)=W2×ReLU(W1×x+b1)+b2 Wherein, x represents the normalized image training data, H(x) represents the nonlinear transformation of the normalized image training data, F(x) represents the output of the residual block, W1 represents the weight matrix of the first convolutional layer, W2 represents the weight matrix of the second convolutional layer, b1 represents the bias vector of the first convolutional layer, b2 represents the bias vector of the second convolutional layer, and ReLU(·) represents the activation function.

7. The method according to claim 6, characterized in that: The performing of spatial attention mechanism processing on the traffic feature data and generating traffic feature enhanced data comprises: The traffic characteristic data is subjected to global average pooling processing and global maximum pooling processing, and the calculation methods of the global average pooling processing and the global maximum pooling processing include: Wherein, x represents the input traffic feature data, W and H represent the height and width of the traffic feature image respectively, GAP(x) represents the result after global average pooling processing, and GMP(x) represents the result after global maximum pooling processing; Aggregate the results of global average pooling and global maximum pooling to generate a pooling aggregation result; The pooling aggregation result is converted into a channel importance vector through convolution calculation; The channel importance vector is subjected to sigmoid function calculation to obtain an attention weight, and the calculation method of the attention weight includes: AttentionMap=σ(Conv(concatenate(GAP(x),GMP(x)))) Among them, σ represents the sigmoid function, and AttentionMap represents the calculated attention weight; The traffic feature data is multiplied by the corresponding attention weight to generate the traffic feature enhancement data, and the traffic feature enhancement data is calculated as follows: AttendedFeatureMap=x⊙AttentionMap Among them, ⊙ represents an element-by-element multiplication operation, and AttendedFeatureMap represents the traffic feature enhancement data.

8. A traffic congestion monitoring device, characterized in that: The device comprises: A video data acquisition module (401) is used to connect to a traffic monitoring camera and acquire traffic video data through the traffic monitoring camera; An image data generating module (402) is used to extract a traffic frame image from the traffic video data according to the frame rate of the traffic monitoring camera and a preset frame interval; An image format conversion module (403), used for converting the traffic frame image into a traffic RGB image, and inputting the traffic RGB image into the trained RSAN model; The traffic condition monitoring module (404) is used to calculate and output the traffic condition result of the current traffic RGB image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images