Road traffic state detection method and device based on deep learning, medium and equipment

Abstract

This invention discloses a deep learning-based road traffic state detection method, comprising: acquiring a traffic image training set; performing R-Mosaic data augmentation processing on the training samples in the traffic image training set; training a pre-defined H-FPN deep convolutional network using the R-Mosaic data-augmented training samples, wherein the H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image; acquiring a traffic image to be processed; extracting the corresponding feature parameters of the traffic image using the trained H-FPN deep convolutional network; and generating and outputting a traffic state prediction value based on the traffic flow feature parameters using a pre-defined discriminant model. This invention solves the problems of existing technologies in detecting road traffic states, such as single indicators, slow detection speed, low information extraction, and low detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium and equipment for detecting road traffic conditions based on deep learning. Background Technology

[0002] Traffic conditions are typically described using terms like free flow, steady flow, unstable flow, and forced flow. The standards for traffic flow conditions vary across countries, and the criteria for judging traffic congestion differ. The level of service (LSS) is used to describe the state of traffic flow. According to standards issued in the United States, LSS is usually measured using factors such as speed, workload, traffic density, driving freedom, and comfort. LSS is typically defined as having six levels. Based on the U.S. road traffic service level standards, Japan has also established a three-level classification of traffic service.

[0003] In my country, congestion levels are classified based on the average speed of vehicles on urban arterial roads. This indicator is too simplistic and cannot fully describe the actual road conditions. Previously, vehicle information was often obtained using sensing devices and infrared sensors, which were costly and inaccurate. Current road traffic state recognition methods based on deep convolutional neural networks typically involve first inputting traffic images into the backbone network to extract target information; then, a prediction model is used to further predict the state for the next time period. However, this approach suffers from slow model training, low information extraction accuracy and speed, and low discrimination precision. Summary of the Invention

[0004] This invention provides a method, apparatus, medium, and device for detecting road traffic conditions based on deep learning, in order to solve the problems of single indicators, slow detection speed, low information extraction, and low detection accuracy in the existing technology for detecting road traffic conditions.

[0005] A deep learning-based road traffic state detection method, the method comprising:

[0006] Obtain a traffic image training set, and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set;

[0007] The H-FPN deep convolutional network is trained using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image.

[0008] Acquire a traffic image to be processed, and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network;

[0009] Traffic state prediction values ​​are generated and output based on the characteristic parameters of the traffic flow using a preset discrimination model.

[0010] Optionally, the step of acquiring a traffic image training set and performing R-Mosaic data augmentation processing on the training samples in the traffic image training set includes:

[0011] Obtain a traffic image training set, wherein each training sample in the traffic image training set includes a reference box;

[0012] The training samples are grouped into pairs and then horizontally flipped, scaled, and subjected to color gamut changes.

[0013] Construct an image frame that is twice the size of the training sample. Place one training sample in the upper left corner of the image frame and another training sample in the upper right corner of the image frame.

[0014] A portion of the training sample image is cropped and pasted onto the image frame, and then stretched vertically to fill the image frame;

[0015] The size and position of the image frame are readjusted, and the integrated image frame is translated, scaled, and cropped to obtain an enhanced image of the same size as the training sample.

[0016] Optionally, the H-FPN deep convolutional network includes three prediction layers, namely a first prediction layer, a second prediction layer, and a third prediction layer;

[0017] The first prediction layer is used to obtain a feature map with a receptive field of 76*76 by upsampling, and to predict the target in the upper receptive field by a preset first reference box, and output the prediction information of the upper half of the feature map.

[0018] The second prediction layer is used to obtain a feature map with a receptive field of 38*38 by upsampling, and to predict the target in the entire receptive field by a preset second reference box, and output the prediction information of the entire feature map.

[0019] The third prediction layer is used to obtain a feature map with a receptive field of 19*19 through upsampling, predict the target in the lower half of the receptive field through a preset third reference box, and output the prediction information of the lower half of the feature map.

[0020] The H-FPN deep convolutional network also includes a detection and tracking module, which is used to obtain the feature parameters of traffic flow based on the prediction information output by the first prediction layer, the second prediction layer and the third prediction layer.

[0021] Optionally, the characteristic parameters include traffic flow speed, traffic flow density, and traffic volume.

[0022] Optionally, the classification loss function of the H-FPN deep convolutional network consists of a regression loss function and a classification loss function, and its calculation formula is as follows:

[0023] Loss = reg_loss + pb_loss

[0024] Where reg_loss represents regression loss and pb_loss represents classification loss;

[0025] The formula for calculating regression loss reg_loss is:

[0026]

[0027] Among them, y i Indicates the actual observed value; Indicates the predicted value;

[0028] The formula for calculating the classification loss pb_loss is:

[0029]

[0030] Where pre_label represents the predicted state; rel_label represents the actual state.

[0031] Optionally, the preset discrimination model includes a matrix composed of discrimination thresholds for traffic flow speed, traffic flow density, and traffic volume;

[0032]

[0033] Where P represents the discrimination matrix, i.e., the matrix composed of discrimination thresholds, i≤3, j≤3; P11 represents the minimum speed of traffic flow in a smooth state; P12 represents the minimum speed of traffic flow in a congested state; P13 represents the maximum speed of traffic flow in a congested state; P21 represents the minimum density of traffic flow in a smooth state; P22 represents the minimum density of traffic flow in a congested state; P23 represents the maximum density of traffic flow in a congested state; P31 represents the maximum quantity of traffic volume in a smooth state; P32 represents the maximum quantity of traffic volume in a congested state; P33 represents the minimum quantity of traffic volume in a congested state.

[0034] Optionally, the step of generating and outputting traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model includes:

[0035] The traffic flow speed, traffic flow density, and traffic volume are compared with the discrimination thresholds in the discrimination matrix using the discrimination model to obtain the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume.

[0036] If the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume are all different, the output traffic state prediction value is congestion.

[0037] If the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume are not completely the same, the traffic state information that appears most frequently is selected as the traffic state prediction value.

[0038] A deep learning-based road traffic condition detection device, the device comprising:

[0039] The preprocessing module is used to acquire a traffic image training set and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set.

[0040] The training module is used to train a pre-defined H-FPN deep convolutional network using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image.

[0041] The extraction module is used to acquire the traffic image to be processed and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network.

[0042] The prediction module is used to generate and output traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the deep learning-based road traffic state detection method described above.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the deep learning-based road traffic state detection method as described above.

[0045] This invention provides a method for detecting road traffic conditions based on deep learning. It acquires a traffic image training set and performs R-Mosaic data augmentation on the training samples within the set. The augmented samples are then used to train a pre-defined H-FPN deep convolutional network, which employs three prediction layers to extract traffic flow feature parameters from the input images. The method then acquires the traffic images to be processed and extracts the corresponding feature parameters using the trained H-FPN network. Finally, a pre-defined discrimination model generates and outputs predicted traffic conditions based on these feature parameters. This invention enhances the robustness of road traffic condition detection through a robust multi-index evaluation system and the establishment of a deep learning convolutional neural network. It also improves detection speed, information extraction, and detection accuracy, thereby enhancing the overall accuracy of the detection results. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a road traffic state detection method based on deep learning provided in an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating the implementation of step S101 in a deep learning-based road traffic state detection method according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of R-Mosaic data augmentation processing provided in an embodiment of the present invention, wherein... Figure 3 (a) is a schematic diagram corresponding to step S201. Figure 3 (b) is a schematic diagram corresponding to steps S202 and S203. Figure 3 (c) is a schematic diagram corresponding to steps S204 and S205;

[0050] Figure 4 This is a schematic diagram of H-FPN network operation provided in an embodiment of the present invention;

[0051] Figure 5 This is a flowchart illustrating the implementation of step S104 in a deep learning-based road traffic state detection method according to an embodiment of the present invention.

[0052] Figure 6This is a schematic diagram of the structure of a road traffic condition detection device based on deep learning provided in an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] This invention provides a road traffic condition detection method based on deep learning. By improving the multi-index evaluation system and establishing a deep learning convolutional neural network, the robustness of the road traffic condition detection effect is enhanced, the detection speed is increased, and the accuracy of the detection results is improved.

[0056] The following provides a detailed description of the deep learning-based road traffic state detection method provided in this embodiment, such as... Figure 1 As shown, the deep learning-based road traffic state detection method includes:

[0057] In step S101, a traffic image training set is obtained, and R-Mosaic data augmentation processing is performed on the training samples in the traffic image training set.

[0058] Optionally, the R-Mosaic data augmentation processing refers to semi-structured mosaic data augmentation processing. For example... Figure 2 As shown, step S101, which involves obtaining a traffic image training set and performing R-Mosaic data augmentation processing on the training samples in the traffic image training set, includes:

[0059] In step S201, a traffic image training set is obtained, wherein each training sample in the traffic image training set includes a reference box.

[0060] In this embodiment of the invention, the traffic image training set includes several training samples, each of which includes a reference box, and the target object is initially defined by the reference box.

[0061] In step S202, the training samples are grouped into pairs and subjected to horizontal flipping, scaling, and color gamut changes.

[0062] In this embodiment of the invention, two training samples in the traffic image training set are divided into a group, and preprocessed in groups. The two training samples are horizontally flipped, scaled, and their color gamut is changed to generate an enhanced image.

[0063] In step S203, an image frame with a size twice that of the training sample is constructed, one training sample is placed in the upper left corner of the image frame, and another training sample is placed in the upper right corner of the image frame.

[0064] In step S204, a portion of the training sample image is cropped and pasted onto the image frame, and then stretched vertically to fill the image frame.

[0065] Here, for two training samples in a set, after horizontal flipping, scaling, and color gamut changes, one training sample is placed in the upper left corner of an image frame twice the size (width and height) of the original training sample, and the other training sample is placed in the upper right corner of the same frame. Then, a portion of the training sample is cropped and pasted onto the image frame, vertically stretched to fill the entire image. For ease of understanding, Figure 3 This is a schematic diagram of R-Mosaic data augmentation processing provided in an embodiment of the present invention, wherein... Figure 3 (a) is a schematic diagram corresponding to step S201. Figure 3 (b) is a schematic diagram corresponding to steps S202 and S203. Figure 3 (c) is a schematic diagram corresponding to steps S204 and S205.

[0066] In step S205, the size and position of the image frame are readjusted, and the integrated image frame is translated, scaled, and cropped to obtain an enhanced image of the same size as the training sample.

[0067] Finally, the size and position of the image frame are readjusted. The integrated image frame is translated, scaled, cropped, and resized to match the size of the training samples, resulting in the enhanced image. This enhanced image is used to train the H-FPN deep convolutional network model.

[0068] This invention uses the R-MSAic data augmentation method to enhance the features of sample images, greatly enriching the background of detected objects. This is used to train the H-FPN deep convolutional network model, which helps improve the accuracy of information extraction by the H-FPN deep convolutional network model, obtains more accurate traffic flow feature parameters, and thus further strengthens the robustness of road traffic state detection.

[0069] In step S102, a pre-defined H-FPN deep convolutional network is trained using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image.

[0070] Here, since most cameras collecting traffic image training sets are in fixed positions and at fixed angles, targets are constantly moving closer to or further away from the camera when shooting roads. For the same target, the farther away it is from the camera, the smaller it appears in the image; the closer it is to the camera, the larger it appears. However, the shape of the same target remains basically the same regardless of its distance from the camera. To address this characteristic, in this embodiment of the invention, after extracting the feature map, for feature maps with a small receptive field, the main predicted target is a small target. Since targets in the lower part of the image tend to be large targets, small-scale reference boxes are not used in the lower half of the feature map; that is, small targets are not predicted in the lower half. Similarly, for feature maps with a large receptive field, the main predicted target is a large target. Since targets in the upper half of the image tend to be small targets, large-scale reference boxes are not used in the upper half of the feature map; that is, large targets are not predicted in the upper half. This reduces computation and improves detection speed.

[0071] Optionally, the H-FPN refers to a semi-structural feature pyramid. The H-FPN deep convolutional network includes three prediction layers: a first prediction layer, a second prediction layer, and a third prediction layer. Each prediction layer receives a corresponding reference box anchor as input. In this embodiment, the set of reference box anchors with the smallest scale is input to the first prediction layer, the set of reference box anchors with the largest scale is input to the third prediction layer, and the remaining set of reference box anchors is input to the second prediction layer.

[0072] The first prediction layer is used to obtain a feature map with a receptive field of 76*76 through upsampling. It then predicts the target in the upper receptive field using a preset first reference box, outputting the prediction information for the upper half of the feature map. In this embodiment, the 76*76 feature map is divided into upper and lower layers, with the upper receptive field being 76*38. The first reference box is the smallest set of reference boxes (anchors). Only the upper feature map is used as input; that is, the first reference box only acts on the upper feature map with a receptive field of 76*38, predicting and outputting the target in the upper half, discarding the lower feature map, and finally outputting the prediction information for the upper half of the feature map.

[0073] The second prediction layer is used to obtain a feature map with a receptive field of 38*38 through upsampling, and to predict the target in the entire receptive field using a preset second reference box, outputting the prediction information of the entire feature map. Here, the input and output of the second prediction layer remain unchanged, with a receptive field of 38*38, and the corresponding reference box anchor is used as input for prediction to obtain the prediction information of the entire feature map.

[0074] The third prediction layer is used to obtain a feature map with a receptive field of 19*19 through upsampling. It then predicts targets in the lower half of the receptive field using a preset third reference box, outputting the prediction information for the lower half of the feature map. Here, the receptive field is 19*19, which is the largest receptive field, allowing for larger target detection. Similarly, the 19*19 feature map is divided into upper and lower layers, with the lower half having a receptive field of 19*10. The third reference box is the largest set of reference boxes (anchors), and it is only used as input to the lower half of the feature map. That is, the third reference box only acts on the lower half of the feature map with a receptive field of 19*10, predicting and outputting targets in the lower half, discarding the upper half of the feature map, and finally outputting the prediction information for the lower half of the feature map.

[0075] For ease of understanding, Figure 4 This is a schematic diagram of H-FPN network operation provided in an embodiment of the present invention. Here, the embodiment of the present invention modifies the H-FPN network structure, reducing the number of parameters and computational cost of the H-FPN deep convolutional network, thereby further improving the accuracy of information extraction.

[0076] The H-FPN deep convolutional network further includes a detection and tracking module, which is used to obtain traffic flow feature parameters based on the prediction information output by the first prediction layer, the second prediction layer, and the third prediction layer. Here, the feature parameters corresponding to the traffic image include, but are not limited to, traffic flow speed V, traffic flow density K, and traffic volume N.

[0077] Wherein, the traffic volume N is the number of vehicles passing through the designated road segment per unit time (1 hour); the formula for calculating the traffic flow velocity V is:

[0078]

[0079] Where v i The speed of the i-th vehicle during the observation period is represented by ; n represents the number of vehicles observed per unit time.

[0080] The formula for calculating traffic flow density K is:

[0081]

[0082] Where k represents traffic flow density; N represents traffic volume; and L represents the length of the observed road segment.

[0083] The average traffic density of roads during a certain period is:

[0084]

[0085] Where K represents the average traffic density of a certain road segment within a certain time period, and the unit is vehicles / km; i represents the number of vehicles in the i-th captured frame; n represents the number of captured frames within the time period; L represents the length of the road segment.

[0086] Optionally, in the training process of the H-FPN deep convolutional network, the classification loss function of the H-FPN deep convolutional network consists of a regression loss function and a classification loss function, and its calculation formula is as follows:

[0087] Loss = reg_loss + pb_loss

[0088] Where reg_loss represents regression loss and pb_loss represents classification loss;

[0089] The formula for calculating regression loss reg_loss is:

[0090]

[0091] Among them, y i Indicates the actual observed value; Indicates the predicted value;

[0092] The formula for calculating the classification loss pb_loss is:

[0093]

[0094] Where pre_label represents the predicted state; rel_label represents the actual state.

[0095] This invention incorporates classification loss into the prediction stage of the H-FPN deep convolutional network, improving its loss function and enhancing prediction accuracy and discrimination precision. Compared to existing technologies that only use regression loss, the improved loss function enhances the fitting ability of the H-FPN deep convolutional network and improves the accuracy of road traffic state detection.

[0096] After training the H-FPN deep convolutional network, the network is tested. This embodiment of the invention obtains traffic images and their corresponding real traffic flow feature parameters, namely traffic volume, traffic flow speed, and traffic flow density. The traffic images are input into the H-FPN deep convolutional network to obtain predicted values ​​of the traffic flow feature parameters output by the network. The predicted values ​​of the traffic flow feature parameters are then compared with the real values ​​to calculate regression and classification errors, and the parameters are iteratively updated in reverse to the H-FPN deep convolutional network.

[0097] In step S103, the traffic image to be processed is acquired, and the feature parameters corresponding to the traffic image are extracted through the trained H-FPN deep convolutional network.

[0098] The traffic image to be processed is obtained and provided as input parameters to the H-FPN deep convolutional network trained in step S102 above. The corresponding traffic flow feature parameters, namely traffic flow speed, traffic flow density and traffic volume, are extracted through the H-FPN deep convolutional network.

[0099] In step S104, a traffic state prediction value is generated and output based on the characteristic parameters of the traffic flow using a preset discrimination model.

[0100] Here, the preset discrimination model includes a matrix composed of discrimination thresholds for traffic flow speed, traffic flow density, and traffic volume;

[0101]

[0102] Where P represents the discrimination matrix, i.e., the matrix composed of discrimination thresholds, i≤3, j≤3; P11 represents the minimum speed of traffic flow in a smooth state; P12 represents the minimum speed of traffic flow in a congested state; P13 represents the maximum speed of traffic flow in a congested state; P21 represents the minimum density of traffic flow in a smooth state; P22 represents the minimum density of traffic flow in a congested state; P23 represents the maximum density of traffic flow in a congested state; P31 represents the maximum quantity of traffic volume in a smooth state; P32 represents the maximum quantity of traffic volume in a congested state; P33 represents the minimum quantity of traffic volume in a congested state.

[0103] Alternatively, as a preferred example of the present invention, such as Figure 5 As shown, step S104, which involves generating and outputting traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model, includes:

[0104] In step S401, the traffic flow speed, traffic flow density, and traffic volume are respectively compared with the discrimination thresholds in the discrimination matrix through the discrimination model to obtain the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume.

[0105] In step S402, if the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume is not the same, the traffic state prediction value is output as congestion.

[0106] In step S403, if the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume is not completely the same, the traffic state information with the most occurrences is selected as the traffic state prediction value.

[0107] Here, by comparing with the discrimination threshold, three traffic state information can be obtained, namely unobstructed, crowded, and congested. In the embodiment of the present invention, the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume is comprehensively considered to convert the traffic state prediction value. When the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume does not completely satisfy the same state, the principle of the minority obeying the majority is used to determine the traffic state prediction value; if all three are not in the same state, the traffic state prediction value is congestion.

[0108] Optionally, for the sake of understanding, assume there is the following discrimination model:

[0109]

[0110] If the characteristic parameters of the traffic flow output after a traffic image passes through the H-FPN deep convolutional network are: traffic flow speed V > P11, traffic flow density K < P21, and traffic volume N < P31.

[0111] It can be seen that after comparing with the discrimination thresholds in the discrimination matrix, the traffic state information corresponding to the traffic flow speed is unobstructed, the traffic state information corresponding to the traffic flow density is unobstructed, and the traffic state information corresponding to the traffic volume is crowded. The three are not completely the same, and the traffic state information with the most occurrences is selected as the traffic state prediction value, that is, unobstructed is used as the traffic state prediction value.

[0112] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0113] In one embodiment, the present invention further provides a road traffic state detection device based on deep learning. The road traffic state detection device based on deep learning corresponds one-to-one with the road traffic state detection method based on deep learning in the above embodiment. As Figure 6As shown, the deep learning-based road traffic state detection device includes a preprocessing module 51, a training module 52, an extraction module 53, and a prediction module 54. Detailed descriptions of each functional module are as follows:

[0114] Preprocessing module 51 is used to acquire a traffic image training set and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set;

[0115] Training module 52 is used to train a preset H-FPN deep convolutional network using training samples augmented with R-Mosaic data, wherein the H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image;

[0116] Extraction module 53 is used to acquire the traffic image to be processed and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network;

[0117] The prediction module 54 is used to generate and output traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model.

[0118] Optionally, the preprocessing module 51 includes:

[0119] An acquisition unit is used to acquire a traffic image training set, wherein each training sample in the traffic image training set includes a reference box;

[0120] The first preprocessing unit is used to perform horizontal flipping, scaling, and color gamut changes on the training samples, taking each pair of training samples as a group.

[0121] The building unit is used to build an image frame that is twice the size of the training sample. One training sample is placed in the upper left corner of the image frame, and another training sample is placed in the upper right corner of the image frame.

[0122] The cropping unit is used to crop a portion of the training sample image, paste it onto the image frame, and stretch it vertically to fill the image frame;

[0123] The second preprocessing unit is used to readjust the size and position of the image frame, and to translate, scale, and crop the integrated image frame to obtain an enhanced image of the same size as the training sample.

[0124] Optionally, the H-FPN deep convolutional network includes three prediction layers, namely a first prediction layer, a second prediction layer, and a third prediction layer;

[0125] The first prediction layer is used to obtain a feature map with a receptive field of 76*76 by upsampling, and to predict the target in the upper receptive field by a preset first reference box, and output the prediction information of the upper half of the feature map.

[0126] The second prediction layer is used to obtain a feature map with a receptive field of 38*38 by upsampling, and to predict the target in the entire receptive field by a preset second reference box, and output the prediction information of the entire feature map.

[0127] The third prediction layer is used to obtain a feature map with a receptive field of 19*19 through upsampling, predict the target in the lower half of the receptive field through a preset third reference box, and output the prediction information of the lower half of the feature map.

[0128] The H-FPN deep convolutional network also includes a detection and tracking module, which is used to obtain the feature parameters of traffic flow based on the prediction information output by the first prediction layer, the second prediction layer and the third prediction layer.

[0129] Optionally, the characteristic parameters include traffic flow speed, traffic flow density, and traffic volume.

[0130] Optionally, the classification loss function of the H-FPN deep convolutional network consists of a regression loss function and a classification loss function, and its calculation formula is as follows:

[0131] Loss = reg_loss + pb_loss

[0132] Where reg_loss represents regression loss and pb_loss represents classification loss;

[0133] The formula for calculating regression loss reg_loss is:

[0134]

[0135] Among them, y i Indicates the actual observed value; Indicates the predicted value;

[0136] The formula for calculating the classification loss pb_loss is:

[0137]

[0138] Where pre_label represents the predicted state; rel_label represents the actual state.

[0139] Optionally, the preset discrimination model includes a matrix composed of discrimination thresholds for traffic flow speed, traffic flow density, and traffic volume;

[0140]

[0141] Where P represents the discrimination matrix, i.e., the matrix composed of discrimination thresholds, i≤3, j≤3; P11 represents the minimum speed of traffic flow in a smooth state; P12 represents the minimum speed of traffic flow in a congested state; P13 represents the maximum speed of traffic flow in a congested state; P21 represents the minimum density of traffic flow in a smooth state; P22 represents the minimum density of traffic flow in a congested state; P23 represents the maximum density of traffic flow in a congested state; P31 represents the maximum quantity of traffic volume in a smooth state; P32 represents the maximum quantity of traffic volume in a congested state; P33 represents the minimum quantity of traffic volume in a congested state.

[0142] Optionally, the prediction module 54 includes:

[0143] The comparison unit is used to compare the traffic flow speed, traffic flow density, and traffic volume with the discrimination thresholds in the discrimination matrix through the discrimination model, so as to obtain the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume.

[0144] The first prediction unit is used to output a traffic state prediction value of congestion when the traffic state information corresponding to the traffic flow speed, traffic flow density and traffic volume are all different.

[0145] The second prediction unit is used to select the traffic state information that appears most frequently as the traffic state prediction value when the traffic flow speed, traffic flow density and traffic volume are not completely the same.

[0146] Specific limitations regarding the deep learning-based road traffic condition detection device can be found in the limitations of the deep learning-based road traffic condition detection method described above, and will not be repeated here. Each module in the aforementioned deep learning-based road traffic condition detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in memory as software, so that the processor can call and execute the corresponding operations of each module.

[0147] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a deep learning-based road traffic state detection method.

[0148] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0149] Obtain a traffic image training set, and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set;

[0150] The H-FPN deep convolutional network is trained using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network uses three prediction layers to extract traffic flow feature parameters from the input image.

[0151] Acquire a traffic image to be processed, and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network;

[0152] Traffic state prediction values ​​are generated and output based on the characteristic parameters of the traffic flow using a preset discrimination model.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0154] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0155] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A road traffic state detection method based on deep learning, characterized in that, include: Obtain a traffic image training set, and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set; The R-Mosaic data augmentation process includes: acquiring a traffic image training set, where each training sample in the traffic image training set includes a reference box; grouping the training samples in pairs and performing horizontal flipping, scaling, and color gamut changes on the training samples; constructing an image frame twice the size of the training samples, placing one training sample in the upper left corner of the image frame and another training sample in the upper right corner of the image frame; cropping a portion of the training sample image and pasting it onto the image frame, then vertically stretching it to fill the image frame; readjusting the size and position of the image frame, and then translating, scaling, and cropping the integrated image frame to obtain an augmented image of the same size as the training samples; A pre-defined H-FPN deep convolutional network is trained using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network comprises three prediction layers: a first prediction layer, a second prediction layer, and a third prediction layer. The first prediction layer upsamples a feature map with a receptive field of 76*76, predicts targets in the upper half of the receptive field using a pre-defined first reference box, and outputs the prediction information for the upper half of the feature map. The second prediction layer upsamples a feature map with a receptive field of 38*38, predicts targets in the entire receptive field using a pre-defined second reference box, and outputs the prediction information for the entire layer of the feature map. The third prediction layer upsamples a feature map with a receptive field of 19*19, predicts targets in the lower half of the receptive field using a pre-defined third reference box, and outputs the prediction information for the lower half of the feature map. The H-FPN deep convolutional network also includes a detection and tracking module, which obtains traffic flow feature parameters based on the prediction information output by the first, second, and third prediction layers. Acquire the traffic image to be processed, and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network; Traffic state prediction values ​​are generated and output based on the characteristic parameters of the traffic flow using a preset discrimination model.

2. The road traffic state detection method based on deep learning as described in claim 1, characterized in that, The characteristic parameters include traffic flow speed, traffic flow density, and traffic volume.

3. The deep learning-based road traffic state detection method as described in claim 1 or 2, characterized in that, The classification loss function of the H-FPN deep convolutional network consists of a regression loss function and a classification loss function, and its calculation formula is as follows: Loss = reg_loss + pb_loss Where reg_loss represents regression loss and pb_loss represents classification loss; The formula for calculating regression loss reg_loss is: Among them, y i Indicates the actual observed value; Indicates the predicted value; The formula for calculating the classification loss pb_loss is: Where pre_label represents the predicted state; rel_label represents the actual state.

4. The road traffic state detection method based on deep learning as described in claim 1, characterized in that, The preset discrimination model includes a matrix composed of discrimination thresholds for traffic flow speed, traffic flow density, and traffic volume; Where P represents the discrimination matrix, i.e., the matrix composed of discrimination thresholds, i≤3, j≤3; P11 represents the minimum speed of traffic flow in a smooth state; P12 represents the minimum speed of traffic flow in a congested state; P13 represents the maximum speed of traffic flow in a congested state; P21 represents the minimum density of traffic flow in a smooth state; P22 represents the minimum density of traffic flow in a congested state; P23 represents the maximum density of traffic flow in a congested state; P31 represents the maximum quantity of traffic volume in a smooth state; P32 represents the maximum quantity of traffic volume in a congested state; P33 represents the minimum quantity of traffic volume in a congested state.

5. The road traffic state detection method based on deep learning as described in claim 4, characterized in that, The step of generating and outputting traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model includes: The traffic flow speed, traffic flow density, and traffic volume are compared with the discrimination thresholds in the discrimination matrix using the discrimination model to obtain the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume. If the traffic state information corresponding to the traffic flow speed, traffic flow density and traffic volume are all different, the output traffic state prediction value is congestion. If the traffic state information corresponding to the traffic flow speed, traffic flow density, and traffic volume are not completely the same, the traffic state information that appears most frequently is selected as the traffic state prediction value.

6. A road traffic condition detection device based on deep learning, characterized in that, The device includes: A preprocessing module is used to acquire a traffic image training set and perform R-Mosaic data augmentation processing on the training samples in the traffic image training set. The R-Mosaic data augmentation processing includes: acquiring the traffic image training set, wherein each training sample in the traffic image training set includes a reference box; taking two training samples as a group, performing horizontal flipping, scaling, and color gamut changes on the training samples; constructing an image frame with a size twice that of the training sample, placing one training sample in the upper left corner of the image frame and another training sample in the upper right corner of the image frame; cropping a portion of the image of the training sample and pasting it into the image frame, and vertically stretching it to fill the image frame; readjusting the size and position of the image frame, and translating, scaling, and cropping the integrated image frame to obtain an augmented image of the same size as the training sample. The training module is used to train a pre-defined H-FPN deep convolutional network using training samples augmented with R-Mosaic data. The H-FPN deep convolutional network includes three prediction layers: a first prediction layer, a second prediction layer, and a third prediction layer. The first prediction layer upsamples a feature map with a receptive field of 76*76, predicts targets in the upper half of the receptive field using a pre-defined first reference box, and outputs the prediction information for the upper half of the feature map. The second prediction layer upsamples a feature map with a receptive field of 38*38, predicts targets in the entire receptive field using a pre-defined second reference box, and outputs the prediction information for the entire layer of the feature map. The third prediction layer upsamples a feature map with a receptive field of 19*19, predicts targets in the lower half of the receptive field using a pre-defined third reference box, and outputs the prediction information for the lower half of the feature map. The H-FPN deep convolutional network also includes a detection and tracking module, which obtains traffic flow feature parameters based on the prediction information output by the first, second, and third prediction layers. The extraction module is used to acquire the traffic image to be processed and extract the feature parameters corresponding to the traffic image through a trained H-FPN deep convolutional network. The prediction module is used to generate and output traffic state prediction values ​​based on the characteristic parameters of the traffic flow using a preset discrimination model.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the deep learning-based road traffic state detection method as described in any one of claims 1 to 5.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the deep learning-based road traffic state detection method as described in any one of claims 1 to 5.

Images