A device for extracting static information of traffic targets

Abstract

The present invention relates to a device for extracting static information of traffic targets, comprising: an image acquisition module for acquiring video images of traffic targets in a traffic scene; a static information extraction module for inputting key frames of the video images into a trained detection network to obtain multiple static information of traffic targets; wherein the detection network includes: a feature extraction unit for extracting features from the key frames to obtain a number of ROI regions; a combination unit for combining the number of ROI regions to obtain a combined feature; a multi-branch prediction unit for analyzing and processing the combined feature to obtain multiple static information of traffic targets; and a storage module for storing the static information. The present invention can obtain rich static information of traffic targets and complete the task of describing traffic targets from various levels of vehicle attributes.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic target information extraction, and particularly to a device for extracting static information of traffic targets. Background Art

[0002] The key to describing the static attributes of vehicles is to achieve target detection, and obtain the position and category information of the target from a two-dimensional image through the target detection result, so as to provide a data basis for the establishment of a feature model. Existing target detection methods can be mainly divided into two categories: traditional methods and deep learning methods. Traditional methods mainly rely on manually designed features, with simple algorithms and small computational amounts. However, in the face of factors such as camera jitter, target occlusion, light changes, and weather changes, they generally have disadvantages such as incomplete feature extraction, high false detection rate, and poor robustness. With the use of deep learning technology, there have been significant breakthroughs in target detection algorithms. Deep learning methods can be divided into two types of frameworks: two-stage and one-stage according to the algorithm process. Among them, the representative algorithm of the former is Faster R-CNN, which proposes an RPN network to replace the original sliding window mechanism, and combines anchors for rough positioning of the target area, and then refines the target position through a bounding box regression classification network, greatly improving the algorithm performance. The representative algorithm of the latter is SSD, which combines the regression idea with the anchors of Faster R-CNN, and uses the multi-scale feature maps at each position after the original image is convolved for regression and classification, improving the target area positioning accuracy while maintaining the detection speed, and having good adaptability to small targets. In order to avoid pre-setting anchors, CenterNet regards the detection of bounding boxes as key point detection, uses the center point, the upper left point, and the lower right point as key points, and makes full use of the internal features of the target to improve the algorithm accuracy. Almost all current deep learning target detection algorithms are based on these two types of frameworks.

[0003] For the multi-task recognition problem, since the tasks to be completed include: license plate extraction and recognition, key frame capture, target detection and tracking, etc. Existing methods mostly use deep learning networks, but such networks can only solve a few tasks and cannot solve multiple tasks through one network. Therefore, how to design a multi-task deep network to directly solve the tasks of license plate extraction and recognition, key frame capture, and target detection and tracking is an urgent problem to be solved. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a device for extracting static information of traffic targets, which can obtain rich static information of traffic targets and complete the task of describing traffic targets from all levels of vehicle attributes.

[0005] The technical solution adopted by the present invention to solve its technical problems is to provide a device for extracting static information of traffic targets, including:

[0006] An image acquisition module for acquiring video images of traffic targets in a traffic scene;

[0007] A static information extraction module for inputting key frames of the video images into a trained detection network to obtain multiple static information of traffic targets; wherein, the detection network includes: a feature extraction unit for extracting features from the key frames to obtain a number of ROI regions; a combination unit for combining the number of ROI regions to obtain a combined feature; a multi-branch prediction unit for analyzing and processing the combined feature to obtain multiple static information of the traffic targets;

[0008] A storage module for storing the static information.

[0009] The traffic scene target feature set used by the detection network during training includes an actual scene target feature set and a simulated scene target feature set. The actual scene target feature set is traffic scene videos collected from actual surveillance videos, and the simulated scene target feature set is virtual traffic scene videos generated by placing virtual cameras in a virtual traffic scene and using the virtual cameras to record.

[0010] The device for extracting static information of traffic targets further includes a key frame extraction module, and the key frame extraction module is used to extract valuable video frames from the video images as key frames.

[0011] The key frame extraction module includes a convolutional network layer, a fully connected layer, and a classification layer connected in sequence. The input of the convolutional network layer is the video frame to be screened and the previous frame image of the video frame, and the output is a one-dimensional feature vector; the fully connected layer is used to combine the one-dimensional feature vector; the classification layer is used to analyze the combined one-dimensional feature vector to complete the classification of the video frame to be screened.

[0012] The feature extraction unit performs convolutional processing on the input key frames to generate a set of convolutional feature maps, and then obtains a set of proposals based on the convolutional feature maps by using a region proposal network, and obtains a number of ROI regions according to the sizes of the proposals; wherein, the region proposal network uses deconvolution and bilinear kernels for pooling.

[0013] The multi-branch prediction unit includes a fully connected layer, and a three-dimensional prediction branch and a multi-level prediction branch are arranged after the fully connected layer. The three-dimensional prediction branch is used to detect the key points of the vehicle, and at the same time, a pyramid mechanism is added to adapt to vehicle targets of multiple scales; the multi-level prediction branch is used to detect the target license plate, vehicle color, and number of vehicle axles to obtain different vehicle static features.

[0014] The loss function of the detection network is as follows: where loss total is the overall loss function, L1, L2, L3, and L4 are four loss parts, P is the standard softmax loss, C is the proposal in this batch, C = 1 when the proposal is a positive example, R is the smooth L1 loss, λ cls , λ reg are the regularization constants for the category and regression, d* and d are the two-dimensional target rectangle box regression vectors predicted by the detection network and the true regression vectors of the two-dimensional target box respectively; λ 3d is the regularization constant for the three-dimensional prediction branch, v* and v are the three-dimensional regression vectors predicted by the detection network and the true three-dimensional regression vectors respectively; λ sim is the regularization parameter for the template similarity degree, t* and t are the template vectors predicted by the detection network and the true template vectors respectively; λ part is the regularization parameter for the part detection, s* and s are the target part vectors predicted by the detection network and the positions of the true target part vectors respectively.

[0015] Before storing the static information, the storage module also encodes the static information.

[0016] Beneficial effects

[0017] Due to the adoption of the above technical solution, compared with the prior art, the present invention has the following advantages and positive effects: The present invention designs an integrated multi-task depth detection network, which can obtain rich static information of traffic targets and complete the task of describing traffic targets from various levels of vehicle attributes. In order to improve the calculation speed of the entire network, the present invention also pre-judges whether a video frame is a key frame through a key frame screening network. The present invention also encodes the extracted target information, enabling the extracted target information to be used in various environments and making the implementation method more flexible. Description of the drawings

[0018] Figure 1 is a schematic structural diagram of an embodiment of the present invention;

[0019] Figure 2 is a schematic structural diagram of the key frame extraction module in an embodiment of the present invention;

[0020] Figure 3 is a schematic structural diagram of the detection network in an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of the encoding of the storage module in an embodiment of the present invention. Detailed implementation manners

[0022] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0023] An embodiment of the present invention relates to a device for extracting static information of traffic targets, such as Figure 1 shown, including: an image acquisition module for acquiring video images of traffic targets in a traffic scene; a static information extraction module for inputting key frames of the video images into a trained detection network to obtain multiple static information of traffic targets; and a storage module for storing the static information.

[0024] In this embodiment, the device for extracting static information of traffic targets further includes a key frame extraction module, and the key frame extraction module is used to extract valuable video frames from the video images as key frames. As Figure 2 shown, the key frame extraction module includes a convolutional network layer, a fully connected layer, and a classification layer connected in sequence. The input of the convolutional network layer is the video frame to be screened and the previous frame image of the video frame, and the output is a one-dimensional feature vector; the fully connected layer is used to combine the one-dimensional feature vectors; the classification layer is used to analyze the combined one-dimensional feature vectors to complete the classification of the video frame to be screened.

[0025] In a traffic video sequence, there are often a large number of redundant video frames. If each video frame is analyzed, it will seriously affect the calculation speed. Therefore, for the original video sequence, it is first sent to the key frame screening network (i.e., the key frame extraction module) to extract valuable video frames from the original video sequence for subsequent analysis, thereby improving the processing efficiency. First, a large-scale video frame screening image library needs to be constructed. This image library covers two types of images through manual annotation: useless frames (label 0) and key frames (label 1). Useless frames are frames with fewer traffic targets or no traffic targets in the scene, and frames with a relatively low proportion of the road surface area in the scene. Key frames are frames with a large number of traffic targets and a relatively high proportion of the road surface area in the image. Secondly, the design of the key frame screening network is as follows. The frame image to be screened and its previous frame image (as a comparison reference) are sent into the convolutional network to obtain a one-dimensional vector, which includes the features of the two images. Then, the fully connected layer is used to combine the image features, and softmax is used to complete the classification of the image frame. Finally, the classification result is output, that is, whether the image is a key frame or not.

[0026] The detection network in this embodiment includes: a feature extraction unit for extracting features from the key frame to obtain a number of ROI regions; a combination unit for combining the number of ROI regions to obtain a combined feature; and a multi-branch prediction unit for analyzing and processing the combined feature to obtain multiple static information of the traffic target.

[0027] Among them, the feature extraction unit performs convolution processing on the input key frame to generate a set of convolution feature maps, and then obtains a set of proposals based on the convolution feature maps by using a region proposal network, and obtains a number of ROI regions according to the sizes of the proposals; among them, the region proposal network uses deconvolution and bilinear kernels for pooling.

[0028] The multi-branch prediction unit includes a fully connected layer, and a three-dimensional prediction branch and a multi-level prediction branch are arranged after the fully connected layer. The three-dimensional prediction branch is used to detect the key points of the vehicle, and at the same time, a pyramid mechanism is added to adapt to vehicle targets of multiple scales; the multi-level prediction branch is used to detect the target license plate, vehicle color, and number of vehicle axles to obtain different static features of the vehicle.

[0029] For traffic targets, it is necessary to perceive their two-dimensional, three-dimensional and other static attribute information in the traffic scene, so as to provide rich and complete target descriptions in a large-scale traffic scene with multi-camera linkage. The two-dimensional and three-dimensional static information of traffic targets includes: a series of information such as target position, target category, target size, number of vehicle axles, license plate, vehicle color, etc. According to the key frame screening network, a key frame is obtained, and the key frame is input into the detection network. Through network operations, the above target static information is output. The design scheme of this detection network is as Figure 3As shown below. First, input the traffic scene video frames, which pass through the convolutional layer to generate a set of convolutional feature maps. Based on these feature maps, a set of proposals are obtained by using the Region Proposal Network (RPN). The RPN predicts the bounding boxes that may contain objects. Deconvolution and pooling with bilinear kernels are used to expand the features of small proposal regions, thus avoiding the problem of insensitive small object detection caused by representing small traffic targets with repeated values. This pooling operation is applied to multiple levels of the convolutional neural network, and the pooling features of these different convolutional layers are concatenated together to fuse the low-level detailed information and high-level semantic information. Then, the network is divided into multiple branches according to the size of the proposal regions, which reduces the training burden of traffic targets of different scales and sizes, thereby improving the detection accuracy of large and small objects. Based on the above, after combining the features of the ROI regions, for targets of different scales, different branches are used for prediction. This detection network is divided into three prediction branches, corresponding to small targets, medium-sized targets, and large targets respectively. A fully connected layer is added to the three prediction branches, and then a three-dimensional prediction branch is added to the multi-level prediction branch, which is responsible for detecting the key points (i.e., the positions of the vertices) of the vehicle. At the same time, a pyramid mechanism is added to adapt to vehicle targets of multiple scales; a multi-level prediction branch for detecting the license plate, vehicle color, and number of vehicle axles of the detection target is also added to obtain different static features of the vehicle. Finally, all the prediction results from multiple branches are fused into the final detection result. The loss function of this detection network in this embodiment is as follows:

[0030]

[0031] Among them, the overall loss function loss total consists of four parts, namely L1, L2, L3, and L4. P is the standard softmax loss, C is the proposal in this batch, C = 1 when the proposal is a positive example, and R is the smooth L1 loss. In L1, λ cls , λ reg are the regularization constants for classification and regression, d * , d are the two-dimensional target rectangle box regression vectors predicted by the integrated network and the true regression vectors of the two-dimensional target box respectively; in L2, λ 3d is the regularization constant for the three-dimensional detection branch, v * , v are the three-dimensional regression vectors predicted by the integrated network and the true three-dimensional regression vectors respectively; in L3, λ sim is the regularization parameter for the template similarity degree, t * , t are the template vectors predicted by the integrated network and the true template vectors respectively; in L4, λ part is the regularization parameter for part detection, s *, s are the target component vectors predicted by the integrated network and the positions of the true target component vectors respectively. Through the propagation calculation of the loss function, the static attributes of the target in each traffic scene, such as the position, category, size, number of vehicle axles, license plate, vehicle color, etc., are obtained, and the design of the integrated multi-level description network for two-dimensional and three-dimensional detection of traffic targets is completed. This network can obtain rich static information of traffic targets and complete the task of describing traffic targets at all levels of vehicle attributes.

[0032] It is worth mentioning that in order to describe the static characteristics of the target, a large number of target samples need to be collected. For a large-scale traffic scene with multi-camera linkage, the number and types of traffic targets are relatively rich. Although the currently publicly available large-scale target datasets, such as the COCO, PascalVoc datasets, and BIT vehicle dataset, which include the features of many common items, are not sufficient for traffic scenes. Since this embodiment needs to consider a large-scale traffic scene, the camera shooting range is wide, and the targets on the road will undergo drastic deformation when driving towards and away from the camera. At the same time, in order to meet the diversity of traffic scenes and consider the traffic target situation in complex traffic environments, when training the detection network, it is necessary to construct multiple datasets oriented to traffic scenes. The traffic scene target feature set constructed in this embodiment is divided into two parts: the actual scene target feature set and the simulated scene target feature set. In the actual traffic scene target feature set, more than ten thousand high-definition resolution images of actual road and bridge tunnel monitoring cameras at home and abroad are collected, covering traffic target features with various scale changes and shape changes, and considering the problem of insufficient light caused by bad weather conditions such as cloudy days and rainy days. In the simulated traffic scene target feature set, the real scenes simulated by the unmanned and autonomous driving platform CARLA are used. By placing virtual cameras in the virtual traffic scene and using the virtual cameras to record and generate virtual traffic scene videos. Through the videos, many scene description information can be obtained: the position and angle of the camera, the internal and external parameter matrices of the camera, the weather in the scene, the congestion degree of vehicles in the scene, etc.; many traffic target description information can also be obtained: the position, speed, type, number, etc. of traffic targets. This embodiment collects more than 300 traffic scenes and more than 4,000 traffic videos including more than 20 million video frames in the simulated traffic scene target feature set. The above actual scene and simulated scene target feature sets complement each other's advantages, taking into account both the real state of traffic target operation and enriching the diversity of traffic scenes, providing complete target and scene feature information for the detection network.

[0033] Before storing the static information, the storage module in this embodiment also encodes the static information. Specifically, based on the constructed detection network, since the static information such as the target position, target category, target size, number of vehicle axles, license plate, and vehicle color of the traffic target has been obtained, a feature model belonging to the target is constructed based on this static information. The unified information encoding format of the obtained target static attributes is stored in binary format. The specific encoding format is as Figure 4 shown. As can be seen in Figure 4 , the traffic target to be modeled currently is a red vehicle, and its image position, category, size, number of axles, license plate, and color have all been obtained using the integrated multi-level description network for two-dimensional and three-dimensional detection of traffic targets. After converting and storing this information using the encoder, the modeling of the feature model of this traffic target is completed. When the specific traffic information of the target is required, the specific information of the target can be restored through the decoder. The traffic target feature model can uniquely describe the static attributes of the target, so the traffic target feature model can be used for the uniqueness judgment of the target in the multi-camera linkage traffic scene. At the same time, the target information recorded by the traffic target feature model can be used in various environments through encoding and decoding.

[0034] It is not difficult to find that the present invention designs an integrated multi-task deep detection network, which can obtain rich static information of traffic targets and complete the task of describing traffic targets from various levels of vehicle attributes. To improve the computing speed of the entire network, the present invention also pre-judges whether a video frame is a key frame through a key frame screening network. The present invention also encodes the extracted target information, enabling the extracted target information to be used in various environments and making the implementation method more flexible.

Claims

1. A traffic target static information extraction device, characterized in that, Including: An image acquisition module, configured to acquire video images of traffic targets in a traffic scene; A static information extraction module, configured to input key frames of the video images into a trained detection network to obtain multiple static information of traffic targets; wherein, the detection network includes: a feature extraction unit, configured to extract features from the key frames to obtain a plurality of ROI regions; a combination unit, configured to combine the plurality of ROI regions to obtain a combined feature; a multi-branch prediction unit, configured to analyze and process the combined feature to obtain multiple static information of the traffic targets; the multi-branch prediction unit includes a fully connected layer, and a three-dimensional prediction branch and a multi-level prediction branch are arranged after the fully connected layer, the three-dimensional prediction branch is configured to detect key points of a vehicle, and a pyramid mechanism is added to adapt to vehicle targets of multiple scales; the multi-level prediction branch is configured to detect a target license plate, vehicle color and vehicle axle number to obtain different vehicle static features; the loss function of the detection network is: Among them, loss total is the overall loss function, L1, L2, L3, L4 are four loss parts, P is the standard softmax loss, C is the proposal in the batch, C = 1 when the proposal is a positive example, R is the smooth L1 loss, λ cls , λ reg are the regularization constants for classification and regression, d* and d are the two-dimensional target rectangle box regression vectors predicted by the detection network and the true regression vectors of the two-dimensional target box respectively; λ 3d is the regularization constant for the three-dimensional prediction branch, v* and v are the three-dimensional regression vectors predicted by the detection network and the true three-dimensional regression vectors respectively; λ sim is the regularization parameter for the template similarity degree, t* and t are the template vectors predicted by the detection network and the true template vectors respectively; λ part is the regularization parameter for part detection, s* and s are the target part vectors predicted by the detection network and the positions of the true target part vectors respectively; A storage module, configured to store the static information.

2. The traffic target static information extraction device according to claim 1, wherein The traffic scene target feature set used by the detection network during training includes an actual scene target feature set and a simulated scene target feature set. The actual scene target feature set is traffic scene videos collected from actual surveillance videos, and the simulated scene target feature set is virtual traffic scene videos generated by placing virtual cameras in a virtual traffic scene and using the virtual cameras to record.

3. The traffic target static information extraction device according to claim 1, wherein It further includes a key frame extraction module, configured to extract valuable video frames from the video images as key frames.

4. The traffic target static information extraction device according to claim 3, wherein, The key frame extraction module includes a convolutional network layer, a fully connected layer and a classification layer connected in sequence. The input of the convolutional network layer is the video frame to be screened and the previous frame image of the video frame, and the output is a one-dimensional feature vector; the fully connected layer is configured to combine the one-dimensional feature vectors; the classification layer is configured to analyze the combined one-dimensional feature vectors to complete the classification of the video frames to be screened.

5. The traffic target static information extraction device according to claim 1, characterized in that, The feature extraction unit performs convolutional processing on the input key frames to generate a set of convolutional feature maps, and then obtains a set of proposals based on the convolutional feature maps by using a region proposal network, and obtains a plurality of ROI regions according to the sizes of the proposals; wherein, the region proposal network uses deconvolution and bilinear kernels for pooling.

6. The traffic target static information extraction device according to claim 1, characterized in that, The storage module also encodes the static information before storing the static information.

Images