Traffic signal lamp adaptive control method based on multi-view depth camera

Abstract

The invention discloses a traffic signal lamp adaptive control method based on a multi-view depth camera, and relates to the field of traffic control, and the method comprises the steps: obtaining real-time video streams of an intersection through a plurality of depth cameras, and the real-time video streams comprise a real-time color video stream and a real-time depth video stream; performing target tracking on the real-time depth video stream acquired by each depth camera by using a 3D-YOLO-based target detection model to obtain an average vehicle speed; performing three-dimensional reconstruction on the real-time color video streams and the real-time depth video streams collected by the plurality of depth cameras, and inputting the real-time color video streams and the real-time depth video streams to a target detection model for target detection to obtain vehicle queuing lengths and vehicle categories; and switching to a corresponding operation mode according to the average vehicle speed, the vehicle queuing length and the vehicle category, and executing a signal lamp timing scheme in the mode. According to the method, the operation mode of the current intersection is adaptively switched by acquiring more accurate lane information, and the accuracy and timeliness of a signal lamp adaptive control strategy are improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic control, and in particular to a traffic light adaptive control method based on a multi-view depth camera. Background Art

[0002] Traffic is an indispensable part of modern society. It plays an important role in people's lives and economic development. With the acceleration of urbanization, the urban population has grown and the number of vehicles has increased dramatically, and traffic congestion and other problems have become increasingly prominent. In addition, urban road resources are limited and cannot be expanded indefinitely. Traditional traffic signal control has great limitations. The fixed phase time and cycle cannot be adjusted according to the real-time traffic flow. In the case of large traffic flow, it is easy to waste green light time or red light time is too long, aggravating traffic congestion. Therefore, intelligent means are needed to optimize the utilization of existing resources.

[0003] Traffic light adaptive control is an intelligent traffic management method that aims to optimize traffic flow, reduce congestion, and improve traffic efficiency by adjusting the phase and timing of traffic signals in real time. In the existing traffic light control method based on induction coils, induction coils are installed on the road to detect the passage of vehicles, so as to adjust the phase and timing of traffic lights in real time and optimize traffic flow. However, in the above method and some existing detection methods, the induction coil may fail or be damaged, resulting in missing or inaccurate data, and the maintenance cost and difficulty are very high; when the method of counting the number of vehicles passing by is adopted, only the number of vehicles passing by can be obtained, and the specific type of vehicles cannot be distinguished, and the adaptive strategy cannot be better selected; the traditional neural network algorithm has poor scalability, narrow application scenarios, and cannot be integrated with other algorithms. Summary of the invention

[0004] In view of the above problems and technical needs, the inventors have proposed a traffic light adaptive control method based on a multi-view depth camera, which is dedicated to reconstructing intersections with images collected by a multi-view depth camera to improve the accuracy and timeliness of the traffic light adaptive control strategy. The technical solution of the present invention is as follows:

[0005] A traffic light adaptive control method based on a multi-view depth camera comprises the following steps:

[0006] Multiple depth cameras deployed at the intersections are used to obtain real-time video streams of the intersections. The multiple depth cameras shoot the intersections from different perspectives, and the field of view of the multiple depth cameras covers the entire lane. The real-time video streams obtained by each depth camera include real-time color video streams and real-time depth video streams.

[0007] Use the 3D-YOLO-based target detection model to track the real-time depth video stream collected by each depth camera to obtain the average vehicle speed;

[0008] The real-time color video streams and real-time depth video streams collected by multiple depth cameras are reconstructed in three dimensions and then input into the 3D-YOLO-based target detection model for target detection to obtain the vehicle queue length and vehicle category;

[0009] Switch to the corresponding operating mode according to the average vehicle speed, vehicle queue length and vehicle type, and execute the traffic light timing plan in this mode.

[0010] The further technical solution is to use the 3D-YOLO-based target detection model to track the real-time depth video stream collected by each depth camera to obtain the average vehicle speed, including:

[0011] Calculate the average grayscale value within a preset range in the real-time depth video stream collected by each depth camera, where the preset range includes video frames around the lane entrance and video frames around the stop line;

[0012] When the average grayscale value reaches the set grayscale threshold, it is considered that a vehicle has passed through or left the preset range. The vehicle is tracked using a 3D-YOLO-based target detection model, and the duration of the vehicle's passage from passing through to leaving the preset range is recorded to calculate the average vehicle speed.

[0013] Its further technical solution is that the target detection model based on 3D-YOLO includes:

[0014] The input layer is used to decompose the voxelized 3D point cloud data into bird's-eye views in different directions;

[0015] The encoder is used to perform weighted processing on the bird's-eye view images in different directions to obtain voxel feature maps, and use multiple 3D convolutional layers and pooling layers to gradually extract high-level features from the voxel feature maps;

[0016] The decoder is used to restore the feature maps of different levels to the original voxel resolution using the 3D upsampling layer, and then fuse the features of different levels to obtain a fused feature map;

[0017] The output layer is used to identify the target based on the fused feature map and predict the target category probability.

[0018] Its further technical solution is that the method further comprises:

[0019] Use a depth camera to collect 3D point cloud data containing various vehicle categories, and form training samples and verification samples after preprocessing and annotation;

[0020] Build a target detection model based on 3D-YOLO, and use training samples and verification samples to train and verify the model.

[0021] A further technical solution is that the preprocessing and labeling method includes:

[0022] For each target vehicle’s acquired 3D point cloud data,

[0023] Label the target category, as well as the 3D coordinates, 3D size, and orientation of the 3D bounding box;

[0024] The three-dimensional point cloud data is voxelized, and the point cloud data in each voxel is aggregated into a feature vector;

[0025] Perform data augmentation operations on voxelized 3D point cloud data.

[0026] A further technical solution is to use training samples and verification samples to train and verify the model, including:

[0027] Initialize the parameters of the object detection model using ImageNet pre-trained weights;

[0028] Forward propagate the training samples through the model to obtain the predicted 3D bounding box and category probability, and calculate the total loss based on the predicted results and the true labels;

[0029] Calculate the gradient through back-propagation and use the Adam optimizer to update the model parameters;

[0030] Use validation samples to evaluate model performance and save the best performing model during training.

[0031] Its further technical solution is to switch to the corresponding operation mode according to the average vehicle speed, vehicle queue length and vehicle type, including:

[0032] When the length of the vehicle queue in the lane reaches the set length threshold, it automatically switches to the unblocking operation mode to alleviate the vehicle backlog;

[0033] When the average speed in the lane reaches the set speed threshold, it automatically switches to the passing mode to provide priority passing opportunities for fast-moving convoys;

[0034] When the number of large vehicles in the lane reaches the set vehicle threshold, it automatically switches to economic operation mode to reduce emissions from large vehicles when idling and waiting.

[0035] Its further technical solution is, in the economic operation mode:

[0036] If the model tracks a large vehicle approaching the lane exit and the remaining green light time is greater than the set threshold, the green light release time is increased;

[0037] If the model detects that the queue length of large vehicles in the fleet is greater than the set threshold and the remaining time of the green light is less than the set threshold, the corresponding phase of the next phase sequence will be advanced.

[0038] A further technical solution is to perform three-dimensional reconstruction on the real-time color video streams and real-time depth video streams collected by multiple depth cameras, including:

[0039] Extract the two-dimensional color image and depth data in the corresponding video stream at each sampling moment from the real-time color video stream and real-time depth video stream collected by each depth camera;

[0040] Using the depth data, each pixel in the two-dimensional color image is converted into three-dimensional point cloud data in a three-dimensional coordinate system established with the position of the depth camera as the origin;

[0041] The transformation relationship between the three-dimensional coordinate system of the depth camera and the intersection coordinate system is determined according to the installation position of each depth camera at the intersection. The three-dimensional point cloud data in the three-dimensional coordinate system of each depth camera is transformed into the intersection coordinate system according to multiple transformation relationships to obtain the three-dimensional reconstructed image at each sampling time.

[0042] A further technical solution is that the three-dimensional coordinates of the three-dimensional point cloud data in the three-dimensional coordinate system of the depth camera are:

[0043]

[0044] Among them, (x, y) is the two-dimensional coordinate of any pixel point in the two-dimensional color image, f is the focal length of the depth camera, and d is the depth data corresponding to the pixel point in the depth map.

[0045] The beneficial technical effects of the present invention are:

[0046] 1) The traffic light adaptive control method based on a multi-view depth camera proposed in the present invention does not require the use of an induction coil, thereby reducing the cost and difficulty of equipment construction and subsequent maintenance.

[0047] 2) Compared with the algorithm using a multi-view depth camera and a video camera and a radar as data acquisition devices, the method of the present invention can obtain more accurate lane information, including queue length, vehicle type, and average speed, and can provide more effective data support for adaptive control of adjusting phase duration and phase sequence.

[0048] 3) The present invention uses the average gray value change around the lane entrance and the stop line as the trigger condition of the target tracking algorithm, so that the target detection model does not need to work continuously for a long time, reducing unnecessary calculations. It can effectively save computing resources for the situation at night when the number of passing vehicles is very small, ensuring that the algorithm can run for a long time and extending the service life of the equipment.

[0049] 4) The present invention adopts the idea of ​​3D reconstruction and restoration of road conditions by multi-view depth cameras, which is applicable to a variety of different application scenarios. It overcomes the problems of poor scalability and narrow application scenarios of traditional neural network algorithms, and can complete tasks simply and quickly according to needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of a method for adaptively controlling a traffic light based on a multi-view depth camera provided in this application.

[0051] Figure 2 It is a schematic diagram of the mapping relationship between a two-dimensional color image and a three-dimensional space provided in this application.

[0052] Figure 3 It is a network feature structure diagram of the target detection model based on 3D-YOLO provided in this application. DETAILED DESCRIPTION

[0053] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0054] In one embodiment of the present application, a traffic light adaptive control method based on a multi-view depth camera is provided. Figure 1 As shown, the method specifically comprises the following steps:

[0055] Step 2: Use multiple depth cameras deployed at the intersection to obtain real-time video streams of the intersection. Multiple depth cameras shoot the intersection from different perspectives, and ensure that the field of view of multiple depth cameras covers the entire lane. The real-time video stream obtained by each depth camera includes a real-time color video stream and a real-time depth video stream, and the obtained real-time color video stream and real-time depth video stream are aligned.

[0056] In this embodiment, a depth camera is arranged at the entrance and exit of the lane to be tested and facing the road, and each depth camera establishes a communication connection with the computing unit of the traffic light at the intersection, and transmits the real-time collected color video stream and depth video stream to the computing unit for image processing and target detection and tracking.

[0057] Step 4: Use the 3D-YOLO-based target detection model to track the real-time depth video stream collected by each depth camera to obtain the average vehicle speed.

[0058] Specifically, the average grayscale value within a preset range in the real-time depth video stream collected by each depth camera is first calculated, and the preset range includes video frames around the lane entrance and video frames around the stop line. In this embodiment, two virtual coils are defined perpendicular to the road direction in the depth video stream sequence, which are respectively set at the lane entrance and the stop line to replace the real underground induction coil, and the change of the average grayscale value within the range of two meters before and after the two virtual coils is detected. When the average grayscale value reaches the set grayscale threshold, it is considered that a vehicle passes through or leaves the preset range (i.e., the corresponding virtual coil). When a vehicle passes through the import virtual coil, the number of vehicle queues is increased by one, and then the target detection model based on 3D-YOLO is used to track the vehicle. When the vehicle passes through the stop line virtual coil, the number of vehicle queues is reduced by one, and the travel time of the vehicle from passing to leaving the preset range is recorded, thereby calculating the average speed of the target vehicle.

[0059] Step 6: Perform 3D reconstruction on the real-time color video streams and real-time depth video streams collected by multiple depth cameras, and then input them into the 3D-YOLO-based target detection model for target detection to obtain the length of the vehicle queue and the vehicle category. The implementation method of 3D reconstruction specifically includes:

[0060] (1) From the real-time color video stream and real-time depth video stream captured by each depth camera, extract the two-dimensional color image and depth data in the corresponding video stream at each sampling moment.

[0061] (2) Using the depth data, each pixel in the two-dimensional color image is converted into three-dimensional point cloud data in a three-dimensional coordinate system with the position of the depth camera as the origin. Figure 2 As shown in the schematic diagram, each depth camera establishes a three-dimensional coordinate system of the depth camera with the installation position O as the origin. The three-dimensional coordinate system takes the plane where the target surface of the depth camera is located as the XY plane, the positive direction of the Z axis is perpendicular to the target surface, and the X-axis direction and the Y-axis direction are set according to requirements. The image coordinate system where the two-dimensional color image obtained by the depth camera is located takes the focus F of the depth camera as the origin, the x-axis of the image coordinate system is in the same direction as the x-axis of the three-dimensional coordinate system, and the y-axis of the image coordinate system is in the same direction as the y-axis of the three-dimensional coordinate system.

[0062] Based on this imaging principle, the three-dimensional coordinates of the three-dimensional point cloud data M obtained by converting any pixel point m in the color image acquired by each depth camera can be determined as:

[0063]

[0064] Among them, (x, y) is the two-dimensional coordinate of any pixel m in the two-dimensional color image, f is the focal length of the depth camera, and d is the depth data corresponding to the pixel in the depth map.

[0065] (3) After obtaining the 3D point cloud data in the 3D coordinate system of each depth camera, since there are multiple depth cameras with different viewing angles at the intersection, it is necessary to fuse the 3D point cloud data collected by multiple depth cameras. Specifically, the transformation relationship between the 3D coordinate system of the depth camera and the intersection coordinate system is determined according to the installation position of each depth camera at the intersection, and the 3D point cloud data in the 3D coordinate system of each depth camera is transformed into the intersection coordinate system according to multiple transformation relationships, and then some operations such as deduplication and abnormal point removal are performed to obtain the 3D reconstructed image at each sampling time.

[0066] Finally, the 3D reconstructed images at each sampling moment are input into the 3D-YOLO-based target detection model for target detection. The length of the 3D bounding box of each target vehicle is calculated to obtain the total length of the current queue, and the number of large vehicles such as buses or trucks is recorded.

[0067] Step 8: Switch to the corresponding operation mode according to the average vehicle speed, vehicle queue length and vehicle type, and execute the signal light timing plan in this mode. A variety of operation modes are stored in the calculation unit of the intersection signal, and the operation mode can be adaptively switched according to the different needs of the deployed intersection, mainly including: unblocking operation mode, passing operation mode and economic operation mode. The switching methods include:

[0068] (1) When the length of the vehicle queue in the lane reaches the set length threshold, it automatically switches to the unblocking operation mode. In this mode, the green light duration of the corresponding phase is dynamically adjusted according to the vehicle backlog situation, and the traffic lights at other intersections are communicated to avoid simultaneous release and congestion again.

[0069] (2) When the average speed in the lane reaches the set speed threshold, the system automatically switches to the passing mode, which provides priority passing opportunities to fast-moving convoys, thus reducing waiting time in queues.

[0070] (3) When the number of large vehicles in the lane reaches the set vehicle number threshold, it automatically switches to the economic operation mode. This mode is mainly for the sections and time periods where large freight and passenger vehicles pass. The adjustment of phase duration and phase sequence in this mode specifically includes: If the model tracks a large vehicle approaching the lane exit virtual coil and the remaining green light time is greater than the set threshold, the green light release time is increased. If the model detects that the queue length of large vehicles in the fleet is greater than the set threshold, and the remaining green light time is less than the set threshold, the corresponding phase of the next phase sequence is advanced. The advanced phase is based on the phase with the largest number of large vehicles in the current lane. Running this mode can clear the queue of large vehicles as much as possible, reduce the emissions of large vehicles when idling and waiting, and achieve a green and environmentally friendly effect.

[0071] In the above method, a trained target detection model needs to be used when performing target tracking and detection. Therefore, this method also includes the following steps before step 2:

[0072] Step 0: Use a depth camera to collect 3D point cloud data containing various vehicle categories, and form training samples and verification samples after preprocessing and annotation. The specific implementation method of preprocessing and annotation includes: first, annotate the target category of each target vehicle's 3D point cloud data, as well as the 3D coordinates (x', y', z'), 3D size (length, width and height) and orientation of the 3D bounding box. Then, voxelize the 3D point cloud data and divide it into a fixed 3D voxel grid. The point cloud data in each voxel is aggregated into a feature vector. Finally, in order to increase the generalization ability of the model, the voxelized 3D point cloud data is subjected to data enhancement operations such as random rotation, translation and scaling, and then divided into training samples and verification samples according to a certain ratio.

[0073] Step 1: Build a 3D-YOLO-based target detection model, and use training samples and validation samples to train and validate the model.

[0074] This embodiment constructs a 3D-YOLO model based on the traditional YOLO framework. The new model is based on the efficient detection framework of the original model, which ensures high speed and high accuracy of detection; on this basis, a hybrid attention module is integrated to improve the classification accuracy and generalization ability of the model. The model includes an input layer, an encoder, a decoder, and an output layer connected in sequence. The input layer takes voxelized three-dimensional point cloud data as input and decomposes it into bird's-eye views in different directions, such as three bird's-eye views directly above, on the left, and on the right. The hybrid attention module integrated in the encoder is used to perform weighted processing on bird's-eye views in different directions to obtain a voxel feature map, and use multiple 3D convolutional layers and pooling layers to gradually extract high-level features from the voxel feature map. The network feature structure diagram is shown in the figure. Figure 3 As shown in the figure, the decoder uses a 3D upsampling layer to restore the feature maps of different levels to the original voxel resolution, and then fuses the features of different levels to obtain a fused feature map to improve the detection accuracy. The output layer is used to identify the target on the fused feature map and predict the category probability of the target.

[0075] It should be noted that the hybrid attention module is implemented using the existing channel attention module and spatial attention module, and the internal composition of each module will not be described here.

[0076] When training and validating the above model, the parameters of the object detection model are initialized using the ImageNet pre-trained weights. Then the training samples are forward propagated through the model to obtain the predicted 3D bounding box and category probability. The total loss is calculated based on the predicted results and the true labels. The gradient is then calculated through backpropagation, and the model parameters are updated using the Adam optimizer. Finally, the model performance is evaluated using the validation samples, and the best performing model during the training process is saved for subsequent steps. The performance results of the trained model on various public datasets are shown in Table 1.

[0077] Table 1 Performance of 3D-YOLO model on public datasets

[0078]

[0079] As can be seen from Table 1, the 3D-YOLO model constructed in this application is suitable for detection in a variety of scenarios. It has good detection accuracy in each detection scenario, and the detection speed is also excellent, indicating that the target detection model of this application has a more outstanding detection advantage.

[0080] The above is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. A traffic light adaptive control method based on a multi-view depth camera, characterized in that: The method comprises: Multiple depth cameras deployed at the intersections are used to obtain real-time video streams of the intersections. The multiple depth cameras shoot the intersections from different perspectives, and the field of view of the multiple depth cameras covers the entire lane. The real-time video streams obtained by each depth camera include real-time color video streams and real-time depth video streams. Use the 3D-YOLO-based target detection model to track the real-time depth video stream collected by each depth camera to obtain the average vehicle speed; The real-time color video streams and real-time depth video streams collected by multiple depth cameras are three-dimensionally reconstructed, and then input into the target detection model based on 3D-YOLO for target detection to obtain the length of the vehicle queue and the vehicle category; The corresponding operation mode is switched according to the average vehicle speed, the length of the vehicle queue and the vehicle type, and the signal light timing plan in this mode is executed.

2. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 1, characterized in that: The target detection model based on 3D-YOLO is used to track the real-time depth video stream collected by each depth camera to obtain the average vehicle speed, including: Calculate the average grayscale value within a preset range in the real-time depth video stream collected by each depth camera, where the preset range includes video frames around the lane entrance and video frames around the stop line; When the average grayscale value reaches the set grayscale threshold, it is considered that a vehicle has passed through or left the preset range. The vehicle is tracked using a 3D-YOLO-based target detection model, and the duration of the vehicle's passage from passing through to leaving the preset range is recorded to calculate the average vehicle speed.

3. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 1, characterized in that: The target detection model based on 3D-YOLO includes: The input layer is used to decompose the voxelized 3D point cloud data into bird's-eye views in different directions; An encoder, for performing weighted processing on the bird's-eye view images in different directions to obtain a voxel feature map, and using multiple 3D convolutional layers and pooling layers to gradually perform high-level feature extraction on the voxel feature map; The decoder is used to restore the feature maps of different levels to the original voxel resolution using the 3D upsampling layer, and then fuse the features of different levels to obtain a fused feature map; The output layer is used to perform target recognition on the fused feature map and predict the category probability of the target.

4. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 1, characterized in that: The method further comprises: Use a depth camera to collect 3D point cloud data containing various vehicle categories, and form training samples and verification samples after preprocessing and annotation; The 3D-YOLO-based target detection model is constructed, and the training samples and verification samples are used to train and verify the model.

5. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 4, characterized in that: The pre-processed and labeled method comprises: For each target vehicle’s acquired 3D point cloud data, Label the target category, as well as the 3D coordinates, 3D size, and orientation of the 3D bounding box; voxelize the three-dimensional point cloud data, and aggregate the point cloud data in each voxel into a feature vector; Perform data augmentation operations on voxelized 3D point cloud data.

6. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 4, characterized in that: The model is trained and verified using the training samples and the verification samples, including: Initialize the parameters of the object detection model using ImageNet pre-trained weights; Forward propagation of the training sample through the model to obtain predicted three-dimensional bounding boxes and category probabilities, and calculating the total loss based on the predicted results and the true labels; Calculate the gradient through back-propagation and use the Adam optimizer to update the model parameters; The model performance is evaluated using the validation samples, and the best performing model during the training process is saved.

7. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 1, characterized in that: Switching to a corresponding operation mode according to the average vehicle speed, the length of the vehicle queue and the vehicle type includes: When the length of the vehicle queue in the lane reaches the set length threshold, it automatically switches to the unblocking operation mode to alleviate the vehicle backlog; When the average speed in the lane reaches the set speed threshold, it automatically switches to the passing mode to provide priority passing opportunities for fast-moving convoys; When the number of large vehicles in the lane reaches the set vehicle threshold, it automatically switches to economic operation mode to reduce emissions from large vehicles when idling and waiting.

8. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 7, characterized in that: In the described economical operating mode: If a large vehicle is tracked by the model to be approaching the lane exit and the remaining green light time is greater than the set threshold, the green light release time is increased; If the model detects that the queue length of large vehicles in the fleet is greater than a set threshold, and the remaining time of the green light is less than the set threshold, the corresponding phase of the next phase sequence is advanced.

9. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 1, characterized in that: The three-dimensional reconstruction of the real-time color video streams and real-time depth video streams collected by multiple depth cameras includes: Extract the two-dimensional color image and depth data in the corresponding video stream at each sampling moment from the real-time color video stream and real-time depth video stream collected by each depth camera; Using the depth data, convert each pixel in the two-dimensional color image into three-dimensional point cloud data in a three-dimensional coordinate system established with the position of the depth camera as the origin; According to the installation position of each depth camera at the intersection, the conversion relationship between the three-dimensional coordinate system of the depth camera and the intersection coordinate system is determined, and the three-dimensional point cloud data in the three-dimensional coordinate system of each depth camera is converted to the intersection coordinate system according to multiple conversion relationships to obtain a three-dimensional reconstructed image at each sampling time.

10. The method for adaptively controlling traffic lights based on a multi-view depth camera according to claim 9, characterized in that: The three-dimensional coordinates of the three-dimensional point cloud data in the three-dimensional coordinate system of the depth camera are: Wherein, (x, y) is the two-dimensional coordinate of any pixel point in the two-dimensional color image, f is the focal length of the depth camera, and d is the depth data corresponding to the pixel point in the depth map.

Images