Traffic indication method and system based on multispectral pedestrian detection and vehicle speed detection

Abstract

The application provides a traffic indication method based on multispectral pedestrian detection and vehicle speed detection, obtains the number of waiting pedestrians and the number of crossing pedestrians of a target intersection through pedestrian detection based on YOLOv4; combines the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to calculate and obtain the switching time of the signal light of the target intersection; combines the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to obtain a driving speed threshold of the associated road section; detects the driving speed of the road section associated with the target intersection, and when the driving speed is not within the driving speed threshold, a warning signal is sent out at the target intersection; the pedestrian detection based on YOLOv4 includes fusing color flow and temperature flow, generating fused flow to extract features, and transmitting information of the fused flow from the bottom to the top of the neural network at different stages; and further includes fusing color flow and temperature flow according to the lighting conditions of the target intersection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-spectral pedestrian detection, and in particular to a traffic indication method and system based on multi-spectral pedestrian detection and vehicle speed detection. BACKGROUND

[0002] With the increase of urban population, urban planners and designers must focus more on designing smart cities and addressing major challenges such as traffic problems and the impact of increased vehicle usage. Intelligent transportation systems and people-centered transportation methods must be developed to protect our pedestrians and cyclists and ensure that they can safely, efficiently, and comfortably reach their destinations.

[0003] In recent years, pedestrian detection technology, especially pedestrian detection based on visible light cameras, has developed rapidly using deep convolutional neural networks, and the performance of pedestrian detectors has also greatly improved. Pedestrian detection technology is a core technology in the fields of autonomous driving, robotics, and intelligent video surveillance. Pedestrian detection identifies and tracks road pedestrians through image processing, computer vision-related algorithms, and machine learning techniques, and has wide application prospects in intelligent vehicles, autonomous navigation, and motion analysis.

[0004] In existing traffic systems, crossing facilities include intersection flat crossing, road section flat crossing, and overpass crossing, and generally the flat crossing method should be preferred. However, in actual driving scenarios, the surrounding environment is very complex. There are differences in surrounding buildings, traffic signs, and background objects of different traffic. Seasonal changes will also cause significant changes in road scenes, and the most important change in road scenes is lighting. During the day, buildings and trees can cause partial shadows. In rainy and foggy weather, the overall brightness of the environment will decrease. At night, due to the lack of light, pedestrians are almost integrated into the background, and due to poor light conditions, numerous traffic accidents occur at night.

[0005] Detectors based on visible light images are generally only suitable for well-lit road scenes and cannot effectively detect pedestrians at night. Far-infrared (FIR) cameras also play an important role in pedestrian detection, especially at night or in low light conditions. The effective wavelength range of FIR cameras is 6-15 microns, while the wavelength of the heat emitted by the human body is about 9.3 microns. At night, pedestrians are brighter than the surrounding environment in the image. FIR images have the advantage of isolating pedestrians from the surrounding environment, making them suitable for pedestrian detection at night or in low light conditions, and thus can be used as a supplement to visible light sensors.

[0006] Pedestrian detection technology is the core technology in the research fields of automatic driving, robots and intelligent video monitoring. Pedestrian detection identifies and tracks road pedestrians through image processing, computer vision related algorithms and machine learning technology, and has wide application prospects in intelligent vehicles, automatic navigation, motion analysis and other fields.

[0007] In real-world object detection applications, the environment is usually open and dynamic, and models and algorithms are needed to cope with the challenges brought by openness, such as rain, fog, occlusion, poor light, low resolution, etc. Under these conditions, it is difficult to achieve high accuracy using only visible band sensor data algorithms.

[0008] And in the existing traffic system, pedestrians rely on the prompt of the opposite traffic light and their own estimation of the speed of the oncoming vehicle when using the crossing facility. When the speed of the car on the adjacent section is abnormal, the pedestrian cannot avoid it in time relying on the prompt of the existing traffic light. The construction time of the road section is earlier, and the road section cannot be transformed at low cost and high quality, which causes the visual limitation of the pedestrian crossing and the vehicle passing through the intersection, and the crossing signal light and other equipment provide less information, which cannot meet the demand of the pedestrian crossing safely. The crossing facility cannot give effective prompt to the oncoming vehicle according to the density of the pedestrians. In the case of small visible distance (such as fog, haze, rain, snow, dust, hail, etc.), although the pedestrian crossing signal light is set on the road section with high demand for crossing, the driver of the car cannot accurately judge whether the crossing facility in front has pedestrians using it. In addition, in the case of lack of lighting or poor lighting conditions in the intersection section and pedestrian crossing space at night, pedestrians cannot judge the speed of the oncoming vehicle. The detection system based on the net has detection delay, long distance transmission and data loss when returning data, which further increases the cost of deployment and maintenance. The traffic signal light cannot be automatically adjusted according to the pedestrian flow, which easily causes the motor vehicle to wait for a long time.

[0009] The existing traffic vision system may face some major challenges and difficulties: poor video quality due to low camera resolution, light conditions, dirty lens, no adjustment or weather conditions; processing stretched video collected by traffic cameras and wide-angle lenses; non-ideal angle, position and direction of the camera; camera vibration and shaking due to wind or passing cars; distortion of light at night; inconsistent light changes, shadow effects; moving or stationary objects that may block the target view. SUMMARY

[0010] In order to overcome the above technical defects, the purpose of the present application is to provide a multi-spectral pedestrian detection and vehicle speed detection traffic indication method and system for intersections with large changes in pedestrian flow, in order to reduce the risk of accidents for pedestrians, reduce the cost of traffic control manpower, and reduce the waiting time of vehicles at the intersection when there are few pedestrians.

[0011] The application discloses a traffic indication method based on multispectral pedestrian detection and vehicle speed detection, comprising the following steps: obtaining the number of waiting pedestrians and the number of crossing pedestrians at a target intersection through pedestrian detection based on YOLOv4; combining the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to calculate and obtain the switching time of a signal lamp at the target intersection; combining the number of waiting pedestrians and the number of crossing pedestrians with road information of the road section associated with the target intersection to obtain a driving speed threshold of the associated road section; detecting the driving speed of the road section associated with the target intersection, and when the driving speed is not within the driving speed threshold, a warning signal is sent out at the target intersection; the pedestrian detection based on YOLOv4 comprises fusing color flow and temperature flow, generating fused flow to extract features, and transmitting information of the fused flow from the bottom to the top of the neural network at different stages; and further comprising fusing color flow and temperature flow according to the lighting conditions of the target intersection.

[0012] Preferably, the fusing color flow and temperature flow, generating fused flow to extract features, and transmitting information of the fused flow from the bottom to the top of the neural network at different stages comprises: first generating a proposal of a detection parent, and then classifying the proposal to determine whether the proposal is a pedestrian or a background; using single-stage detection of a full convolutional architecture, and predicting a bounding box of a confidence score of each spatial position through a convolutional neural network.

[0013] Preferably, the pedestrian detection comprises: using dimension clusters as anchor boxes of three different scales to predict bounding boxes; using logistic regression to predict an objectiveness score of each bounding box; and using multi-label classification to predict the class of each box.

[0014] Preferably, the pedestrian detection comprises: performing a data enhancement process to increase the variability of input images; and using mosaics and adversarial training to provide images for the data enhancement process.

[0015] Preferably, the pedestrian detection comprises: using a convolutional neural network to extract feature detection of a target pedestrian; wherein, according to the size of the features, the convolutional layer is divided into five stages. Given an input image with a size of HxW, the sizes of the three layers before the head are The receiving field is enhanced through a spatial pyramid pooling module backbone, and a path aggregation network is used as a detection neck to aggregate features from different backbone levels.

[0016] Preferably, the pedestrian detection comprises: generating three different scale bounding boxes, using dimension clusters as anchor boxes for each head to predict bounding boxes, and using a full intersection on a joint as a loss function for regression; predicting an objectiveness score of each bounding box through logistic regression; and using multi-label classification to predict the class of each bounding box.

[0017] Preferably, the fusing of the color stream and the temperature stream according to the lighting condition of the target intersection comprises: input fusion, early fusion I, early fusion II, mid fusion, late fusion and direct fusion.

[0018] Preferably, the fusing of the color stream and the temperature stream according to the lighting condition of the target intersection comprises: detecting target pedestrians in a plurality of sequential frames; predicting the next position in the next frame according to the current position and walking speed of the target pedestrian using the optimal state estimator; after receiving the next frame, comparing the predicted next position with the actual position detected in the next frame; if the predicted next position matches the actual position detected in the next frame, regarding the target pedestrian as the previous target pedestrian and continuing to predict to obtain the trajectory of the target pedestrian; thereby obtaining the trajectories of all target pedestrians in the frame; and regarding the target pedestrian whose actual position detected does not match any position predicted before as a new target pedestrian and increasing the target pedestrian counter.

[0019] Preferably, the fusing of the color stream and the temperature stream according to the lighting condition of the target intersection comprises: connecting the features of the color stream and the temperature stream together in the channel dimension. The concatenated feature map can be shown as: F i =Concat(C I , T i ); wherein Ci, Ti are the color stream and the temperature stream of the i-th level. r is the step length, r = 2 i ; a channel feature vector F1 is generated using a global average pool and a global maximum pool; the channel elements of the global average pool and the global maximum pool are calculated using the following formula: A new compact module F 2 is created to learn the fusion weights of the color and thermal functions; this is achieved through a low-dimensional fully connected (FC) layer: z = F 2 = FC (F 1 ); normalization is performed using softmax, and the learned weights αc and βc are used to select the corresponding features for the final fusion Fc; αc and βc are the scale values of the channel c and αc, and αc, βc ∈ [0, 1]; Fc = αc·Cci+ βc·Tci.

[0020] The application further discloses a traffic indication system based on multispectral pedestrian detection and vehicle speed detection, which comprises a pedestrian detection module based on YOLOv4, a vehicle speed detection module and a signal lamp control module; the number of waiting pedestrians and the number of crossing pedestrians of a target intersection are acquired through the pedestrian detection module based on YOLOv4; the signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to calculate and acquire the switching time of the signal lamp of the target intersection; the signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to acquire a vehicle speed threshold of the associated road section; the vehicle speed detection module detects the vehicle speed of the road section associated with the target intersection, and when the vehicle speed is not within the vehicle speed threshold, the signal lamp control module sends a warning signal to vehicles at the target intersection; the pedestrian detection module based on YOLOv4 comprises a channel feature fusion unit and a multispectral fusion unit; the channel feature fusion unit fuses color flow and temperature flow to generate a fusion flow to extract features, and information of the fusion flow is transmitted from the bottom to the top of the neural network at different stages; the multispectral fusion unit fuses color flow and temperature flow according to the lighting conditions of the target intersection.

[0021] After the above technical scheme is adopted, the following beneficial effects are obtained compared with the prior art:

[0022] 1. When pedestrians pass through the crossing facilities, the pedestrian detection can be performed to prompt the crossing situation of the road section in front of the vehicle; when the vehicle speed exceeds the safety value, or the pedestrians do not comply with the traffic rules and cannot distinguish the traffic light information, the vehicle and the pedestrians are prompted to effectively reduce the accident probability; in the case that the visual distance of the automobile driver is small, the pedestrian condition of the intersection and the crossing facilities in front of the automobile driver can be prompted; the intersection signal lamp time is dynamically adjusted according to the pedestrian flow of the crossing facilities, and the waiting time and safety are optimized;

[0023] 2. The fusion algorithm used in the application can realize low-error pedestrian recognition and pedestrian trajectory prediction under various climate conditions according to different lighting environments, and has higher robustness and practicality than existing recognition methods; the single-level detection mechanism is used, the hardware information transmission process is more direct, the prompt information output of the traffic signal device does not need to go through complex operation, and the prompt information transmission is more timely;

[0024] 3. The system device is only installed on the pedestrian crossing facilities and the adjacent vehicle warning light section, without long-distance communication, easy to replace and maintain; the vehicle speed is measured by using the Doppler speedometer, which has high applicability and low maintenance cost, and the actual use scene requirement is not high; the system device is established on the mature hardware that can be provided by the industry, such as the Doppler speedometer for vehicle speed detection, the camera for multi-spectrum pedestrian detection, and the traffic warning light can be based on the hardware of the existing facilities, without too much modification cost, so that it can be deployed with high efficiency, less modification, without making great changes to the road section, and effectively prompting the pedestrians and the drivers. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flow chart of the traffic indication method based on multi-spectrum pedestrian detection and vehicle speed detection provided by the present application is provided.

[0026] Figure 2 The YOLOv4 preferred algorithm diagram of the traffic indication method based on multi-spectrum pedestrian detection and vehicle speed detection provided by the present application is provided.

[0027] Figure 3 The fusion algorithm diagram of the traffic indication method based on multi-spectrum pedestrian detection and vehicle speed detection provided by the present application is provided.

[0028] Figure 4 The multi-spectrum channel feature fusion (MCFF) module diagram provided by the present application is provided. DETAILED DESCRIPTION

[0029] The advantages of the present application are further described below in combination with the drawings and specific embodiments.

[0030] Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] The terms used in the present disclosure are merely for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "an" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. can be employed in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0033] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation to the present application.

[0034] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between two elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0035] In the subsequent description, the suffix such as "module", "component" or "unit" used to represent elements is only for the convenience of description of the present application, and has no specific meaning in itself. Therefore, "module" and "component" can be used interchangeably.

[0036] In view of the fact that multispectral imaging technology can provide a combined formation from multispectral cameras (such as visible light and thermal energy), by fusing the complementarity of different modes, the perceptibility, reliability and robustness of the detection algorithm can be further improved.

[0037] The vehicle speed detector refers to an instrument for testing the speed of a vehicle in motion, and its principle is based on the Doppler effect, that is, the vehicle speed is proportional to the change of microwave frequency. The detector transmits microwaves, and the Doppler effect of the reflected waves can immediately indicate the vehicle speed reading.

[0038] The application discloses a traffic indication system based on multispectral pedestrian detection and vehicle speed detection, comprising a pedestrian detection module based on YOLOv4, a vehicle speed detection module and a signal lamp control module. The signal lamp control module comprises a processor and an intersection signal lamp device. The pedestrian detection module based on YOLOv4 and the vehicle speed detection module are connected with the processor, which is used for receiving data from the pedestrian detection module based on YOLOv4 and the vehicle speed detection module, processing the data to obtain result data, and controlling the intersection signal lamp device through the result data.

[0039] Preferably, the intersection signal lamp device comprises a traffic light (signal lamp), a car approaching prompt lamp and an abnormal vehicle speed warning lamp.

[0040] Specifically, the pedestrian detection module based on YOLOv4 is used to detect and obtain the number of waiting pedestrians and the number of crossing pedestrians at a target intersection. The processor of the signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with the road information of the road section associated with the target intersection to calculate and obtain the switching time of the signal lamp at the target intersection, so as to realize dynamic adjustment of the signal lamp.

[0041] The processor of the signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with the road information of the road section associated with the target intersection to obtain the normal vehicle speed threshold of the associated road section. The vehicle speed detection module detects the vehicle speed of the road section associated with the target intersection. When the vehicle speed is not within the vehicle speed threshold, the signal lamp control module sends a car warning signal at the target intersection, specifically through the abnormal vehicle speed warning lamp and / or the car approaching prompt lamp.

[0042] It should be noted that the road section associated with the target intersection of the application can be understood as a plurality of road sections (referred to as first associated road sections) sharing the same traffic intersection, and can also include road sections (referred to as second associated road sections) not connected with the target traffic intersection but influencing the first associated road sections. Thus, more data can be obtained to form a real-time and dynamic detection mechanism, and the processing result of the data is more accurate and closer to the actual traffic.

[0043] The road information includes the length, width and curvature of the road section.

[0044] The pedestrian detection module comprises a channel feature fusion unit and a multispectral fusion unit. The channel feature fusion unit based on YOLOv4 fuses color flow and temperature flow to generate a fusion flow to extract features, and transmits the information of the fusion flow from the bottom to the top of the neural network at different stages. The several multispectral fusion units fuse color flow and temperature flow according to the lighting conditions of the target intersection.

[0045] The application discloses a traffic indication method based on multispectral pedestrian detection and vehicle speed detection, comprising the following steps:

[0046] The number of waiting pedestrians and the number of crossing pedestrians at the target intersection are obtained through pedestrian detection based on YOLOv4; the number of waiting pedestrians and the number of crossing pedestrians are combined with the road information of the road section associated with the target intersection to calculate the switching time of the signal light at the target intersection;

[0047] The number of waiting pedestrians and the number of crossing pedestrians are combined with the road information of the road section associated with the target intersection to obtain the driving speed threshold of the associated road section;

[0048] The driving speed of the road section associated with the target intersection is detected, and when the driving speed is not within the driving speed threshold, a warning signal is issued at the target intersection.

[0049] Referring to the accompanying drawings Figure 1 The pedestrian detection specifically includes the following method flow: first, the video stream of the camera is obtained, the video is processed for quality enhancement, and the brightness and contrast are adjusted, and for wide-angle lenses that may cause the image to protrude, a correction algorithm can also be used to convert the video to a more natural view; then the foreground and background of the video are estimated, and then the computer is used for visual feature extraction, and the detected features are input into the machine learning model.

[0050] The machine learning model has been previously learned and trained, and specifically includes feature extraction, feature selection, and or reducing data dimension. The machine learning model includes R-CNN (Region-based Convolutional Network) and YOLO algorithm. Compared with other classic machine learning algorithms, one of the great advantages of the YOLO method is that no manual adjustment is needed for the generation of the YOLO model. The algorithm automatically learns to generate the best convolution feature set that can best represent the image. In addition, when the training data set is not large enough, it is usually difficult to train a deep neural network. In this case, the transfer learning method of using a pre-trained neural network model on another data set can be very helpful to alleviate and speed up the training phase.

[0051] The learned machine learning model outputs the detection result as a trajectory prediction of a sequence frame, obtains a trajectory graph of the pedestrian through the trajectory prediction, thereby tracking and counting the pedestrian, and performing statistics to obtain the final number of waiting pedestrians and the number of crossing pedestrians.

[0052] Specifically, after detecting the target objects (such as pedestrians or cyclists) in several sequential frames, a best state estimator is used to estimate the trajectory of each target object. Since several objects (for example, several pedestrians walking together in the same direction or in different directions) can exist in each frame at a time, the trajectory of each object must be estimated separately.

[0053] First, the target pedestrians in several sequential frames are detected. Then, a Kalman filter is used as the best state estimator to predict the next position in the next frame based on the current position and walking speed of the target pedestrian. After receiving the next frame, the predicted next position is compared with the actual position detected in the next frame. If the predicted next position matches the actual position detected in the next frame, the target pedestrian is considered as the previous target pedestrian, and the trajectory of the target pedestrian is obtained by continuous prediction; thus, the trajectories of all target pedestrians in the frame are obtained. The target pedestrian whose actual position does not match any of the previously predicted positions is considered as a new target pedestrian, and the target pedestrian counter is increased to avoid repeated counting in sequential frames.

[0054] The present application proposes a multispectral pedestrian detector with aligned color-thermal images as input. In good light conditions, color images are more detailed than thermal images (e.g., edge, texture, and color information). However, when color images are captured in weak lighting conditions or at night, functional information is lost, and detection effectiveness is reduced. Thermal sensors are not sensitive to light and can capture the outline of pedestrians in insufficient light. However, the disadvantage of thermal imaging is also the lack of texture and color information. Therefore, fusing color and thermal images helps to cope with the challenge of insufficient lighting. Fusing color images and thermal images can be used to enrich image information, reduce the error rate of pedestrian detection, and improve the robustness of the model under various lighting conditions.

[0055] The YOLOv4-based pedestrian detection of the present application can replace the current most advanced two-stage fast RCNN for multispectral pedestrian detection and directly predict the bounding box with a confidence score. To further improve the detection performance, the existing multispectral fusion method is analyzed, and a new multispectral channel feature fusion (MCFF) module is proposed to integrate the features of color flow and temperature flow according to the lighting conditions.

[0056] Specifically, color flow and temperature flow are fused to generate a new fusion flow to extract features and transmit fusion information from the bottom to the top of the convolutional neural network at different stages. The features of color flow and temperature flow are integrated according to the lighting conditions. Thus, the missing of the baseline can be significantly reduced, and more excellent performance is provided for multispectral pedestrian detection.

[0057] Convolutional neural networks (CNNs) are similar to multilayer perceptrons, which are a subset of deep learning and are more commonly used for classification and computer vision tasks. The difference between convolutional neural networks and other neural networks is their superior performance in image, speech, or audio signal input. It has three main types of layers: convolutional layers, pooling layers, and fully connected (FC) layers.

[0058] The convolutional layer is the first layer of the convolutional network. While there can be additional convolutional layers or pooling layers after the convolutional layer, the fully connected layer is the final layer. As the flow of each layer, the complexity of the CNN increases, recognizing larger parts of the image. Early image layers focus on simple functions such as color and edges. As the image data passes through the layers of the CNN, it begins to recognize larger elements or shapes of the object until it eventually recognizes the intended object. The convolutional layer is the core building block of the CNN and where most of the computation occurs.

[0059] Regardless of whether a one-stage or two-stage multi-spectral pedestrian detection is used, multi-modal information fusion is throughout the entire detection stage. Different modes and different stages of features have different expression capabilities. The most commonly used fusion method in multi-spectral pedestrian detection is the method of network in network (MIN). MIN is used to fuse features of different modes and reduce the dimensionality of multi-modal features after concatenation of the convolutional layer to 1x1. Therefore, the problems of how and where the fusion occurs are two important problems in the backbone of multi-spectral detection. In view of the above two problems, the present patent proposes to use an attention fusion method to fuse color and heat flow together to detect pedestrians in weak lighting conditions. Specifically, the one-stage object detection method YOLOv4 is used to extract features from color and heat flow to improve performance, which can achieve real-time, high-quality and convincing object detection results.

[0060] At the same time, in order to further improve the fusion of different mode image data, the present patent proposes a cross-mode multi-spectral channel attention fusion mechanism to fuse image features between color flow and temperature flow.

[0061] The multi-spectral pedestrian detection method of the present patent can be divided into two-stage and single-stage methods according to the number of detection stages. In two-stage detection, the first stage is used to generate proposals of detection parents, and the second stage is used to classify the proposals to determine whether they are pedestrians or backgrounds.

[0062] Currently, the most difficult to operate multi-spectral pedestrian detection method is based on fast RCNN, which has achieved remarkable results, but the detection process is complex, and anchors must use hyperparameters (anchors are manually predefined with multiple scales, ratios and parameters), which is easy to cause detection errors. And the present invention adopts a single-stage detection of a full convolutional architecture, and predicts a bounding box of a confidence score for each spatial position through a convolutional neural network, and the bounding box is obtained by regression according to a predefined anchor or center point of a positive region. In the single-stage method, only one step is needed to obtain the final detection result; compared with two-stage detection, the single-stage detector is more efficient due to its simple structure. Compared with two-stage pedestrian detection, the single-stage has a faster inference speed. The single-stage multi-spectral object detector preferably adopts GFD-SSD, FusionCSPNe.

[0063] Referring to the accompanying drawings Figure 2YOLOv4 is an anchor-based single-stage object detection network that can directly obtain detection results by predicting bounding boxes with objectiveness scores and classes. YOLOv4 uses dimension clustering as three different scale anchor boxes to predict bounding boxes. The objectiveness score of each bounding box is predicted using logistic regression. Multi-label classification is used to predict the class of each box. The preferred can consist of three modules: data augmentation, feature extraction, and detection head.

[0064] The data augmentation module increases the variability of the input image so that the detector has higher robustness to images obtained from different environments. Mosaic and adversarial training are introduced to provide images for data augmentation.

[0065] The feature extraction module uses CNN to extract pedestrian feature maps for detection. In this detection method, CSPDarknet-53 is used as the backbone, which is divided into five stages according to the feature size. Given an input image with size HxW, the size of the three layers before the head is The backbone is enhanced with a spatial pyramid pooling (SPP) module, and the path aggregation network (PAN) is used as the detection neck to aggregate features from different backbone levels to improve the expression ability of the features.

[0066] The detector head module is used to generate three different scale bounding boxes. For each head, dimension clustering is used as an anchor box to predict the bounding box, and the complete intersection on the joint (CIoU) is used as the loss function for regression. The objectiveness score of each bounding box is predicted by logistic regression. Multi-label classification is used to predict the class of each box.

[0067] In multispectral pedestrian detection, fusing data in different ways is a very important step. An efficient fusion method should be able to supervise the fusion of information in different modes and improve the performance of the detector. The existing multispectral feature fusion methods are mainly SUM and MIN methods. The SUM operator represents the sum of the elements of the feature, which can be regarded as linear feature fusion with the same weight. The MIN method is used to reduce the dimension of the multi-modal feature after concatenation of the 1x1 convolution kernel; this is an unsupervised nonlinear feature fusion.

[0068] Color and thermal images play different roles under different lighting conditions. Except in some cases (standing in the shadow), most pedestrians have good lighting conditions during the day; however, thermal images are not sensitive to lighting. In contrast, thermal imaging can capture better visual features of pedestrians at night. The MCFF proposed in this invention can adaptively adjust the channel features between the color and thermal modes fused under different lighting conditions.

[0069] See the attachedFigure 3 Unlike other fusion algorithms that fuse different branches into one branch and lack the information transfer from bottom to top between each modality, this algorithm designs a novel feature fusion architecture based on color and heat to extract features. It fuses color and heat to generate a new fused stream to extract features and transfer fused information from bottom to top at different stages. For ease of illustration, the backbone can be divided into five blocks, which are named conv1-x, conv2-x, conv3-x, conv4-x, and conv5-x according to the feature map size. MCFF is used to fuse conv-c features in the color stream and conv-t features in the heat stream to obtain a new branch conv-f for pedestrian detection.

[0070] The present application also designs several fusion architectures based on MCFF, including input fusion, early fusion I, early fusion II, mid-fusion, late fusion, and direct fusion to transfer feature information from bottom to top at different stages. The experimental results of KAIST and U-tokyo pedestrian benchmarks show that the best performance of all architectures is obtained using mid-fusion.

[0071] Input fusion is data fusion before feature extraction. MIN is used to stack color and heat images in the channel dimension, and then reduce the channel size of multi-modal color-heat images to the channel size of input single-color modal images, as shown in a of Figure 3 MCFF is used to fuse the functions after the input layer.

[0072] Early fusion I fuses these features after the first convolutional block conv1; see Figure 3 b for details. MCFF is used to fuse the features of conv1-c in the color stream and conv1-t in the heat stream, which reduces the size of the fusion layer to 256, similar to the input of the new fusion branch conv-f. Early fusion models fuse low-level functions.

[0073] Early fusion II fuses these features after the second convolutional block conv2 of the new branch. MCFF is also used to fuse the functions of conv2-c in the color image branch and conv2-t in the heat image branch, as shown in c of Figure 3

[0074] As shown in d of Figure 3 Mid-fusion starts to fuse at the middle stage of feature extraction. Unlike early fusion, it starts to fuse two streams after the third convolutional block fusion 3. MCFF is also used after the concatenation layer for the same reason as discussed earlier. The functions of conv4 layers contain more semantic meaning than the functions of conv1, although they retain some fine visual details. Late fusion performs feature fusion at the end stage, as shown in​Figure 3 Only block conv5-f is used to transmit information. The fusion function of Conv3, Conv4 and Conv5-f is used to detect pedestrians, as shown by e in FIG. 3.

[0075] DirectFusion has the same fusion architecture as FusionCSPNet, but has a different fusion method. MCFF is introduced instead of MIN to balance the functions of the two streams. There is no new stream that fuses color and heat streams at different stages. The features of different modes are directly fused with the 3rd, 4th and 5th stages, as shown by f in FIG. 4. Figure 3

[0076] The features of the color stream and the temperature stream are concatenated together in the channel dimension. The concatenated feature map can be shown as: F i = Concat(C I , T i ); where Ci, Ti are the color stream and the temperature stream of the i-th level. r is the step, r = 2 i ; global average pooling and global maximum pooling are used to generate the channel feature vector F1; the channel elements of the global average pooling and the global maximum pooling are calculated using the following formula: A new compact module F 2 is created to learn the fusion weights of color and heat functions; it is implemented through a low-dimensional fully connected (FC) layer: z = F 2 = FC(F 1 ); normalized by softmax, the learned weights αc and βc are used to select the corresponding features for the final fusion Fc; αc and βc are the scale values of the channel c and αc, αc, βc ∈ [0, 1]; Fc = αc·Cci + βc·Tci.

[0077] Referring to FIG. 5, the fusion method of DirectFusion is shown. Figure 4 ​The output of the MCFF can be used as the input of the new fusion branch for feature extraction. The multi-modal feature fusion is performed in the third stage of the backbone, and the input image size is 1x640x640x3. The feature of the third stage thermal image is conv3-f, and the feature of the third stage color image is conv3-c. The feature size of the two modes is 1x80x80x256. First, the conv3-c and the conv3-f are concatenated in the channel dimension. Then, the global information of the feature is obtained by using the global average pool and the global maximum pool, and the weight of each channel is learned by using the FC layer and the illumination scale is extracted. The scale size is 1x1x256x2. Finally, the multi-modal feature is fused according to the learning scale of each channel reference formula. Under sufficient illumination, the weight of the color feature should be relatively large; otherwise, in the case of night or insufficient light, the weight of the thermal feature should be relatively large. According to the feature fusion method of the third stage, the fusion features of different modes in the fourth stage and the fifth stage can be obtained. The fusion feature branch also uses the same convolution module as the backbone to extract features and learn the model.

[0078] The present application firstly proposes a multi-spectral fusion architecture based on a single-stage object detection method, which transmits the information of the fusion stream from the bottom to the top and fuses the functions at different stages to improve the performance. And a new multi-spectral attention fusion method is provided to fuse the color stream and the temperature stream according to the illumination condition.

[0079] It should be noted that the embodiments of the present application have better implementation, and do not limit the present application in any form, and any skilled person in the art can change or modify the above disclosed technical content into equivalent effective embodiments, as long as it does not deviate from the content of the technical solution of the present application, and any modification or equivalent change and modification of the above embodiments according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A traffic indication method based on multispectral pedestrian detection and vehicle speed detection, characterized in that, The method comprises the following steps: acquiring the number of waiting pedestrians and the number of crossing pedestrians at a target intersection through pedestrian detection based on YOLOv4; combining the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to calculate the switching time of a signal light at the target intersection; combining the number of waiting pedestrians and the number of crossing pedestrians with road information of a road section associated with the target intersection to acquire a driving speed threshold of the associated road section; detecting the driving speed of the road section associated with the target intersection, and when the driving speed is not within the driving speed threshold, issuing a warning signal for stopping at the target intersection; the pedestrian detection based on YOLOv4 comprises fusing color flow and temperature flow, generating fused flow to extract features, and transmitting information of the fused flow from the bottom to the top of the neural network at different stages; and further comprises fusing color flow and temperature flow according to the lighting conditions of the target intersection; the fusing color flow and temperature flow according to the lighting conditions of the target intersection comprises: detecting target pedestrians in a plurality of sequential frames; predicting the next position in the next frame according to the current position and walking speed of the target pedestrian using an optimal state estimator; after receiving the next frame, comparing the predicted next position with the actual position detected in the next frame: if the predicted next position matches the actual position detected in the next frame, regarding the target pedestrian as a previous target pedestrian, and continuing to predict to acquire the trajectory of the target pedestrian; thereby acquiring the trajectories of all target pedestrians in the frame; and regarding a target pedestrian whose actual position is detected but does not match any previously predicted position as a new target pedestrian, and increasing the target pedestrian counter.

2. The traffic indication method according to claim 1, characterized by, the fusing color flow and temperature flow, generating fused flow to extract features, and transmitting information of the fused flow from the bottom to the top of the neural network at different stages comprises: first generating proposals of detection parents, and then classifying the proposals to determine whether the proposals are pedestrians or backgrounds; using a single-level detection of a full convolutional architecture, and predicting a bounding box of a confidence score of each spatial position through a convolutional neural network.

3. The traffic indication method according to claim 1, characterized by, the pedestrian detection comprises: using dimension clusters as anchor boxes of three different scales to predict bounding boxes; predicting an objectiveness score of each bounding box through logistic regression; and using multi-label classification to predict the class of each box.

4. The traffic indication method according to claim 1, characterized by, the pedestrian detection comprises: performing a data enhancement process to increase the variability of input images; and using mosaics and adversarial training to provide images for the data enhancement process.

5. The traffic indication method according to claim 1, wherein the pedestrian detection comprises: The features of the target pedestrian are detected by using a convolutional neural network; wherein, according to the size of the features, the convolutional layer is divided into five stages; given an image with an input size of HxW, the sizes of the three layers before the head are × , × , × ; enhancing the receptive field through a spatial pyramid pooling module backbone, and using a path aggregation network as a detection neck to aggregate features from different backbone levels.

6. The traffic indication method according to claim 1, wherein, the pedestrian detection comprises: generating bounding boxes of three different scales, using dimension clusters as anchor boxes for each head to predict bounding boxes, and using a full intersection on the joint as a loss function for regression; predicting an objectiveness score of each bounding box through logistic regression; using multi-label classification to predict the class of each bounding box.

7. The traffic indication method according to claim 1, wherein the fusing color flow and temperature flow according to the lighting conditions of the target intersection comprises: Input fusion, early fusion I, early fusion II, mid-fusion, late fusion and direct fusion.

8. A traffic indication system based on multispectral pedestrian detection and vehicle speed detection, characterized in that, The YOLOv4-based pedestrian detection module, the vehicle speed detection module and the signal lamp control module are included. The number of waiting pedestrians and the number of crossing pedestrians at a target intersection are obtained through the YOLOv4-based pedestrian detection module. The signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with the road information of the road section associated with the target intersection to calculate the switching time of the signal lamp at the target intersection. The signal lamp control module combines the number of waiting pedestrians and the number of crossing pedestrians with the road information of the road section associated with the target intersection to obtain the driving speed threshold of the associated road section; the vehicle speed detection module detects the driving speed of the road section associated with the target intersection, and when the driving speed is not within the driving speed threshold, the signal lamp control module sends a warning signal to the target intersection. The YOLOv4-based pedestrian detection module includes a channel feature fusion unit and a multispectral fusion unit. The channel feature fusion unit fuses color flow and temperature flow to generate a fusion flow to extract features, and transmits information of the fusion flow from the bottom to the top of the neural network at different stages; the multispectral fusion unit fuses color flow and temperature flow according to the lighting conditions of the target intersection. The fusion of color flow and temperature flow according to the lighting conditions of the target intersection includes: Detecting target pedestrians in a plurality of sequential frames; Using a best state estimator to predict the next position in the next frame according to the current position and walking speed of the target pedestrian; After receiving the next frame, comparing the predicted next position with the actual position detected in the next frame: If the predicted next position matches the actual position detected in the next frame, the target pedestrian is considered to be the previous target pedestrian, and the trajectory of the target pedestrian is obtained by continuing to predict; thus, the trajectories of all target pedestrians in the frame are obtained; And the target pedestrian whose actual position detected does not match any of the previously predicted positions is considered to be a new target pedestrian, and the target pedestrian counter is increased.

Images