Method and system for detecting and early warning target in tunnel based on multi-source information fusion

Abstract

The invention provides an in-tunnel target detection early warning method and system based on multi-source information fusion, and relates to the technical field of target detection.The method comprises the steps that firstly, a visual input image and a radar input image which are obtained in advance are fused through a multi-source information fusion target detection network, and a fusion feature map is generated; the multi-source information fusion target detection network comprises a feature extraction fusion network and a tunnel target prediction network, and the feature extraction fusion network comprises a single-stage double-path feature extraction and fusion module; and then, based on the fusion feature map, predicting and outputting the position and the category of a target in the tunnel through a tunnel target prediction network, and then determining an abnormal condition in the tunnel and performing early warning. Through the method, the technical problems of limited application range, low accuracy and poor stability of target detection in the tunnel in the prior art are solved, and the technical effects of expanding the application range and improving the detection accuracy and stability are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method and system for detecting and warning targets in a tunnel based on multi-source information fusion. Background Art

[0002] Since the structure of the tunnel is usually closed, the natural lighting conditions are extremely poor, and most tunnels do not have a complete lighting system. Therefore, in a low-light or no-light tunnel environment, the detection and monitoring warning of pedestrians, motor vehicles and non-motor vehicles has always been a thorny problem faced by the traffic management department. In the prior art, radar sensor detection, visible light sensor, infrared sensor and other methods are generally used to achieve target detection and warning in the tunnel. However, the existing detection methods have limitations in the use of tunnel environments. Radar sensors are easily affected by complex reflection environments. In dense traffic in tunnels, multi-target reflection signals will interfere with each other, resulting in a decrease in the accuracy of target detection positions; the lighting conditions in the tunnel are complex, the light is dim, the image quality is poor, and the changes in lighting at the entrance and exit such as strong light reflection will affect the quality of the pictures collected by the visible light sensor; infrared sensors are based on thermal radiation imaging, and the temperature difference between the target and the background directly affects the detection effect. They are easily affected by factors such as ambient temperature and humidity, which may lead to detection errors. It can be seen that a single type of sensor has disadvantages and problems in the application of tunnel target detection. In other words, the prior art has technical problems such as limited application scope, low accuracy and poor stability of target detection in tunnels. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for early warning of target detection in a tunnel based on multi-source information fusion, so as to alleviate the technical problems existing in the prior art of limited application scope, low accuracy and poor stability of target detection in a tunnel.

[0004] In the first aspect, an embodiment of the present invention provides a method for detecting and warning targets in a tunnel based on multi-source information fusion, the method comprising: fusing a pre-acquired visual input image and a radar input image through a multi-source information fusion target detection network to generate a fusion feature map; the multi-source information fusion target detection network comprises a feature extraction fusion network and a tunnel target prediction network; the feature extraction fusion network comprises a single-stage dual-path feature extraction and fusion module; based on the fusion feature map, the position and category of the target in the tunnel are predicted and output through the tunnel target prediction network; based on the predicted position and category of the target in the tunnel, abnormal conditions in the tunnel are determined and an early warning is issued.

[0005] In some optional implementations, the method further includes: using an infrared sensor and a radar to respectively collect original infrared images and radar three-dimensional point cloud data in the tunnel; preprocessing the original infrared images and the radar three-dimensional point cloud data to respectively obtain visual input images and radar three-dimensional point cloud data.

[0006] In some optional implementations, the method further includes: using an infrared sensor to collect original infrared images in the tunnel, and using a visible light sensor to collect original visible light images in the tunnel; the multi-source information fusion target detection network further includes a first single-stage two-path feature extraction and fusion network structure; preprocessing the original infrared image and the original visible light image respectively to generate an infrared input image and a visible light input image; and extracting and fusing features of the infrared input image and the visible light input image through the first single-stage two-path feature extraction and fusion network structure to generate a visual fusion feature map.

[0007] In some optional implementations, the method further includes: using a laser radar to collect three-dimensional point cloud data of a first radar in a tunnel, and using a millimeter-wave radar to collect three-dimensional point cloud data of a second radar in a tunnel; the multi-source information fusion target detection network further includes a second single-stage two-path feature extraction and fusion network structure; pre-processing the first radar three-dimensional point cloud data and the second radar three-dimensional point cloud data respectively to generate a first radar input image and a second radar input image; using the second single-stage two-path feature extraction and fusion network structure to extract features from and fuse the first radar input image and the second radar input image to generate a radar fusion feature map.

[0008] In some optional implementations, a pre-acquired visual input image and a radar input image are fused through a multi-source information fusion target detection network to generate a fusion feature map, including: inputting the above-mentioned visual fusion feature map and the above-mentioned radar fusion feature map into the above-mentioned feature extraction fusion network; and adaptively weighted fusion of the above-mentioned visual fusion feature map and the above-mentioned radar fusion feature map through the above-mentioned single-stage two-way feature extraction and fusion module in the above-mentioned feature extraction fusion network to generate a fusion feature map.

[0009] In some optional implementations, the method further includes: based on the relative position and posture between the radar sensor and the visual sensor, performing sensor calibration to obtain calibration images of multiple perspectives; the radar sensor is used to collect radar three-dimensional point cloud data in the tunnel; based on the calibration images of the multiple perspectives, converting the coordinates of the radar three-dimensional point cloud data to the coordinate system of the visual sensor and projecting them; mapping the projected point cloud data to the image coordinate system of the visual sensor to generate a three-channel two-dimensional image; the three-channel two-dimensional image includes the horizontal and vertical coordinate values ​​of the radar three-dimensional point cloud data in the two-dimensional coordinate system and the three-channel values ​​of the points corresponding to the coordinate values; the three-channel values ​​include the number of points, average height, and reflection intensity at the positions corresponding to the coordinate values.

[0010] In some optional implementations, the tunnel target prediction network includes a prediction network dynamically designed based on the actual task to obtain an output dimension corresponding to the actual task; the prediction network includes: a detection head, a category prediction branch, and a position prediction branch; the detection head is used to process and resize the features of the output target; the category prediction branch is used to output the category probability of the target; and the position prediction branch is used to output the bounding box information of the target.

[0011] In the second aspect, an embodiment of the present invention provides a tunnel target detection and early warning system based on multi-source information fusion, the system comprising: a multi-source information fusion target detection module, used to fuse a pre-acquired visual input image and a radar input image through a multi-source information fusion target detection network to generate a fusion feature map; the multi-source information fusion target detection network comprises a feature extraction fusion network and a tunnel target prediction network; the feature extraction fusion network comprises a single-stage dual-path feature extraction and fusion module; a tunnel target detection module, used to predict and output the position and category of the target in the tunnel based on the fusion feature map through the tunnel target prediction network; an early warning module, used to determine the abnormal situation in the tunnel and issue an early warning based on the predicted output position and category of the target in the tunnel.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of any one of the methods described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute any method described in the first aspect.

[0014] The present invention provides a method and system for detecting and warning targets in tunnels based on multi-source information fusion. The method comprises: firstly, a pre-acquired visual input image and a radar input image are fused through a multi-source information fusion target detection network to generate a fusion feature map, wherein the multi-source information fusion target detection network comprises a feature extraction fusion network and a tunnel target prediction network, wherein the feature extraction fusion network comprises a single-stage dual-path feature extraction and fusion module; then, based on the fusion feature map, the position and category of the target in the tunnel are predicted and output through the tunnel target prediction network, and then the abnormal situation in the tunnel is determined and a warning is issued. The method solves the technical problems of limited application scope, low accuracy and poor stability of target detection in tunnels in the prior art, and achieves the technical effect of expanding the application scope and improving detection accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic diagram of a flow chart of a method for detecting and warning targets in a tunnel based on multi-source information fusion provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the structure of a depthwise convolution and a pointwise convolution provided in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a standard convolution structure and a depth-separable convolution structure provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of a mobilenet network structure provided in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a single-stage dual-path feature extraction fusion network structure provided by an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of a multi-source information fusion target detection network structure provided by an embodiment of the present invention;

[0022] Figure 7 A schematic diagram of the structure of a multi-spectral feature fusion module provided in an embodiment of the present invention;

[0023] Figure 8A schematic diagram of the structure of a tunnel target detection and early warning system based on multi-source information fusion provided by an embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Some embodiments of the present invention are described in detail below in conjunction with the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] Since the structure of the tunnel is usually closed, the natural lighting conditions are extremely poor, and most tunnels do not have a complete lighting system. Therefore, in a low-light or no-light tunnel environment, target detection and monitoring warning for pedestrians, motor vehicles and non-motor vehicles has always been a thorny problem faced by traffic control departments. In the prior art, radar sensor detection, visible light sensor, infrared sensor and other methods are generally used to achieve target detection and warning in the tunnel. However, the existing detection methods have limitations in the use of tunnel environments. Radar sensors are easily affected by complex reflective environments. In dense traffic in tunnels, multi-target reflection signals will interfere with each other, resulting in a decrease in the accuracy of target detection positions; the lighting conditions in the tunnel are complex, the light is dim, the image quality is poor, and strong light reflections and other lighting changes at the entrance and exit will affect the quality of the pictures collected by the visible light sensor; infrared sensors are based on thermal radiation imaging, and the temperature difference between the target and the background directly affects the detection effect. They are easily affected by factors such as ambient temperature and humidity, which may lead to detection errors.

[0029] It can be seen that a single type of sensor has disadvantages and problems in the application of tunnel target detection. That is to say, the existing technology has technical problems such as limited application scope of target detection in tunnels, low accuracy and poor stability.

[0030] Based on this, an embodiment of the present invention provides a method and system for early warning of target detection in a tunnel based on multi-source information fusion to alleviate the technical problems existing in the prior art of limited application scope, low accuracy and poor stability of target detection in a tunnel.

[0031] To facilitate understanding of this embodiment, firstly, a method for detecting and warning targets in a tunnel based on multi-source information fusion disclosed in an embodiment of the present invention is described in detail. Figure 1 The flowchart of a method for detecting and warning a target in a tunnel based on multi-source information fusion is shown. The method can be executed by an electronic device and mainly includes the following steps S110 to S130:

[0032] S110: fusing the pre-acquired visual input image and the radar input image through a multi-source information fusion target detection network to generate a fusion feature map; the multi-source information fusion target detection network includes a feature extraction fusion network and a tunnel target prediction network; the feature extraction fusion network includes a single-stage dual-path feature extraction and fusion module;

[0033] The feature extraction network is built based on the dual-path structure of MobileNet, which is an efficient and easy-to-implement lightweight neural network whose basic unit is the depthwise separable convolution. The depthwise separable convolution can be decomposed into two smaller operations: depthwise convolution (DW convolution) and pointwise convolution (PW convolution), such as Figure 2 As shown in Figure 2. The design of MobileNet includes a standard convolutional structure and a depth-separable convolutional structure, as shown in Figure 2. Figure 3 As shown in Figure 1. The computational complexity of depth-separable convolution is much smaller than that of standard convolution. Through depth-separable convolution, the number of parameters and computational complexity of MobileNet are significantly reduced (see the MobileNet network structure for details). Figure 4 MobileNet is able to perform inference quickly on devices with low computing power while maintaining high accuracy.

[0034] In order to improve the real-time performance of the detector, effectively reduce the computational complexity of the algorithm and have relatively good feature extraction capabilities, the feature extraction network is constructed based on the dual-path structure of MobileNet, as follows Figure 5 According to the different spatial sizes of the feature maps, MobileNet is divided into 5 levels (stage1 to stage5), and the downsampling rate of the input image corresponding to the output feature map of each level is 2l , l∈{1,2,...,5}. In order to preserve the spatial and semantic information of the feature map as much as possible, the feature maps of the two parallel backbone networks at stage3 and stage4 are first upsampled by deconvolution operations to obtain feature maps F1 and F2; then F1 and F2 at each level are weighted fused through the feature fusion module Fusion, and the fused feature is represented as F out ; Finally, the F of stage3 and stage4 of the backbone network out Concatenate to get the fused feature map.

[0035] In one embodiment, before the above step S110, the method further includes:

[0036] (S010) using an infrared sensor and a radar to collect original infrared images and radar three-dimensional point cloud data in the tunnel respectively;

[0037] (S020) Preprocessing the original infrared image and the radar three-dimensional point cloud data to obtain a visual input image and radar three-dimensional point cloud data respectively.

[0038] Then, through the feature extraction and fusion network in the multi-source information fusion target detection network, the visual input image and radar 3D point cloud data obtained in the above steps are subjected to feature extraction and fusion to generate a fusion feature map.

[0039] Due to the complex reflection environment in the tunnel, radar sensors are prone to false detection. In dense traffic, the reflected signals of multiple targets will interfere with each other, resulting in a decrease in the accuracy of target detection position. Although lidar can generate high-resolution three-dimensional point clouds, it will be interfered by smoke and dust. Millimeter-wave radars usually have a small field of view and low spatial resolution. The generated point cloud data is sparse, making it difficult to distinguish different types of targets (such as vehicles, pedestrians, and bicycles), and the classification performance is poor.

[0040] The lighting conditions in the tunnel are complex, with dim light, poor image quality, strong light reflections and other lighting changes at the entrance and exit, which will affect the use of infrared light, visible light or a fusion of the two. Infrared sensors are based on thermal radiation imaging. The temperature difference between the target and the background directly affects the detection effect. They are easily affected by factors such as ambient temperature and humidity, which may lead to detection errors. In the case of small temperature differences (such as in tunnels in summer), it is difficult to detect targets, and it is difficult to accurately identify and classify targets.

[0041] It can be seen that a single type of sensor has disadvantages and problems in tunnel target detection applications. LiDAR provides high-resolution point clouds, and millimeter-wave radar provides speed information and penetration capabilities. The combination of the two improves the accuracy and robustness of detection. The all-weather working ability of millimeter-wave radar makes up for the shortcomings of LiDAR in harsh environments. Radar sensors can make up for the influence of complex lighting on image sensors, and image sensors can make up for the influence of signal reflection and coverage in the tunnel of radar sensors.

[0042] Through the application of multi-sensor fusion and intelligent algorithms, the disadvantages and problems of a single sensor in target detection in tunnels can be effectively solved, and a more reliable and efficient target detection system can be achieved.

[0043] Based on this, the above multi-source information fusion target detection network can also use two parallel backbone networks tunnel1 and tunnel2 to extract visible light and infrared source images respectively, and perform weighted fusion at two different levels (stage3 and stage4) of the backbone network to form a visual fusion feature map. Similarly, the two-dimensional image of the laser radar and the two-dimensional image of the millimeter-wave radar pass through the backbone networks tunnel1 and tunnel2 respectively to form a radar fusion feature map. In other words, the multi-source information fusion target detection network includes two single-stage two-way feature extraction and fusion networks and a tunnel target prediction network (see Figure 6 ), making full use of the complementary advantages of visible light, infrared, LiDAR and millimeter wave radar to achieve high-precision, all-weather target detection.

[0044] In one embodiment, before the above step S110, the method may further include:

[0045] (S030) using an infrared sensor to collect an original infrared image in the tunnel, and using a visible light sensor to collect an original visible light image in the tunnel; the multi-source information fusion target detection network also includes a first single-stage dual-path feature extraction fusion network structure;

[0046] (S040) preprocessing the original infrared image and the original visible light image respectively to generate an infrared input image and a visible light input image;

[0047] (S050) Through the first single-stage dual-path feature extraction and fusion network structure, feature extraction and fusion are performed on the infrared input image and the visible light input image to generate a visual fusion feature map.

[0048] (S060) using a laser radar to collect first radar three-dimensional point cloud data in the tunnel, and using a millimeter wave radar to collect second radar three-dimensional point cloud data in the tunnel; the multi-source information fusion target detection network also includes a second single-stage dual-path feature extraction fusion network structure;

[0049] (S070) preprocessing the first radar three-dimensional point cloud data and the second radar three-dimensional point cloud data respectively to generate a first radar input image and a second radar input image;

[0050] (S080) Using a second single-stage dual-path feature extraction and fusion network structure, feature extraction and fusion are performed on the first radar input image and the second radar input image to generate a radar fusion feature map.

[0051] Accordingly, in one embodiment, the above S110 fuses the pre-acquired visual input image and the radar input image through a multi-source information fusion target detection network to generate a fusion feature map, including:

[0052] (S111) inputting the visual fusion feature map and the radar fusion feature map into a feature extraction fusion network;

[0053] (S112) Adaptively weighted fusion is performed on the visual fusion feature map and the radar fusion feature map through a single-stage dual-path feature extraction and fusion module in the feature extraction and fusion network to generate a fusion feature map.

[0054] S120: Based on the fused feature map, the position and category of the target in the output tunnel are predicted through the tunnel target prediction network;

[0055] That is to say, the visual fusion feature map and the radar fusion feature map are formed into a radar-visual fusion feature map through the Fusion module, and then through the prediction network to obtain the position and category of the target.

[0056] In one embodiment, the tunnel target prediction network includes a prediction network dynamically designed based on the actual task to obtain an output dimension corresponding to the actual task; wherein the prediction network includes: a detection head, a category prediction branch, and a position prediction branch; the detection head is used to process and resize the features of the output target; the category prediction branch is used to output the category probability of the target; and the position prediction branch is used to output the bounding box information of the target.

[0057] As a specific example, MobileNet is usually used as the backbone of the feature extraction network, and is combined with other target detection frameworks and corresponding loss functions, such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once) to implement a complete target detection system. The design of the prediction network is relatively flexible. A specific output layer structure (including detection head, category prediction branch, and position prediction branch) is designed according to task requirements to obtain the specified output dimension (category probability and bounding box information). The detection head can set the convolution kernel size, dimension, step size, and number of different sizes as needed, and different padding methods to further process and resize the features.

[0058] Considering the output size M×N×C of the Lei Shi fusion feature map after the detection head and the number of network detection categories K, the category prediction branch is designed: a 1x1 convolutional layer is used to reduce the number of channels of the feature map to the number of categories K, and the output feature map size is M×N×(A×K), where A is the number of anchor boxes predicted for each feature point.

[0059] Perform a softmax operation on the category of each anchor box to generate a probability distribution; design a position prediction branch: use a 1x1 convolution layer to reduce the number of channels of the feature map to 4, representing the 4 regression parameters (x, y, w,) of the bounding box, and the output feature map size is M×N×(A×4).

[0060] S130: Based on the predicted position and category of the target in the tunnel, determine the abnormal situation in the tunnel and issue an early warning.

[0061] Among them, abnormal situations in the tunnel may include: vehicle breakdown, pedestrian detention, fire, etc. The sensor data collected in the time and space dimensions of each situation are structured and uploaded to the cloud. Based on further analysis of the data, abnormal situations are determined, such as: abnormal time for the target to pass through the tunnel (too long may cause personnel detention or vehicle failure; too short may cause speeding, etc.), abnormal image resolution (abnormal obstruction of the target, smoke or fog caused by fire, rain and snow weather causing low resolution, etc.). In addition, the early warning mechanism can be linked, and the monitoring system can be linked with the safety system in the tunnel (such as emergency broadcasting and lighting systems) to automatically take countermeasures to remind other vehicles, such as turning on the tunnel abnormality warning light.

[0062] The present invention provides a method for detecting and warning targets in tunnels based on multi-source information fusion, the method comprising: firstly, through a multi-source information fusion target detection network, a pre-acquired visual input image and a radar input image are fused to generate a fusion feature map, the multi-source information fusion target detection network comprises a feature extraction fusion network and a tunnel target prediction network, the feature extraction fusion network comprises a single-stage dual-path feature extraction and fusion module; then, based on the fusion feature map, the position and category of the target in the tunnel are predicted and output through the tunnel target prediction network, and then the abnormal situation in the tunnel is determined and a warning is issued. The method solves the technical problems of limited application scope, low accuracy and poor stability of target detection in tunnels in the prior art, and achieves the technical effect of expanding the application scope and improving detection accuracy and stability.

[0063] The tunnel target detection and early warning method based on multi-source information fusion proposed in the embodiment of the present invention can combine four sensors: visible light, infrared, laser radar, and millimeter wave radar, and use their respective advantages to make up for the shortcomings of a single sensor to improve the accuracy and robustness of target detection. In order to fuse the multi-source information obtained by different types of sensors, a series of processing is required for the multimodal data before fusion, including:

[0064] (1) Multimodal data alignment;

[0065] Position determination: Determine the relative position and attitude between the lidar and the camera.

[0066] Spatial alignment: Align the radar point cloud projection image with the visible light image to ensure that they are in the same spatial coordinate system.

[0067] Time synchronization: ensure that the two images are acquired at the same time point to avoid errors caused by time differences.

[0068] Infrared images: Temperature calibration is usually required to ensure consistency of temperature values ​​at different times and environments.

[0069] (2) Data preprocessing;

[0070] 3D point cloud data: Use statistical filtering, voxel filtering, RANSAC algorithm and other methods to remove noise points and outliers in 3D point cloud data.

[0071] Infrared image: Use Gaussian filtering or mean filter to remove high-frequency noise and random noise in infrared images.

[0072] Visible light images: In low light conditions, histogram equalization or adaptive histogram equalization can be used to enhance the contrast of visible light images.

[0073] (3) Converting 3D point cloud data into 2D images;

[0074] The sensor calibration method determines the data conversion parameters and formulas between the radar and the camera, and converts the point cloud data into a two-dimensional image. In a two-dimensional radar image, the pixel value at each (x, y) represents the number of points at that position, certain statistical characteristics (such as average height), and reflection intensity, forming a three-channel two-dimensional image corresponding to the camera image. This is suitable for computer vision tasks such as object detection and recognition.

[0075] That is, in one embodiment, the above method further includes:

[0076] (S001) Based on the relative position and posture between the radar sensor and the visual sensor, the sensor is calibrated to obtain calibration images of multiple perspectives; the radar sensor is used to collect radar three-dimensional point cloud data in the tunnel;

[0077] (S002) based on the calibration images of multiple perspectives, converting the coordinates of the radar three-dimensional point cloud data into the coordinate system of the visual sensor and projecting it;

[0078] (S003) Mapping the projected point cloud data to the image coordinate system of the visual sensor to generate a three-channel two-dimensional image; the three-channel two-dimensional image includes the horizontal and vertical coordinate values ​​of the radar three-dimensional point cloud data in the two-dimensional coordinate system and the three-channel values ​​of the points corresponding to the coordinate values; the three-channel values ​​include the number of points at the positions corresponding to the coordinate values, the average height, and the reflection intensity.

[0079] As a specific example, sensor calibration is performed to convert radar 3D point cloud data into 2D 3-channel images, which specifically includes the following steps:

[0080] Determine the relative position and posture between the lidar and the camera, use calibration tools such as calibration plates to capture calibration images from multiple perspectives, and obtain the extrinsic parameter matrix through multi-perspective geometric calculations Including the rotation matrix R and the translation matrix t.

[0081] The camera's intrinsic parameter matrix is ​​calculated using Zhang Zhengyou's calibration method and other methods.

[0082] Convert the 3D point cloud coordinates to the camera coordinate system P camera =RP LiDAR +t, that is

[0083]

[0084] Then the three-dimensional point in the camera coordinate system is projected onto the two-dimensional image plane Right now

[0085]

[0086] The projected point cloud information is mapped to the camera image coordinate system to generate a three-channel two-dimensional image.

[0087] In a two-dimensional radar image, the pixel value at each (u, v) represents the number of points at that location, certain statistical characteristics (such as average height), and reflection intensity, forming a three-channel two-dimensional image corresponding to the camera image.

[0088] The three-channel two-dimensional image generated by this method maintains the original point cloud information while converting it into a two-dimensional image form that is easy to process and analyze. This not only makes it easier to integrate other sensor data, but also makes full use of methods such as deep learning to process multimodal data, thereby improving the accuracy and robustness of target detection and recognition.

[0089] In addition, as a specific example, the feature extraction fusion network in the multi-source information fusion target detection network includes a single-stage dual-path feature extraction and fusion module; wherein the feature fusion module Fusion structure is as follows Figure 7 When fusing feature data, it is necessary to consider the contribution of each modal data to the effective information of the network. To achieve this, different weights are given to F1 and F2 in space.

[0090] The feature fusion module is used to adaptively weight the fusion of image features F1 and image features F2. The formula is as follows:

[0091]

[0092] Among them, F i,j and W i,j Respectively represent the feature vector and weight of the image feature map at (i, j), is the feature vector of the fused feature map at (i, j). The weight is defined as follows:

[0093]

[0094] in, and are the intermediate features of channel image 1 and channel image 2 respectively, and are converted into weights for the two modal features through the Softmax function. In the specific implementation, in order to utilize the local background information, a 3x3 convolution is used to calculate the and Thus, different weights are obtained for the two modal features at each position through learning.

[0095] The formulas of each layer of the fusion module are as follows:

[0096]

[0097] Among them, λ m is the intermediate feature, is the ReLU function, S is the Softmax function, f() represents the convolutional layer, F out For fusion features.

[0098] That is to say, the above-mentioned tunnel target detection and early warning method based on multi-source information fusion provided by the embodiment of the present invention designs a single-stage dual-path structure feature extraction and fusion solution based on the mobilenet network, which solves the technical problem of multi-sensor feature fusion extraction. By collecting labeled data, end-to-end model training, and performing reasoning verification.

[0099] A single-stage dual-path feature extraction fusion network structure and feature fusion module are proposed, which effectively reduce the computational complexity of the algorithm and have good multi-sensor feature extraction capabilities.

[0100] A method for converting radar 3D point cloud data into 2D three-channel images is proposed. Radar sensor calibration maps the projected point cloud information to the camera image coordinate system to generate a three-channel 2D image. In a 2D radar image, the pixel value at each (x, y) represents the number of points at that position, certain statistical characteristics (such as average height), and reflection intensity, forming a three-channel 2D image corresponding to the camera image.

[0101] A sensor feature fusion method is proposed, which considers the contribution of each modal data to the effective information of the network and assigns different weights in space.

[0102] In addition, an embodiment of the present invention further provides a tunnel target detection and early warning system based on multi-source information fusion, the system comprising:

[0103] A multi-source information fusion target detection module is used to fuse the pre-acquired visual input image and the radar input image through a multi-source information fusion target detection network to generate a fusion feature map; the multi-source information fusion target detection network includes a feature extraction fusion network and a tunnel target prediction network; the feature extraction fusion network includes a single-stage dual-path feature extraction and fusion module;

[0104] The tunnel target detection module is used to predict the location and category of the target in the tunnel based on the fused feature map through the tunnel target prediction network;

[0105] The early warning module is used to determine abnormal conditions in the tunnel and issue an early warning based on the location and category of the target in the tunnel output by the prediction.

[0106] As a specific example, combining Figure 8 The above-mentioned system may also include: an image acquisition module (visible light camera, infrared camera), a three-dimensional point cloud acquisition module (lidar, millimeter wave radar), a three-dimensional image conversion module, a decision output unit (for uploading structured information to the cloud) and an early warning module for safety early warning.

[0107] The tunnel target detection and early warning system based on multi-source information fusion designed by the present invention can be deployed on edge computing node devices to collect different sensor information of targets entering and exiting the tunnel, perform data processing and annotation, and train models.

[0108] In other words, building a tunnel target detection and early warning system based on multi-source information fusion solves the problem of false detection and missed detection of single sensor detection in special tunnel scenarios, and the system can be deployed to edge devices for early warning. Deploying model services to edge computing nodes avoids data return delays and saves bandwidth.

[0109] In addition, the system can also include: an anomaly detection module, which identifies abnormal situations in the tunnel (such as vehicle breakdown, stranded pedestrians, fire, etc.) and uploads structured data to the cloud; a linkage early warning mechanism, in which the monitoring system is linked with the safety system in the tunnel (such as emergency broadcasting, lighting system), and automatically takes countermeasures to alert other vehicles, such as turning on the tunnel abnormality warning light.

[0110] That is to say, the method and system for detecting and warning targets in tunnels based on multi-source information fusion described in the embodiment of the present invention proposes a multimodal data fusion solution to conduct more comprehensive analysis and decision-making to solve the problems of misdetection and missed detection of targets in tunnels. Combining four sensors, namely visible light, infrared, LiDAR, and millimeter-wave radar, the advantages of each sensor are used to make up for the shortcomings of a single sensor, thereby improving the accuracy and robustness of target detection; using advanced data fusion technology, the data of different sensors are aligned and fused spatially and feature-wise, making full use of multi-source information to improve detection and classification performance; applying deep learning algorithms, combining multi-source data for intelligent analysis and decision-making, and improving the system's adaptive ability and early warning response accuracy. Through the above method, the complementary advantages of visible light, infrared, LiDAR, and millimeter-wave radar are fully utilized to achieve high-precision, all-weather target detection. The embodiment of the present application also provides an electronic device, specifically, the electronic device includes a processor and a storage device; a computer program is stored on the storage device, and the computer program executes the method described in any of the above embodiments when it is run by the processor.

[0111] Fig. 9 A structural schematic diagram of an electronic device provided in an embodiment of the present application, the electronic device 400 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0112] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0113] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the process definition device disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0115] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0116] Corresponding to the above method, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above method.

[0117] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] It should be noted that similar numbers and letters represent similar items in the accompanying drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and warning targets in tunnels based on multi-source information fusion, characterized in that: include: Through the multi-source information fusion target detection network, the pre-acquired visual input image and radar input image are fused to generate a fusion feature map; The multi-source information fusion target detection network includes a feature extraction fusion network and a tunnel target prediction network; The feature extraction and fusion network includes a single-stage dual-path feature extraction and fusion module; Based on the fused feature map, the position and category of the target in the output tunnel are predicted through the tunnel target prediction network; Based on the predicted position and category of the target in the tunnel, abnormal conditions in the tunnel are determined and an early warning is issued.

2. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 1 is characterized in that: The method further comprises: The infrared sensor and radar are used to collect the original infrared image and radar 3D point cloud data in the tunnel respectively; The original infrared image and the radar three-dimensional point cloud data are preprocessed to obtain a visual input image and radar three-dimensional point cloud data respectively.

3. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 1 is characterized in that: The method further comprises: The original infrared image in the tunnel is collected by using an infrared sensor, and the original visible light image in the tunnel is collected by using a visible light sensor; the multi-source information fusion target detection network also includes a first single-stage dual-path feature extraction fusion network structure; Preprocessing the original infrared image and the original visible light image respectively to generate an infrared input image and a visible light input image; The infrared input image and the visible light input image are feature extracted and fused through the first single-stage dual-path feature extraction and fusion network structure to generate a visual fusion feature map.

4. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 3 is characterized in that: The method further comprises: The laser radar is used to collect the first radar three-dimensional point cloud data in the tunnel, and the millimeter wave radar is used to collect the second radar three-dimensional point cloud data in the tunnel; the multi-source information fusion target detection network also includes a second single-stage dual-path feature extraction fusion network structure; Preprocessing the first radar three-dimensional point cloud data and the second radar three-dimensional point cloud data respectively to generate a first radar input image and a second radar input image; The second single-stage two-way feature extraction and fusion network structure is used to extract and fuse features of the first radar input image and the second radar input image to generate a radar fusion feature map.

5. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 4 is characterized in that: Through the multi-source information fusion target detection network, the pre-acquired visual input image and radar input image are fused to generate a fusion feature map, including: Inputting the visual fusion feature map and the radar fusion feature map into the feature extraction fusion network; The single-stage two-way feature extraction and fusion module in the feature extraction and fusion network is used to perform adaptive weighted fusion on the visual fusion feature map and the radar fusion feature map to generate a fusion feature map.

6. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 1, characterized in that: The method further comprises: Based on the relative position and posture between the radar sensor and the visual sensor, the sensor is calibrated to obtain calibration images of multiple perspectives; the radar sensor is used to collect radar three-dimensional point cloud data in the tunnel; Based on the calibration images of the multiple perspectives, the coordinates of the radar three-dimensional point cloud data are converted into the coordinate system of the visual sensor and projected; The projected point cloud data is mapped to the image coordinate system of the visual sensor to generate a three-channel two-dimensional image; the three-channel two-dimensional image includes the horizontal and vertical coordinate values ​​of the radar three-dimensional point cloud data in the two-dimensional coordinate system and the three-channel values ​​of the points corresponding to the coordinate values; the three-channel values ​​include the number of points, average height, and reflection intensity at the positions corresponding to the coordinate values.

7. The method for detecting and warning targets in tunnels based on multi-source information fusion according to claim 1, characterized in that: The tunnel target prediction network includes a prediction network dynamically designed based on the actual task to obtain an output dimension corresponding to the actual task; The prediction network includes: a detection head, a category prediction branch, and a position prediction branch; the detection head is used to process and resize the features of the output target; the category prediction branch is used to output the category probability of the target; and the position prediction branch is used to output the bounding box information of the target.

8. A tunnel target detection and early warning system based on multi-source information fusion, characterized in that: include: A multi-source information fusion target detection module is used to fuse the pre-acquired visual input image and the radar input image through a multi-source information fusion target detection network to generate a fusion feature map; The multi-source information fusion target detection network includes a feature extraction fusion network and a tunnel target prediction network; The feature extraction and fusion network includes a single-stage dual-path feature extraction and fusion module; A tunnel target detection module, configured to predict and output the position and category of the target in the tunnel through the tunnel target prediction network based on the fused feature map; The early warning module is used to determine abnormal conditions in the tunnel and issue an early warning based on the position and category of the target in the tunnel output by the prediction.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute the method according to any one of claims 1 to 7.

Images