Multi-sensor feature level fusion target detection method and system

Abstract

The invention discloses a multi-sensor feature level fusion target detection method and system, and the method comprises the steps: continuously collecting spatial data information and image data information in a vehicle operation process; superposing first data information collected at a plurality of previous historical moments and the current moment, extracting first feature information based on the superposed first data information, and extracting second feature information based on second data information collected at the current moment; performing feature transformation on the first feature information and the second feature information to obtain a first feature aerial view and a second feature aerial view, and performing sampling, channel splicing and feature fusion on the first feature aerial view and the second feature aerial view to obtain a final fused feature aerial view; and performing analysis according to the final fusion feature aerial view to obtain a target detection result. According to the invention, the detection precision and the detection stability of a long-distance target can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent rail, locomotive, subway and passenger car intelligent driving, and particularly to a multi-sensor feature-level fusion target detection method and system. Background Art

[0002] With the development of intelligent driving technology, vehicles have higher and higher requirements for environmental perception. They hope to be able to perceive the target in front of the vehicle as early as possible and respond as soon as possible under different external environmental conditions. Object detection is the core part of the vehicle's intelligent driving system. It is responsible for collecting data from sensors (such as cameras, lidar, ultrasonic sensors, etc.), processing this data into actual environmental information, and obtaining the target's position, size, heading, and category.

[0003] Due to the variety of sensor types, multi-sensor data fusion is often used for target detection. Multi-sensor fusion methods can be divided into three types: (1) Decision-level fusion: feature extraction and decision making are performed on the information collected by each sensor, and then all decision information is fused, relevant processing is performed and the final decision is made; (2) Feature-level fusion: first, the data collected by each sensor is sent to the corresponding feature extraction algorithm, feature extraction is performed on the data, and then the extracted features are fused to obtain the fused detection result; (3) Data-level fusion: first, the raw data collected from the same sensor or different sensors are fused by data alignment, data association and other operations, and then the fused data is pre-processed and sent to the feature extraction algorithm for feature extraction, and finally the final detection result is obtained based on the extracted features. Decision-level fusion may cause one or more sensors to lose the target at a long distance, resulting in insufficient detection distance. Data set fusion is easily affected by sensor noise and calibration parameters between multiple sensors. Therefore, feature-level fusion is currently used to improve detection distance and accuracy.

[0004] However, in the existing feature-level fusion method, the point cloud data collected by the lidar is sparse at long distances, making it difficult to detect small targets at long distances and the detection accuracy is low. Summary of the invention

[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a multi-sensor feature-level fusion target detection method and system with high long-distance detection accuracy and low cost are provided to achieve accurate detection of small targets at a long distance in front of the vehicle.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: Step S1: collecting data; during the operation of the vehicle, continuously collecting first data information and second data information, wherein the first data information is spatial data information, and the second data information is image data information; Step S2: Processing the data to obtain feature information; superimposing the first data information collected at the previous several historical moments with the current moment, extracting the first feature information based on the superimposed first data information, and extracting the second feature information based on the second data information collected at the current moment; Step S3: feature information transformation and fusion: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view, and performing sampling, channel splicing and feature fusion on the first feature bird's-eye view and the second feature bird's-eye view to obtain a final fused feature bird's-eye view; Step S4: Target detection: Analyzing the final fusion feature bird's-eye view to obtain a target detection result.

[0007] Furthermore, the step S3 comprises: Step S31: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view of original sizes; Step S32: downsampling the first feature bird's-eye view and the second feature bird's-eye view of the original size to obtain a plurality of groups of first feature bird's-eye view and second feature bird's-eye view of different sizes, wherein each group of the first feature bird's-eye view and the second feature bird's-eye view have the same size and are smaller than the original size; performing channel stitching and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view of the same size including the original size, to obtain a plurality of corresponding fused feature bird's-eye view; Step S33: upsample all fused feature bird's-eye view images whose sizes are smaller than the original size to obtain a fused feature bird's-eye view image with the same size as the original size, and perform channel stitching and feature fusion with the fused feature bird's-eye view image of the original size to obtain a final fused feature bird's-eye view image.

[0008] Further, the step S32 includes: The sizes of the first feature bird's-eye view and the second feature bird's-eye view are reduced to their original sizes by an m×m layer convolutional feature encoder. , obtaining m groups of first characteristic bird's-eye views and second characteristic bird's-eye views, wherein the first characteristic bird's-eye views and the second characteristic bird's-eye views in each group are of the same size; Perform channel splicing and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view to obtain m fused feature bird's-eye view images; Among them, the first feature bird's-eye view is a lidar feature bird's-eye view, and the second feature bird's-eye view is an image feature bird's-eye view.

[0009] Furthermore, performing channel stitching and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view includes: The first characteristic bird's-eye view and the second characteristic bird's-eye view are superimposed on the channel to obtain the channel splicing bird's-eye view. The calculation expression is: ; In the above formula, represents the feature map height, represents the feature map width, The number of channels representing the first feature bird's-eye view, The number of channels representing the second feature bird's-eye view; A convolution-based feature encoder performs feature encoding on the channel splicing bird's-eye view to obtain a fused feature bird's-eye view.

[0010] Furthermore, the step S33 includes: The m-1 fused feature bird's-eye view images whose sizes are smaller than the original size are upsampled respectively to obtain m-1 fused feature bird's-eye view images with the same size as the original size, and channel splicing and feature fusion are performed with the fused feature bird's-eye view image of the original size to obtain the final fused feature bird's-eye view.

[0011] Furthermore, in step S2, the first data information collected at the previous several historical moments is superimposed with the first data information collected at the current moment, and the first feature information is extracted based on the superimposed first data information, including: Acquire first data information collected at previous historical moments, wherein the first data information is point cloud data information collected by a plurality of the first sensors, the first sensors including a long-range laser radar and a short-range laser radar; According to the vehicle position and posture represented by the historical frame point cloud data information corresponding to each historical moment and the current frame point cloud data information corresponding to the current moment, the historical frame point cloud data information is converted to the current frame point cloud data information coordinate system and superimposed with the point cloud data information at the current moment; The superimposed point cloud data information is spatially divided, feature encoded and feature extracted to obtain the first feature information.

[0012] Furthermore, the historical frame point cloud data information is converted to the current frame point cloud data information coordinate system according to the vehicle position and posture represented by the historical frame point cloud data information corresponding to each historical moment and the current frame point cloud data information corresponding to the current moment, and superimposed with the point cloud data information at the current moment, including: According to the vehicle posture transformation relationship from each historical moment to the current moment, the point cloud data information of the historical moment is converted to the current moment t. The calculation expression is: , ; In the above formula, is the n×n rotation matrix from ti frame to t frame, is the n×1 translation vector, and Represents the point cloud data information of t frame and ti frame, Indicates the vehicle position and posture in the current frame, Indicates the vehicle position and posture in the ti frame; The converted n historical frame point cloud data information is superimposed with the current frame point cloud data information, and the calculation expression is: ; In the above formula, They are the point cloud data information from the first historical moment to the nth historical moment after conversion, It is the superimposed multi-frame point cloud data information.

[0013] Furthermore, the step of performing spatial division, feature coding and feature extraction on the superimposed point cloud data information to obtain the first feature information includes: Dividing the point cloud data information space into grids based on the columnar method, wherein the grid size of the area outside the preset distance is larger than the grid size of the area within the preset distance; Each point cloud data information in the column is encoded, and features of each point cloud data information are extracted through a multi-layer perceptron to obtain high-dimensional features, and then maximum pooling is used to obtain features of each column, thereby obtaining the first feature information.

[0014] Furthermore, in step S2, extracting the second feature information based on the second data information collected at the current moment includes: A residual network is used as a feature extraction network to perform feature extraction on second data information collected by multiple second sensors, wherein the second sensors include a telephoto camera and a short-focus camera, and the second data information is image data information, thereby obtaining two-dimensional feature information of the telephoto camera and the short-focus camera as the second feature information.

[0015] Furthermore, the time synchronization operation of each first sensor and the second sensor is also included between step S1 and step S2, specifically: The clock synchronization source sends a synchronization message, and the first sensor and the second sensor receive and record the receiving time; The clock synchronization source sends a follow-up message to provide the precise sending time of the synchronization message; The first sensor and the second sensor send a delay request message and record the sending time; The clock synchronization source receives the delay request message and responds with a delay response message, providing a receiving timestamp; The first sensor and the second sensor adjust their own clocks according to the network delay to synchronize with the clock synchronization source.

[0016] The present invention also provides a multi-sensor feature-level fusion target detection system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured with the multi-sensor feature-level fusion target detection method.

[0017] Compared with the prior art, the advantages of the present invention are: The present invention can make the spatial data information at the current moment more accurate by superimposing the collected spatial data information at multiple moments, which is conducive to the subsequent accurate detection of distant small targets through spatial data information; the final fused feature bird's-eye view is obtained by sampling, channel splicing and feature fusion to perform target detection, which can avoid the problem of loss of distant and small targets due to the decrease in resolution of a single bird's-eye view in the feature-level fusion process, and further improve the detection accuracy and stability of distant targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of a multi-sensor feature-level fusion target detection method according to an embodiment of the present invention.

[0019] Figure 2 It is a side view of the installation position and coverage range of the first sensor and the second sensor in an embodiment of the present invention.

[0020] Figure 3 1 is a top view of the installation position and coverage range of the first sensor and the second sensor in an embodiment of the present invention.

[0021] Figure 4 This is a flow chart of time synchronization between a first sensor and a second sensor in an embodiment of the present invention.

[0022] Figure 5 The figure is a schematic diagram of a specific process of a multi-sensor feature-level fusion target detection method according to an embodiment of the present invention.

[0023] Figure 6 It is a schematic diagram of channel stitching of different types of feature bird's-eye views in an embodiment of the present invention.

[0024] Figure 7 The flowchart of multi-scale feature fusion of different types of feature bird's-eye views in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a multi-sensor feature-level fusion target detection method, including: Step S1: collecting data; during the operation of the vehicle, continuously collecting first data information and second data information, wherein the first data information is spatial data information, and the second data information is image data information; Step S2: Processing the data to obtain feature information; superimposing the first data information collected at the previous several historical moments with the current moment, extracting the first feature information based on the superimposed first data information, and extracting the second feature information based on the second data information collected at the current moment; Step S3: feature information transformation and fusion: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view, and performing sampling, channel splicing and feature fusion on the first feature bird's-eye view and the second feature bird's-eye view to obtain a final fused feature bird's-eye view; Step S4: Target detection: Target detection results are obtained based on the final fusion feature bird's-eye view analysis.

[0027] It can be understood that this embodiment can make the spatial data information at the current moment more accurate by superimposing the collected spatial data information at multiple moments, which is conducive to the subsequent accurate detection of small targets at a long distance through spatial data information; the final fused feature bird's-eye view is obtained by sampling, channel splicing and feature fusion to perform target detection, which can avoid the problem of loss of long-distance and small targets due to the decrease in resolution of a single bird's-eye view in the feature-level fusion process, and further improve the detection accuracy and stability of long-distance targets.

[0028] In a specific application embodiment, the first data information and the second data information are collected by one or more first sensors and second sensors arranged on the vehicle; wherein the first sensor is used to collect spatial data information, and the second sensor is used to collect image data information. First, the first sensor and the second sensor are described. The first sensor is preferably a laser radar, and the second sensor is preferably a camera. Figure 2 and Figure 3The figure shows the side view and top view of the sensor installation position and coverage range, where the dotted line is the camera coverage range and the solid line is the laser radar coverage range, which ensures that the medium and short distances have a large detection range and a long detection distance. It is preferred to use four sensors to ensure that both short and long distances can be covered, namely, a long-distance laser radar, a short-distance laser radar, a telephoto camera and a short-focus camera. The long-distance laser radar and the telephoto camera have the characteristics of a small field of view and a long detection distance, and the short-distance laser radar and the short-focus camera have the characteristics of a large field of view and a short detection distance. In order to obtain a longer detection range, it is preferred to install the long-distance laser radar and the telephoto camera on the top of the front side of the vehicle or the top of the center line; at the same time, in order to ensure a smaller blind spot and a larger detection range, it is preferred to install the short-distance laser radar and the short-focus camera in the middle of the front side of the vehicle or the middle of the center line. It can be understood that by arranging multiple different types of sensors at different positions, the problems of poor noise resistance, short detection distance and low detection accuracy of a single sensor can be effectively improved, and the stability and accuracy of the detection can be improved.

[0029] In this embodiment, the time synchronization operation of each first sensor and second sensor is also included between step S1 and step S2, specifically: The clock synchronization source sends a synchronization message, and the first sensor and the second sensor receive and record the receiving time; The clock synchronization source sends a follow-up message to provide the precise sending time of the synchronization message; The first sensor and the second sensor send a delay request message and record the sending time; The clock synchronization source receives the delay request message and responds with a delay response message, providing a receiving timestamp; The first sensor and the second sensor adjust their own clocks according to the network delay to be synchronized with the clock synchronization source.

[0030] Specifically, please refer to Figure 4, long and short laser radars and long and short focal length cameras are connected to the switch via Ethernet. The clock synchronization source inputs the synchronization signal into the switch via Ethernet to give time to each sensor, and the computing platform receives the synchronized sensor data for feature-level fusion target detection. Therefore, it is necessary to use PTP to synchronize the time of multiple sensors of the same or different types to ensure that the time of the acquired raw data is consistent. The synchronization process includes four parts: synchronization stage: the clock synchronization source sends a synchronization message, and the laser radar and camera receive and record the receiving time; the clock synchronization source sends a follow-up message to provide the precise sending time of the synchronization message. Delay measurement stage: the laser radar and camera send a delay request message and record the sending time; the clock synchronization source receives the delay request message, and responds to the delay response message to provide the receiving timestamp. Clock adjustment: the laser radar and camera calculate the network delay and adjust their own clocks to synchronize with the master clock. Trigger stage: the camera is triggered to expose when the laser radar beam scans the front of the vehicle (perpendicular to the vehicle).

[0031] Before fusing multiple sensors, it is also necessary to obtain the spatial transformation relationship between the coordinate systems of each sensor and the vehicle body. By combining the laser radar with the vehicle position information of the inertial navigation system, the coordinate information of multiple frames of the same marker is extracted to calculate the position transformation relationship between the laser radar and the vehicle body; the position and attitude relationship between the two sensors is determined by aligning the feature points in the same scene captured by the laser radar and the camera; based on the characteristics of the vehicle's driving path, the calibration parameters are corrected online through the vehicle attitude information and the external environment characteristics.

[0032] Among them, the calculation steps of the spatial transformation relationship are as follows: Step 1, extract the same marker in the long-range and short-range lidars in different directions in multiple frames, and calculate the spatial transformation relationship between the lidar and the vehicle body; Step 2, project the lidar data into the image, and adjust the lidar projection point and the image pixel point to overlap to obtain the spatial transformation relationship between the lidar and the camera; Step 3, combine the above two spatial transformation relationships to obtain the spatial transformation relationship between all sensors and the vehicle body.

[0033] In step S2 of this embodiment, the first data information collected at the previous several historical moments is superimposed with the first data information collected at the current moment, and the first feature information is extracted based on the superimposed first data information, including: Acquire first data information collected at previous historical moments, wherein the first data information is point cloud data information collected by a plurality of first sensors, the first sensors including a long-range laser radar and a short-range laser radar; According to the vehicle position and posture represented by the historical frame point cloud data information corresponding to each historical moment and the current frame point cloud data information corresponding to the current moment, the historical frame point cloud data information is converted to the current frame point cloud data information coordinate system and superimposed with the point cloud data information at the current moment; The superimposed point cloud data information is spatially divided, feature encoded and feature extracted to obtain the first feature information.

[0034] In this embodiment, according to the vehicle position and posture represented by the historical frame point cloud data information corresponding to each historical moment and the current frame point cloud data information corresponding to the current moment, the historical frame point cloud data information is converted to the current frame point cloud data information coordinate system and superimposed with the point cloud data information at the current moment, including: According to the vehicle posture transformation relationship from each historical moment to the current moment, the historical frame point cloud data information is converted to the current moment t. The calculation expression is: , ; (Formula 1) In the above formula, is the n×n rotation matrix from ti frame to t frame, is the n×1 translation vector, and Represents the point cloud data information of t frame and ti frame, Indicates the vehicle position and posture in the current frame, Indicates the vehicle position and posture in the ti frame; The converted n historical frame point cloud data information is superimposed with the current frame point cloud data information, and the calculation expression is: ; (Formula 2) In the above formula, They are the point cloud data information from the first historical moment to the nth historical moment after conversion, It is the superimposed multi-frame point cloud data information.

[0035] In this embodiment, the superimposed point cloud data information is spatially divided, feature encoded, and feature extracted to obtain first feature information, including: Dividing the point cloud data information space into grids based on the columnar method, wherein the grid size of the area outside the preset distance is larger than the grid size of the area within the preset distance; Each point cloud data information in the column is encoded, and the feature of each point cloud data information is extracted through a multi-layer perceptron to obtain high-dimensional features, and then the maximum pooling is used to obtain the features of each column, thereby obtaining the first feature information.

[0036] In step S2 of this embodiment, extracting the second feature information based on the second data information collected at the current moment includes: A residual network is used as a feature extraction network to perform feature extraction on second data information collected by multiple second sensors, wherein the second sensors include a telephoto camera and a short-focus camera, and the second data information is image data information, thereby obtaining two-dimensional feature information of the telephoto camera and the short-focus camera as the second feature information.

[0037] In the specific application examples, please refer to Figure 5 , since the laser radar point cloud is very sparse at long distances, in order to avoid the appearance of target smear in the point cloud and improve the detection accuracy of long-distance targets, it is preferred to transform the point cloud data of the three historical frames to the vehicle position and posture of the current frame according to the vehicle position and posture of the three historical frames, and then fuse them as the input network (that is, the value of n in Formula 1 is 3). According to the vehicle position and posture information of the three historical frames, the vehicle position and posture transformation relationship from each historical frame to the current frame can be calculated. According to the transformation relationship, the laser radar point cloud of the three historical frames can be transformed to the current frame position, and these multiple frames of data are superimposed as the input of the network: (Formula 3) in is the 3×3 rotation matrix from time ti to time t, is a 3×1 translation vector, Represents the vehicle position and attitude at time t. The point cloud at time ti can be converted to time t according to the rotation matrix and translation vector: (Formula 4) Superimpose the three historical frames with the current frame as the input of the feature extraction network: (Formula 5) Subsequently, a three-dimensional feature extraction network is used to extract features from the point cloud data after the fusion of long-distance and short-distance lidars: a column-based method is used to spatially divide the point cloud data and perform feature encoding. First, spatial division is performed: the detection range is set to filter the point cloud outside the range, and the point cloud is divided into a grid of a fixed size. The height of the grid is the height of the entire detection range to form a column. Due to the characteristics of lidar, which is close and sparse, it is preferred to set the long-distance grid size to twice the short-distance grid size of the area within the preset distance of 300m when dividing the grid for the long-distance area beyond the preset distance of 300m. For example, the grid size within 300 meters is 0.1m×0.1m, and the grid size outside 300 meters is 0.2×0.2m, so that each column can contain more point clouds. Then, feature encoding is performed: each point cloud in the column is encoded, including seven dimensions: three-dimensional coordinates, intensity, and relative three-dimensional coordinates relative to the center of the column (x, y, z, intensity, delta_x, delta_y, delta_z). Finally, feature extraction is performed: a multi-layer perceptron is used to extract features from each point to obtain high-dimensional features; the maximum pooling is used to obtain the features of each column, thereby obtaining the feature information of the laser radar point cloud data (i.e., the first feature information). It can be understood that the grid size outside the preset distance is larger than the grid size within the preset distance, which can include more point cloud data in a longer range, thereby improving the detection accuracy of distant targets.

[0038] For the image data information obtained by the telephoto camera and the short-focus camera, a two-dimensional feature extraction network is used to extract features of the long-focus and short-focus camera image data respectively. The specific method uses a residual network as the feature extraction network: the residual network is composed of multiple residual blocks, where the residual block includes a convolutional layer, a batch normalization layer, a ReLU activation function and a jump connection, and finally the two-dimensional features of the two cameras are obtained to obtain the second feature information.

[0039] In this embodiment, step S3 includes: Step S31: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view of the original size; specifically, mapping the columnar coordinates to the pseudo image plane according to the features of each column obtained, performing feature extraction through a two-dimensional feature extraction network to obtain a first feature bird's-eye view; estimating the depth distribution features of the second feature information through a depth estimation network to obtain a second feature bird's-eye view, and the second feature bird's-eye view is a feature bird's-eye view of the image data information; Step S32: downsampling the first feature bird's-eye view and the second feature bird's-eye view of the original size to obtain a plurality of groups of first feature bird's-eye view and second feature bird's-eye view of different sizes, wherein each group of first feature bird's-eye view and second feature bird's-eye view have the same size and are smaller than the original size; performing channel stitching and feature fusion on each group of first feature bird's-eye view and second feature bird's-eye view of the same size including the original size, to obtain a plurality of corresponding fused feature bird's-eye view; Step S33: upsample all fused feature bird's-eye view images whose sizes are smaller than the original size to obtain a fused feature bird's-eye view image with the same size as the original size, and perform channel stitching and feature fusion with the fused feature bird's-eye view image of the original size to obtain the final fused feature bird's-eye view image.

[0040] In the specific application examples, please continue to refer to Figure 5 , because the feature spaces of different types of sensors are different, they cannot be directly fused, so the features need to be transformed to the bird's-eye view and then fused. For the laser radar, the columnar features obtained above are mapped to the pseudo image plane according to the coordinates, and then the general two-dimensional feature extraction network is used to extract the features to obtain the laser radar feature bird's-eye view (i.e., the first feature bird's-eye view). For the camera, the image feature uses the depth estimation network to estimate the depth distribution features. According to the image feature information H×W×C obtained above (H and W represent the size of the image feature, and C is the feature dimension), D depths, i.e., H×W×D, are predicted for each pixel, which is called the cone point cloud; the cone point cloud is converted from the image coordinate system to the vehicle coordinate system using the internal and external parameters of the camera; the depth distribution H×W×C×D corresponding to each pixel is obtained by performing the outer product of the image feature and the predicted depth; finally, according to the coordinate mapping to the image, the image feature bird's-eye view (i.e., the second feature bird's-eye view) is extracted using the two-dimensional feature extraction network. It should be noted that in this embodiment, the point cloud encoding preferably uses Pillar Feature Net, and the image feature transformation preferably uses Lift Splat Shoot.

[0041] Please refer to Figure 6 After obtaining the lidar feature bird's-eye view and the image feature bird's-eye view, feature fusion can be performed. The specific operation is: the different feature bird's-eye view images of the two sensors are spliced ​​on the channel (that is, the feature maps are superimposed in the third dimension), and then the convolution-based feature encoder is used to encode the features to obtain the bird's-eye view fusion features, where the convolution feature encoder performs convolution block operations (convolution, batch normalization layer, ReLU activation function) on the spliced ​​feature maps.

[0042] In this embodiment, step S32 includes: The sizes of the first feature bird's-eye view and the second feature bird's-eye view are reduced to their original sizes by using an m×m layer of convolutional feature encoder. , obtaining m groups of first characteristic bird's-eye views and second characteristic bird's-eye views, wherein the first characteristic bird's-eye views and the second characteristic bird's-eye views in each group are of the same size; Perform channel splicing and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view to obtain m fused feature bird's-eye views; Among them, the first feature bird's-eye view is a lidar feature bird's-eye view, and the second feature bird's-eye view is an image feature bird's-eye view.

[0043] In this embodiment, channel splicing and feature fusion are performed on each group of first feature bird's-eye view images and second feature bird's-eye view images, including: The first characteristic bird's-eye view and the second characteristic bird's-eye view are superimposed on the channel to obtain the channel splicing bird's-eye view. The calculation expression is: ; (Formula 6) In the above formula, represents the feature map height, represents the feature map width, The number of channels representing the first feature bird's-eye view, Indicates the number of channels of the second characteristic bird's-eye view; the number of channels of the spliced ​​bird's-eye view is the sum of the number of channels of the first characteristic bird's-eye view and the second characteristic bird's-eye view; The convolution-based feature encoder performs feature encoding on the channel splicing bird's-eye view to obtain a fused feature bird's-eye view. Specifically, this can be achieved by using a feature encoder including a 3×3 convolution, a batch normalization layer, and a ReLU activation function.

[0044] In this embodiment, step S33 includes: The m-1 fused feature bird's-eye view images with a size smaller than the original size are upsampled respectively to obtain m-1 fused feature bird's-eye view images with the same size as the original size, and channel splicing and feature fusion are performed with the fused feature bird's-eye view image of the original size to obtain the final fused feature bird's-eye view image.

[0045] In a specific application embodiment, since the resolution of the feature map decreases when performing convolution in the convolutional feature encoder, resulting in the loss of long-distance and small targets, in order to improve the detection capability of long-distance targets and small targets, such as Figure 7 As shown in the figure, taking into account the system computing power and target detection accuracy, it is preferred to downsample the laser radar and image feature bird's-eye view by 2 and 4 times respectively. Specifically, downsampling is achieved through a convolutional feature encoder (including a 3×3 convolution layer, a batch normalization layer, and a nonlinear activation function ReLU), so that the size of the laser radar feature and the image feature bird's-eye view becomes 1 / 2 and 1 / 4 of the size of their original feature bird's-eye view. After downsampling is completed, according to Figure 6The operation shown in the figure performs channel stitching and feature fusion on the radar features and image features of the same size (channel stitching can adopt the stitching method shown in formula 6) to obtain the fused feature bird's-eye view Figure 1 (Original size), fused feature bird's eye view Figure 2 (1 / 2 of the original size), fusion feature bird's eye view Figure 3 (1 / 4 of the original size). Since the small-sized feature map has a larger receptive field corresponding to global information, and the large-sized feature map has a smaller receptive field corresponding to local information, and feature maps of different sizes cannot be spliced, in order to combine global and local information, it is necessary to upsample the small-sized feature map and splice it with the original-sized feature map, as follows: Figure 2 , fusion feature bird's eye view Figure 3 Perform 2x upsampling and 4x upsampling respectively (you can use 2 upsampling modules, use the nearest neighbor upsampling method, and set the scale factor to 2.0 for upsampling), and get the same bird's-eye view as the fusion feature Figure 1 The three fused feature bird's-eye views of the same size are then channel-joined to obtain multi-scale fused features, and finally the multi-scale fused features are fused using a convolutional feature encoder to obtain the final fused feature bird's-eye view for target detection. It can be understood that by downsampling to obtain bird's-eye views of different sizes and then upsampling to perform feature fusion to obtain the final fused feature bird's-eye view, compared to the existing method of using multi-scale feature fusion and then weighting, it can more accurately detect targets at long distances.

[0046] After obtaining the final fusion feature bird's-eye view, you can input the detection head to get the target detection result. The detection head uses the anchor-free detection head, which contains two branches: target classification and position regression. The classification branch predicts K channel heat maps, corresponding to K categories. The center position of each target generates a Gaussian distribution as the highlight area in the heat map. Each pixel value in the heat map represents the probability that the position is the center of the target. The position of the target center corresponds to the peak of the Gaussian distribution. The farther from the center, the lower the probability. The position regression branch regresses the position deviation, size, and heading angle of the target. Finally, the category, position, size, and heading angle of the target can be predicted. Specifically, the constrained neural network of the size loss (smoothL1) and the position center loss function (gaussianfocal loss) can directly predict the category, position, size, and heading angle of the target as the neural network prediction value.

[0047] It should be noted that, in the present invention, the point cloud feature encoding method is not limited to the cylindrical feature encoding method, and other encoding methods may also be used; the image feature extraction network, the image feature and bird's-eye view feature conversion method, the feature fusion method, the detection head type, the downsampling and upsampling algorithms can all be implemented using any existing mature method, and are not specifically limited in the present invention.

[0048] The present invention further provides a multi-sensor feature-level fusion target detection system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multi-sensor feature-level fusion target detection method.

[0049] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multi-sensor feature-level fusion target detection method, characterized in that: include: Step S1: collecting data; During the operation of the vehicle, first data information and second data information are continuously collected, wherein the first data information is spatial data information and the second data information is image data information; Step S2: Processing data to obtain feature information; Superimposing first data information collected at previous historical moments with the current moment, extracting first feature information based on the superimposed first data information, and extracting second feature information based on second data information collected at the current moment; Step S3: feature information transformation and fusion: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view, and performing sampling, channel splicing and feature fusion on the first feature bird's-eye view and the second feature bird's-eye view to obtain a final fused feature bird's-eye view; Step S4: Target detection: Analyzing the final fusion feature bird's-eye view to obtain a target detection result.

2. The multi-sensor feature-level fusion target detection method according to claim 1, characterized in that: The step S3 comprises: Step S31: performing feature transformation on the first feature information and the second feature information to obtain a first feature bird's-eye view and a second feature bird's-eye view of original sizes; Step S32: downsampling the first feature bird's-eye view and the second feature bird's-eye view of the original size to obtain a plurality of groups of first feature bird's-eye view and second feature bird's-eye view of different sizes, wherein each group of the first feature bird's-eye view and the second feature bird's-eye view have the same size and are smaller than the original size; performing channel stitching and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view of the same size including the original size, to obtain a plurality of corresponding fused feature bird's-eye view; Step S33: upsample all fused feature bird's-eye view images whose sizes are smaller than the original size to obtain a fused feature bird's-eye view image with the same size as the original size, and perform channel stitching and feature fusion with the fused feature bird's-eye view image of the original size to obtain a final fused feature bird's-eye view image.

3. The multi-sensor feature-level fusion target detection method according to claim 2, characterized in that: The step S32 comprises: The sizes of the first feature bird's-eye view and the second feature bird's-eye view are reduced to their original sizes by using an m×m layer of convolutional feature encoder. , obtaining m groups of first characteristic bird's-eye views and second characteristic bird's-eye views, wherein the first characteristic bird's-eye views and the second characteristic bird's-eye views in each group are of the same size; Perform channel splicing and feature fusion on each group of the first feature bird's-eye view and the second feature bird's-eye view to obtain m fused feature bird's-eye view images; Among them, the first feature bird's-eye view is a lidar feature bird's-eye view, and the second feature bird's-eye view is an image feature bird's-eye view.

4. The multi-sensor feature-level fusion target detection method according to claim 3, characterized in that: The channel stitching and feature fusion of each group of the first feature bird's-eye view and the second feature bird's-eye view include: The first characteristic bird's-eye view and the second characteristic bird's-eye view are superimposed on the channel to obtain the channel splicing bird's-eye view. The calculation expression is: ； In the above formula, represents the feature map height, represents the feature map width, The number of channels representing the first feature bird's-eye view, The number of channels representing the second feature bird's-eye view; A convolution-based feature encoder performs feature encoding on the channel splicing bird's-eye view to obtain a fused feature bird's-eye view.

5. The multi-sensor feature-level fusion target detection method according to claim 2, characterized in that: The step S33 comprises: The m-1 fused feature bird's-eye view images whose sizes are smaller than the original size are upsampled respectively to obtain m-1 fused feature bird's-eye view images with the same size as the original size, and channel splicing and feature fusion are performed with the fused feature bird's-eye view image of the original size to obtain the final fused feature bird's-eye view.

6. The multi-sensor feature-level fusion target detection method according to claim 1, characterized in that: In step S2, the first data information collected at the previous several historical moments is superimposed with the first data information collected at the current moment, and the first feature information is extracted based on the superimposed first data information, including: Acquire first data information collected at previous historical moments, wherein the first data information is point cloud data information collected by a plurality of the first sensors, the first sensors including a long-range laser radar and a short-range laser radar; According to the vehicle position and posture represented by the historical frame point cloud data information corresponding to each historical moment and the current frame point cloud data information corresponding to the current moment, the historical frame point cloud data information is converted to the current frame point cloud data information coordinate system and superimposed with the point cloud data information at the current moment; The superimposed point cloud data information is spatially divided, feature encoded and feature extracted to obtain the first feature information.

7. The multi-sensor feature-level fusion target detection method according to claim 6, characterized in that: The method converts the historical frame point cloud data information into the current frame point cloud data information coordinate system according to the historical frame point cloud data information corresponding to each historical moment and the vehicle position and posture represented by the current frame point cloud data information corresponding to the current moment, and superimposes the historical frame point cloud data information with the current moment point cloud data information, including: According to the vehicle posture transformation relationship from each historical moment to the current moment, the point cloud data information of the historical moment is converted to the current moment t. The calculation expression is: ， ； In the above formula, is the n×n rotation matrix from ti frame to t frame, is the n×1 translation vector, and Represents the point cloud data information of t frame and ti frame, Indicates the vehicle position and posture in the current frame, Indicates the vehicle position and posture in the ti frame; The converted n historical frame point cloud data information is superimposed with the current frame point cloud data information, and the calculation expression is: ； In the above formula, They are the point cloud data information from the first historical moment to the nth historical moment after conversion, It is the superimposed multi-frame point cloud data information.

8. The multi-sensor feature-level fusion target detection method according to claim 6, characterized in that: The step of performing spatial division, feature coding and feature extraction on the superimposed point cloud data information to obtain the first feature information includes: Dividing the point cloud data information space into grids based on the columnar method, wherein the grid size of the area outside the preset distance is larger than the grid size of the area within the preset distance; Each point cloud data information in the column is encoded, and features of each point cloud data information are extracted through a multi-layer perceptron to obtain high-dimensional features, and then maximum pooling is used to obtain features of each column, thereby obtaining the first feature information.

9. The multi-sensor feature-level fusion target detection method according to claim 1, characterized in that: In step S2, extracting the second feature information based on the second data information collected at the current moment includes: A residual network is used as a feature extraction network to perform feature extraction on second data information collected by multiple second sensors, wherein the second sensors include a telephoto camera and a short-focus camera, and the second data information is image data information, thereby obtaining two-dimensional feature information of the telephoto camera and the short-focus camera as the second feature information.

10. A multi-sensor feature-level fusion target detection system, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multi-sensor feature-level fusion target detection method according to any one of claims 1 to 9.

Images