Multi-sensor fusion road target detection method and device

Abstract

The invention discloses a road target detection method and device based on multi-sensor fusion, and belongs to the technical field of target detection, and the method comprises the steps: collecting the multi-modal sensor data of a target road and the position information of each sensor; based on the position information of each sensor and the multi-modal sensor data, analyzing a space-data communication relationship among the sensors, and constructing a sensing adjacency matrix; extracting spatio-temporal characteristics of the multi-modal sensor data based on the sensing adjacency matrix, generating a spatio-temporal characteristic set, and obtaining a neuron activation value of each calculation path in real time; performing redundancy analysis on each calculation path according to the neuron activation value of each calculation path, and screening out a reserved path; performing redundancy removal processing on the space-time feature set based on each reserved path to obtain a sparse space-time feature set; and performing data fusion on the sparse space-time feature set to obtain a target detection result of the target road. By implementing the method and the device, the problem of low target detection accuracy in the prior art can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a multi-sensor fusion road target detection method and device. BACKGROUND

[0002] With the rapid development of Internet of Things technology, intelligent terminals are facing the challenge of processing multiple sensor data. The commonly used method for target detection based on multiple sensors at present is bird's eye view (BEV) 3D detection. This method converts image or point cloud data into BEV features, or directly regresses 3D boxes using a query mechanism, which meets the demand for global perspective in automatic driving 3D perception, unmanned aerial vehicle aerial photography and the like.

[0003] However, the query mechanism of the bird's eye view (BEV) 3D detection method samples features from a single reference point, lacks comprehensive perception of global space-time context, and the differences (different sampling frequencies, coordinate systems, resolutions) between sensors (such as cameras and LiDARs) further exacerbate the difficulty of space-time correlation modeling, and cannot fully exploit the complementary information in the data. In addition, the perspective conversion process from image to BEV needs to go through dense feature sampling and rearrangement, which has extremely high computational complexity; at the same time, the FPN multi-scale fusion strategy used to improve detection accuracy, although it can enrich feature expression, also increases the number of feature maps and the amount of repeated calculation, resulting in a significant increase in overall computational burden, affecting the inference efficiency of the model. SUMMARY

[0004] The present application provides a multi-sensor fusion road target detection method and device, which can solve the problem of low target detection accuracy in the prior art.

[0005] To solve the above technical problems, the present application provides a multi-sensor fusion road target detection method, comprising:

[0006] A plurality of sensors are used to collect data of the target road in real time, forming multi-modal sensor data, and the position information of each sensor is obtained; wherein the types of each sensor are different;

[0007] Based on the position information of each sensor and the multi-modal sensor data, the space-data connectivity relationship between each sensor is analyzed, and a sensor adjacency matrix is constructed;

[0008] Based on the sensor adjacency matrix, a plurality of calculation paths in the space-time convolution network are used to extract the space-time features of the multi-modal sensor data, generate a set of space-time features, and obtain the neuron activation values of each calculation path in real time;

[0009] According to the neuron activation values of each calculation path, redundancy analysis is performed on each calculation path to screen out a plurality of reserved paths;

[0010] Based on each reserved path, redundancy removal processing is performed on the spatio-temporal feature set to obtain a sparse spatio-temporal feature set;

[0011] Data fusion is performed on the sparse spatio-temporal feature set to obtain a target detection result of the target road; wherein the target detection result includes a plurality of target recognition results; the target includes a vehicle and a pedestrian; and the recognition result includes a target position and a target size.

[0012] As a preferred solution, the real-time data acquisition of the target road by the plurality of sensors forms multi-modal sensor data, which includes:

[0013] Real-time data acquisition of the target road by the plurality of sensors forms initial sensor data; wherein the initial sensor data includes point cloud data, image data and millimeter wave radar data;

[0014] Data preprocessing is performed on the initial sensor data to obtain processed sensor data;

[0015] Timestamp alignment processing is performed on the processed sensor data to obtain multi-modal sensor data; wherein the multi-modal sensor data includes sensor data corresponding to the plurality of sensors.

[0016] As a preferred solution, based on the position information of each sensor and the multi-modal sensor data, the spatial-data connectivity relationship between each sensor is analyzed, and a sensor adjacency matrix is constructed, which includes:

[0017] For each two sensors, the sensor distance is calculated according to the position information of the two sensors;

[0018] In the multi-modal sensor data, the sensor data corresponding to the two sensors is determined, and the data similarity is calculated according to the sensor data of the two sensors;

[0019] The sensor correlation degree of the two sensors is calculated according to the sensor distance and the data similarity;

[0020] All sensor correlation degrees are summarized to construct a sensor adjacency matrix.

[0021] As a preferred solution, the sensor correlation degree is calculated according to the following formula:

[0022]

[0023] In the formula, A ijσ represents the sensor correlation degree between sensor i and sensor j; s p is the scale parameter for spatial distance; i p represents the position coordinates of sensor i; j h represents the position coordinates of sensor j; i h is the sensor data for sensor i; j This refers to the sensor data of sensor j.

[0024] As a preferred embodiment, the step of extracting spatiotemporal features of the multimodal sensor data based on the sensing adjacency matrix and using several computational paths in a spatiotemporal convolutional network to generate a spatiotemporal feature set includes:

[0025] For each sensor data in the multimodal sensor data, a spatiotemporal convolution operation is performed on the sensor data based on the sensing adjacency matrix to obtain spatiotemporal convolution features;

[0026] Perform temporal convolution operation on the spatiotemporal convolution features to obtain spatiotemporal features;

[0027] Summarize the spatiotemporal features corresponding to all sensor data to generate a spatiotemporal feature set.

[0028] As a preferred embodiment, the redundancy analysis of each computation path is performed based on the neuron activation values ​​of each computation path to select several retained paths, including:

[0029] Statistically analyze the activation values ​​of neurons in all computational paths, and calculate the mean and standard deviation of the activation values;

[0030] Calculate the dynamic activation value threshold based on the mean activation value and the standard deviation of the activation value;

[0031] For each computation path, the contribution of the computation path to the result is calculated based on the neuron activation value of the computation path.

[0032] The calculation paths whose contribution to the result is greater than or equal to the dynamic activation value threshold are determined as reserved paths.

[0033] As a preferred embodiment, calculating the contribution of the computation path based on the neuron activation values ​​of the computation path includes:

[0034] Calculate the product of the activation values ​​of all neurons in the computation path to obtain the activation value product;

[0035] The norm of the product of the activation values ​​is determined as the degree of contribution of the computation path to the result.

[0036] As a preferred embodiment, the process of performing redundancy removal on the spatiotemporal feature set based on each of the retained paths to obtain a sparse spatiotemporal feature set includes:

[0037] Construct a result contribution matrix based on the degree of contribution of each computation path;

[0038] For each spatiotemporal feature in the spatiotemporal feature set, redundant information in the spatiotemporal feature is removed according to the contribution degree matrix of the result to obtain sparse spatiotemporal features;

[0039] By summarizing all sparse spatiotemporal features, a sparse spatiotemporal feature set is obtained.

[0040] As a preferred embodiment, the step of performing data fusion on the sparse spatiotemporal feature set to obtain the target detection result of the target road includes:

[0041] Calculate the feature similarity between every two sparse spatiotemporal features in the sparse spatiotemporal feature set;

[0042] Calculate the spatial distance between every two sensors;

[0043] For each sensor, the fusion weight coefficient of the sensor is calculated based on the spatial distance data of the sensor and the feature similarity data of the sparse spatiotemporal features corresponding to the sensor.

[0044] The fusion weight coefficients of each sensor and the sparse spatiotemporal features corresponding to each sensor are weighted and integrated to calculate the fusion feature data;

[0045] Based on the fused feature data, the identification results of several targets are determined;

[0046] Based on the identification results of each target, the target detection results of the target road are determined.

[0047] Accordingly, the present invention provides a road target detection device with multi-sensor fusion, comprising: a data acquisition module, a sensor correlation analysis module, a feature extraction module, a redundancy analysis module, a redundancy removal module, and a data fusion module;

[0048] The data acquisition module is used to collect data on the target road in real time using several sensors, forming multimodal sensor data, and acquiring the position information of each sensor; wherein, each sensor is of a different type;

[0049] The sensor association analysis module is used to analyze the spatial-data connectivity relationship between the sensors based on the location information of each sensor and the multimodal sensor data, and to construct a sensor adjacency matrix.

[0050] The feature extraction module is used to extract the spatiotemporal features of the multimodal sensor data based on the sensing adjacency matrix and using several computational paths in the spatiotemporal convolutional network, generate a spatiotemporal feature set, and obtain the neuron activation values ​​of each computational path in real time.

[0051] The redundancy analysis module is used to perform redundancy analysis on each of the computation paths based on the neuron activation values ​​of each computation path, and to select a number of retained paths.

[0052] The redundancy removal module is used to perform redundancy removal processing on the spatiotemporal feature set based on each of the retained paths to obtain a sparse spatiotemporal feature set.

[0053] The data fusion module is used to perform data fusion on the sparse spatiotemporal feature set to obtain the target detection result of the target road; wherein, the target detection result includes the identification result of several targets; the targets include vehicles and pedestrians; the identification result includes the target location and target size.

[0054] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0055] This invention provides a multi-sensor fusion method for road target detection, which involves collecting multimodal sensor data and position information of each sensor on the target road; analyzing the spatial-data connectivity between the sensors based on the position information and multimodal sensor data to construct a sensor adjacency matrix; extracting the spatiotemporal features of the multimodal sensor data based on the sensor adjacency matrix to generate a spatiotemporal feature set, and acquiring the neuron activation values ​​of each computation path in real time; performing redundancy analysis on each computation path based on the neuron activation values ​​to select retained paths; performing redundancy removal processing on the spatiotemporal feature set based on each retained path to obtain a sparse spatiotemporal feature set; and fusing the sparse spatiotemporal feature set to obtain the target detection result of the target road. This invention constructs a sensor adjacency matrix by analyzing the spatial-data connectivity relationships between various sensors. Combining this sensor adjacency matrix for target detection improves the utilization rate of the spatiotemporal relationships between sensors, thereby fully mining the effective information in multimodal sensor data to improve the accuracy of target detection. Furthermore, during spatiotemporal feature extraction, redundancy analysis is used to filter out retained paths, retaining only the spatiotemporal feature information of each retained path. By removing redundant information, the computational burden is reduced and the accuracy of target detection is improved. Attached Figure Description

[0056] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 A flowchart illustrating an embodiment of the multi-sensor fusion road target detection method provided by the present invention;

[0058] Figure 2 This is a schematic diagram of one embodiment of the road target detection device with multi-sensor fusion provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0061] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0062] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0064] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0065] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0066] See Figure 1 To address the problem of low target detection accuracy in existing technologies, an embodiment of the present invention provides a road target detection method based on multi-sensor fusion. This method includes steps 101 to 106, each step of which is detailed below:

[0067] Step 101: Use several sensors to collect data on the target road in real time, forming multimodal sensor data, and obtain the position information of each sensor; wherein, each sensor is of a different type.

[0068] In this embodiment of the invention, although existing target detection methods also integrate data from multiple sensors for analysis, current methods only analyze the sensor data itself, ignoring the correlation between sensors. Therefore, target detection requires not only acquiring multimodal sensor data of the target road, but also obtaining the sensor location information as auxiliary analysis.

[0069] As a preferred embodiment, several sensors are used to collect data on the target road in real time, forming multimodal sensor data, including:

[0070] Several sensors are used to collect data on the target road in real time to form initial sensor data; wherein, the initial sensor data includes point cloud data, image data and millimeter-wave radar data;

[0071] The initial sensor data is preprocessed to obtain processed sensor data;

[0072] The processed sensor data is timestamped to obtain multimodal sensor data; wherein, the multimodal sensor data includes sensor data corresponding to several sensors.

[0073] In this embodiment of the invention, multimodal sensor data can be collected through a smart terminal device. Specifically, the smart terminal device is equipped with multiple sensor interfaces to collect initial sensor data from different sensors in real time. For example, a 3D scanning device such as LiDAR can be used to scan the target road to obtain point cloud data. Each data point in the point cloud data records the 3D coordinates of the sampling location, thereby depicting the 3D outline of the target. An imaging device such as a camera can be used to capture images of the target road to obtain image data. The image data records the light information reflected by the target and is stored in the form of a pixel matrix, reflecting the visual appearance of the target (such as color and texture). A millimeter-wave radar can be used to perform frequency-modulated continuous wave detection on each target in the target road to obtain millimeter-wave radar data for each target, including the distance information, speed information, and angle information of each target.

[0074] In this embodiment of the invention, after acquiring the initial sensor data, the initial sensor data is preprocessed. Specifically, the point cloud data is voxelized and subjected to density thresholding to obtain denoised point cloud data; the image data is filtered to obtain denoised image data; and the millimeter-wave radar data is subjected to probabilistic data association processing to obtain denoised millimeter-wave radar data.

[0075] In this embodiment of the invention, voxelization based on an octree combined with a density thresholding method can be used to remove redundancy in point cloud data: the point cloud space is recursively divided into an octree structure, with each node corresponding to a voxel. Let the side length of the voxel be v, for point p... i =(x i ,y i ,z i Its index in the octree is During the construction of the octree, the number of points |V| and the volume v of each voxel are calculated. 3 This leads to the voxel intrapoint density. When D v When the value is less than ρ, where ρ is a pre-set density threshold, points within that voxel are considered redundant and are removed, ultimately yielding the simplified point cloud data P. ′ This method can effectively reduce the amount of data and lower the computational complexity of subsequent processing while preserving the key features of the point cloud.

[0076] In this embodiment of the invention, a bilateral filtering algorithm can be used to remove noise and redundant information from the image, wherein the bilateral filtering formula is:

[0077]

[0078] The image data is represented by a two-dimensional matrix I = [I ij ] m×n It means, I′ ij W represents the color or grayscale value of the pixel in the i-th row and j-th column of the image. ij It is the normalized weight. g s It is the spatial Gaussian kernel function, and its expression is: Used to measure the spatial distance between pixels. σ s It is the standard deviation of the spatial Gaussian kernel; g r It is the range Gaussian kernel function, expression Used to measure the similarity of pixel values, ΔI = |I ij -I mn |,σ r It is the standard deviation of the Gaussian kernel in the range. By using bilateral filtering, it is possible to remove noise while better preserving the edge and detail information of the image, resulting in the processed image I′.

[0079] In this embodiment of the invention, a probabilistic data association (PDA) method can be used to remove redundant measurement values. Suppose that the millimeter-wave radar has k measurement values ​​z at time t. t,1 ,z t,2 ,…,z t,k The predicted value of the target state is Covariance is P t|t-1 For each measurement value z t,i The Mahalanobis distance between it and the target predicted value is calculated as follows:

[0080]

[0081] Where O is the observation matrix, S t =OP t|t-1 HO T +R, where R is the observation noise covariance, and we assume the distance measurement noise standard deviation is σ. r The standard deviation of the angle measurement noise is σ. θ ,but The association probability β can be calculated using Bayes' theorem. i Combining the prior probability and the likelihood function, the association probability is obtained as follows:

[0082]

[0083] Where, β i It is the correlation probability. It is the measured value z t,i Predicted value of the target state The likelihood function P(z) is used to measure the degree of matching between the measured value and the predicted value. t,i ) is the measured value z t,i The prior probability reflects the likelihood of the measured value occurring; the denominator is... It is a normalization factor that ensures that the sum of all association probabilities is 1.

[0084] Based on Mahalanobis distance and association probabilities, we find that there are k measurements of the target at time t, with corresponding association probabilities β1, β2, ..., β k The updated target state estimate is According to the formula Obtain distance estimate Speed ​​estimate Angle estimate These optimized distance, velocity, and angle estimates are combined into new data points. The optimized millimeter-wave radar data R′={r′1,r′2,…r′ is obtained. k This effectively eliminates redundant measurements and improves the accuracy and reliability of the data.

[0085] In this embodiment of the invention, the preprocessed sensor data is obtained and then timestamped. The time information is incorporated into the sensor data through timestamping, so that the data collected at different times can be distinguishable and correlated in the time dimension.

[0086] First, for the data of the i-th sensor after removing redundancy... T i Let's define the timestamp encoding corresponding to the sensor data. Specifically, the timestamp t can be transformed into a low-dimensional vector representation T(t) through embedding encoding, and its dimension is the same as that of the original data X. i The dimensions are adapted to facilitate subsequent concatenation operations, where the embedding encoding employs a linear transformation:

[0087] T(t) = W t ·t+b t

[0088] Among them, W t It is a weight matrix used to learn the linear transformation of timestamps, b t This is a bias vector used to adjust the timestamp encoding result. For the timestamp encoding in each sensor, it is concatenated with the original data; by using the original data X... i With timestamp encoding T i The pieces are stitched together along a specific dimension, where X iIt is a d i A dimensional vector, T i It is a d T If the vectors are of dimension d, then the concatenated vector will have dimension d. i +d T The concatenated vector is X. i +T i This stitching operation tightly integrates time information with the original sensor data.

[0089] Then, a 3D convolution (Conv3D) operation is performed on the stitched data, and the convolutional features are further nonlinearly transformed and dimensionally adjusted through a multilayer perceptron (MLP) to obtain multimodal sensor data with time stamp alignment, enabling them to be compared and fused in the same feature space.

[0090] Step 102: Based on the location information of each sensor and the multimodal sensor data, analyze the spatial-data connectivity relationship between each sensor and construct a sensor adjacency matrix.

[0091] In this embodiment of the invention, in order to improve the utilization rate of the spatiotemporal relationship between sensors and fully explore the effective information in the multimodal sensor data, the spatial-data connectivity relationship between each sensor can be analyzed to construct a sensor adjacency matrix. Combining this sensor adjacency matrix with subsequent target detection steps can effectively improve the accuracy of target detection results.

[0092] As a preferred embodiment, based on the location information of each sensor and the multimodal sensor data, the spatial-data connectivity relationship between each sensor is analyzed, and a sensor adjacency matrix is ​​constructed, including:

[0093] For every two sensors, calculate the sensor distance based on the position information of the two sensors;

[0094] In the multimodal sensor data, sensor data corresponding to two of the sensors are determined, and data similarity is calculated based on the sensor data of the two sensors;

[0095] The sensor correlation degree between the two sensors is calculated based on the sensor distance and the data similarity.

[0096] Summarize the correlation of all sensors and construct a sensor adjacency matrix.

[0097] In this embodiment of the invention, the spatial connectivity between sensors can be analyzed based on the location information of each sensor, and the data connectivity between sensors can be analyzed based on multimodal sensor data. The sensor correlation degree between each pair of sensors is calculated based on the spatial connectivity and data connectivity, and the sensor correlation degree is used as an element of the sensor adjacency matrix to construct the sensor adjacency matrix A.

[0098] As a preferred embodiment, the sensor correlation degree is calculated according to the following formula:

[0099]

[0100] In the formula, A ij σ represents the sensor correlation degree between sensor i and sensor j; s p is the scale parameter for spatial distance; i p represents the position coordinates of sensor i; j h represents the position coordinates of sensor j; i h is the sensor data for sensor i; j This refers to the sensor data of sensor j.

[0101] Among them, ||p i -p j || represents the Euclidean distance between sensors i and j, used to measure the spatial proximity of the sensors. It is the cosine similarity between data from different sensors, used to measure the similarity of sensor data.

[0102] The sensor adjacency matrix constructed using the above method not only considers the spatial distance between sensors but also fully takes into account the similarity between sensor data, thus more accurately describing the connection relationship between sensor nodes.

[0103] Step 103: Based on the sensing adjacency matrix, extract the spatiotemporal features of the multimodal sensor data using several computational paths in the spatiotemporal convolutional network, generate a spatiotemporal feature set, and obtain the neuron activation values ​​of each computational path in real time.

[0104] In this embodiment of the invention, spatiotemporal features of multimodal sensor data can be extracted through spatiotemporal convolutional networks. However, there are multiple computational paths in the spatiotemporal convolutional network, which may result in computational redundancy. Therefore, it is necessary to synchronously collect the neuron activation values ​​of each computational path to analyze the degree of redundancy of each computational path, thereby simplifying the collected spatiotemporal features and reducing the subsequent computational burden.

[0105] As a preferred embodiment, based on the sensing adjacency matrix, several computational paths in a spatiotemporal convolutional network are used to extract the spatiotemporal features of the multimodal sensor data, generating a spatiotemporal feature set, including:

[0106] For each sensor data in the multimodal sensor data, a spatiotemporal convolution operation is performed on the sensor data based on the sensing adjacency matrix to obtain spatiotemporal convolution features;

[0107] Perform temporal convolution operation on the spatiotemporal convolution features to obtain spatiotemporal features;

[0108] Summarize the spatiotemporal features corresponding to all sensor data to generate a spatiotemporal feature set.

[0109] In this embodiment of the invention, a spatiotemporal convolutional network is used to extract the spatiotemporal features of multimodal sensor data, which includes two core steps: a spatiotemporal convolution operation and a temporal convolution operation.

[0110] First, spatiotemporal convolution operations can extract features of sensor data in the spatiotemporal dimension. By considering the sensor adjacency matrix when performing spatiotemporal convolution operations, effective features can be fully explored.

[0111] The spatiotemporal convolution operation can be represented as:

[0112]

[0113] In the formula, H i (l+1) H represents the spatiotemporal convolutional features of sensor i in the (l+1)th layer of the spatiotemporal graph convolutional network. i (l) Let σ represent the spatiotemporal convolutional features of sensor i in the l-th layer of the spatiotemporal graph convolutional network; σ is the ReLU activation function, which is expressed as ReLU(x) = max(0,x), used to increase the nonlinear expressive power of the model. The k-hop adjacency matrix can be obtained by exponentiation of the sensing adjacency matrix A, i.e. It considers the neighbor relationships of nodes in the graph structure after k steps of connection, enriching the information propagation range of nodes; W k The corresponding convolution weight matrix is ​​obtained through training and is used to adjust the influence of different neighbor node features on the current node; TemporalConv(H (l) ) is a one-dimensional temporal convolution operation used to capture the changing characteristics of data in time series.

[0114] Assume the one-dimensional temporal convolution kernel is K t Size k t Then, the temporal convolution operation can be represented as:

[0115]

[0116] In the formula, T ijThe spatiotemporal feature located at position (i,j) in the feature map; It is the element at position m of the temporal convolution kernel, used to learn features on the time series. It is the spatiotemporal convolutional feature at the input feature map position (i,j+m), b t It is the bias term of temporal convolution, used to adjust the output of the convolution operation.

[0117] For each sensor, the spatiotemporal features at each location are summarized to obtain the complete spatiotemporal features of the sensor. Further summarizing the complete spatiotemporal features of all sensors can form a spatiotemporal feature set.

[0118] H = {H1,H2,…,H} m}

[0119] In the formula, H is the spatiotemporal feature set; H m This represents the complete spatiotemporal characteristics of the m-th sensor.

[0120] Step 104: Based on the neuron activation values ​​of each computation path, perform redundancy analysis on each computation path and select several retained paths.

[0121] As a preferred embodiment, based on the neuron activation values ​​of each computation path, a redundancy analysis is performed on each computation path to select several retained paths, including:

[0122] Statistically analyze the activation values ​​of neurons in all computational paths, and calculate the mean and standard deviation of the activation values;

[0123] Calculate the dynamic activation value threshold based on the mean activation value and the standard deviation of the activation value;

[0124] For each computation path, the contribution of the computation path to the result is calculated based on the neuron activation value of the computation path.

[0125] The calculation paths whose contribution to the result is greater than or equal to the dynamic activation value threshold are determined as reserved paths.

[0126] In this embodiment of the invention, a current dynamic activation value threshold is first constructed based on the neuron activation values ​​of each computation path. Then, the contribution degree of the path is calculated based on the neuron activation values ​​of each computation path. The contribution degree is compared with the dynamic activation value threshold to determine that computation paths with a contribution degree greater than or equal to the dynamic activation value threshold are retained paths, and computation paths with a contribution degree less than the dynamic activation value threshold are redundant paths.

[0127] In this embodiment of the invention, the current dynamic activation value threshold is constructed based on the neuron activation values ​​of each computation path:

[0128] τ=μ-β·σ

[0129] In the formula, τ is the dynamic activation threshold; μ is the mean activation value; σ is the standard deviation of the activation value; and β is an adjustable coefficient, typically ranging from (0,1). The mean and standard deviation of the activation values ​​are calculated based on the activation values ​​of neurons in each computational path. Based on the needs of practical application scenarios, the threshold value can be flexibly controlled by adjusting the adjustable coefficient β. For example, in scenarios with relatively stable data changes and low noise, the value of β can be appropriately increased to make the threshold relatively low, thereby retaining more computational paths to fully utilize data information. Conversely, in scenarios with large data fluctuations and high noise, the value of β can be decreased to make the threshold relatively high, more strictly filtering computational paths and removing more potentially redundant information.

[0130] As a preferred embodiment, the contribution of the computation path to the results is calculated based on the neuron activation values ​​of the computation path, including:

[0131] Calculate the product of the activation values ​​of all neurons in the computation path to obtain the activation value product;

[0132] The norm of the product of the activation values ​​is determined as the degree of contribution of the computation path to the result.

[0133] In this embodiment of the invention, the contribution of the computational path is calculated based on the activation values ​​of the neurons in each computational path. Assume there are two neurons in the computational path, with activation values ​​a and a' respectively. i and a j Then the contribution of the computation path results of this path can be expressed as ||a i ·a j ||.

[0134] Based on the contribution of the computation path results and the dynamic activation value threshold, it can be determined whether the computation path is a reserved path or a redundant path.

[0135]

[0136] In the formula, m ij This serves as the computation path identifier for the feature map position (i,j), when m ij If m = 1, then the computational path is considered to contribute significantly to the final result, and therefore is a retained path; when m ij If the value is 0, then the computation path is considered to contribute little to the final result and is considered a redundant path.

[0137] Step 105: Based on each of the retained paths, perform redundancy removal processing on the spatiotemporal feature set to obtain a sparse spatiotemporal feature set.

[0138] In this embodiment of the invention, after calculating the spatiotemporal feature set and determining the retention path, the spatiotemporal feature set is redundancy removed using the retention path. Only the data corresponding to each retention path is retained, and the data of the redundant paths are removed to form a sparse spatiotemporal feature set.

[0139] As a preferred embodiment, based on each of the retained paths, the spatiotemporal feature set is subjected to redundancy removal processing to obtain a sparse spatiotemporal feature set, including:

[0140] Construct a result contribution matrix based on the degree of contribution of each computation path;

[0141] For each spatiotemporal feature in the spatiotemporal feature set, redundant information in the spatiotemporal feature is removed according to the contribution degree matrix of the result to obtain sparse spatiotemporal features;

[0142] By summarizing all sparse spatiotemporal features, a sparse spatiotemporal feature set is obtained.

[0143] In this embodiment of the invention, redundancy removal is performed for each spatiotemporal feature:

[0144]

[0145] In the formula, Let m be the sparse spatiotemporal convolutional feature representation of the feature map location (i,j) in the l-th layer of the spatiotemporal graph convolutional network; ij This serves as the identifier for the computation path at the feature map location (i,j); Let be the spatiotemporal convolutional feature representation of the feature map location (i,j) in the l-th layer of the spatiotemporal graph convolutional network.

[0146] The above formula indicates that when m ij When = 1, The feature value at this position is preserved; when m ij When = 0, The feature value at this position is set to zero, thereby achieving sparsity of the feature representation.

[0147] This dynamic sparsity operation enables real-time optimization of computational resources during each spatiotemporal graph convolution calculation, removing redundant paths that contribute little to the final fusion result. The aforementioned sparse spatiotemporal convolutional features, after temporal convolution operations, can be aggregated into a sparse spatiotemporal feature set. :

[0148]

[0149] In the formula, It is a sparse spatiotemporal feature set; This represents the complete spatiotemporal characteristics of the m-th sensor.

[0150] Step 106: Perform data fusion on the sparse spatiotemporal feature set to obtain the target detection result of the target road; wherein, the target detection result includes the identification result of several targets; the targets include vehicles and pedestrians; the identification result includes the target location and target size.

[0151] In this embodiment of the invention, the collected sparse spatiotemporal feature set is fused, and then the objective function is jointly optimized to obtain the target position and target size of each target in the target road, and finally the target detection result of the target road is formed.

[0152] As a preferred embodiment, data fusion is performed on the sparse spatiotemporal feature set to obtain the target detection result of the target road, including:

[0153] Calculate the feature similarity between every two sparse spatiotemporal features in the sparse spatiotemporal feature set;

[0154] Calculate the spatial distance between every two sensors;

[0155] For each sensor, the fusion weight coefficient of the sensor is calculated based on the spatial distance data of the sensor and the feature similarity data of the sparse spatiotemporal features corresponding to the sensor.

[0156] The fusion weight coefficients of each sensor and the sparse spatiotemporal features corresponding to each sensor are weighted and integrated to calculate the fusion feature data;

[0157] Based on the fused feature data, the identification results of several targets are determined;

[0158] Based on the identification results of each target, the target detection results of the target road are determined.

[0159] In this embodiment of the invention, to fuse the sparse spatiotemporal features in the sparse spatiotemporal feature set, the fusion weight coefficient of each sensor must first be calculated. The fusion weight coefficient can be calculated by the feature similarity between sparse spatiotemporal features and the spatial distance between sensors.

[0160] The feature similarity between any two sparse spatiotemporal features can be calculated using the following formula:

[0161]

[0162] In the formula, The sparse spatiotemporal characteristics of sensor i; sim represents the sparse spatiotemporal characteristics of sensor j; ijLet be the feature similarity between the sparse spatiotemporal features corresponding to sensor i and sensor j, used to measure the degree of similarity between the features of the two sensors in the feature space, with a value ranging from [-1, 1]. When sim ij The closer the value is to 1, the more similar the characteristics of the two sensors are; the closer it is to -1, the less similar they are.

[0163] The spatial distance between sensors can be calculated using the following formula:

[0164]

[0165] In the formula, d ij Let x be the spatial distance between sensor i and sensor j, and let the spatial coordinates of sensor i be (x, y, j). i ,y i ,z i The spatial coordinates of sensor j are (x, y). j ,y j ,z j ).

[0166] After calculating the spatial distance between the sensors and the feature similarity between the corresponding sparse spatiotemporal features, the fusion weight coefficients of different sensors can be calculated according to the following formula:

[0167]

[0168] In the formula, ω i Let α and γ be the fusion weight coefficients for sensor i; α and γ are hyperparameters used to adjust the influence of spatial distance and feature similarity on the fusion weight coefficients; the denominator of the above equation is... It performs an exponential summation of all related terms between sensors and sensor i, and its function is to calculate the numerator... Normalize the coefficients so that all fusion weights ω i The sum is 1, that is

[0169] In this embodiment of the invention, the fusion weight coefficients of each sensor and the sparse spatiotemporal features corresponding to each sensor are weighted and integrated to calculate the fused feature data:

[0170]

[0171] In the formula, F represents the fused feature data; ω i For sensor i, the fusion weighting coefficient is denoted as . Let represent the sparse spatiotemporal characteristics of sensor i; m represents the number of sensors.

[0172] In this embodiment of the invention, by jointly optimizing the objective function, the target detection result of the target road can be determined based on the fused feature data.

[0173] Specifically, optimization objective functions are set for target detection tasks such as identifying target location and target size. The objective function loss is calculated based on the fused feature data. When the model does not converge, optimization algorithms such as stochastic gradient descent are used to iteratively update the system network parameters. At the end of the model iteration, the fused inference result, namely the target location and target size, is obtained, and finally the target detection result of the target road is formed.

[0174] In this embodiment of the invention, the target detection results of the target road are obtained, which is equivalent to achieving accurate perception of key targets such as vehicles and pedestrians on the target road. These structured perception results provide a reliable basis for vehicle path planning and decision control, which is the key to achieving safe autonomous driving.

[0175] Implementing the above embodiments has the following effects:

[0176] This invention provides a multi-sensor fusion method for road target detection, which involves collecting multimodal sensor data and position information of each sensor on the target road; analyzing the spatial-data connectivity between the sensors based on the position information and multimodal sensor data to construct a sensor adjacency matrix; extracting spatiotemporal features from the multimodal sensor data based on the sensor adjacency matrix to generate a spatiotemporal feature set, and acquiring the neuron activation values ​​of each computation path in real time; performing redundancy analysis on each computation path based on the neuron activation values ​​to select retained paths; performing redundancy removal processing on the spatiotemporal feature set based on the retained paths to obtain a sparse spatiotemporal feature set; and fusing the sparse spatiotemporal feature set to obtain the target detection result for the target road. This invention constructs a sensor adjacency matrix by analyzing the spatial-data connectivity relationships between various sensors. Combining this sensor adjacency matrix for target detection improves the utilization rate of the spatiotemporal relationships between sensors, thereby fully mining the effective information in multimodal sensor data to improve the accuracy of target detection. Furthermore, during spatiotemporal feature extraction, redundancy analysis is used to filter out retained paths, retaining only the spatiotemporal feature information of each retained path. By removing redundant information, the computational burden is reduced and the accuracy of target detection is improved.

[0177] Current target detection methods fail to fully utilize the spatiotemporal relationships between sensors, thus failing to fully extract effective information from the data; and when extracting spatiotemporal features, there is significant data and computational redundancy, resulting in a heavy computational burden.

[0178] like Figure 2 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided;

[0179] An embodiment of the present invention provides a road target detection device with multi-sensor fusion, comprising: a data acquisition module, a sensor correlation analysis module, a feature extraction module, a redundancy analysis module, a redundancy removal module, and a data fusion module;

[0180] The data acquisition module is used to collect data on the target road in real time using several sensors, forming multimodal sensor data, and acquiring the position information of each sensor; wherein, each sensor is of a different type;

[0181] The sensor association analysis module is used to analyze the spatial-data connectivity relationship between the sensors based on the location information of each sensor and the multimodal sensor data, and to construct a sensor adjacency matrix.

[0182] The feature extraction module is used to extract the spatiotemporal features of the multimodal sensor data based on the sensing adjacency matrix and using several computational paths in the spatiotemporal convolutional network, generate a spatiotemporal feature set, and obtain the neuron activation values ​​of each computational path in real time.

[0183] The redundancy analysis module is used to perform redundancy analysis on each of the computation paths based on the neuron activation values ​​of each computation path, and to select a number of retained paths.

[0184] The redundancy removal module is used to perform redundancy removal processing on the spatiotemporal feature set based on each of the retained paths to obtain a sparse spatiotemporal feature set.

[0185] The data fusion module is used to perform data fusion on the sparse spatiotemporal feature set to obtain the target detection result of the target road; wherein, the target detection result includes the identification result of several targets; the targets include vehicles and pedestrians; the identification result includes the target location and target size.

[0186] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the multi-sensor fusion road target detection method provided by any of the above-described method embodiments of the present invention.

[0187] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0188] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A road target detection method using multi-sensor fusion, characterized in that, include: Several sensors are used to collect data on the target road in real time, forming multimodal sensor data, and the position information of each sensor is obtained; wherein, each sensor is of a different type; Based on the location information of each sensor and the data from the multimodal sensor, the spatial-data connectivity relationship between each sensor is analyzed, and a sensor adjacency matrix is ​​constructed. Based on the sensing adjacency matrix, the spatiotemporal features of the multimodal sensor data are extracted using several computational paths in the spatiotemporal convolutional network, a spatiotemporal feature set is generated, and the neuron activation values ​​of each computational path are obtained in real time. Based on the neuron activation values ​​of each computation path, a redundancy analysis is performed on each computation path to select several retained paths. Based on each of the preserved paths, the spatiotemporal feature set is subjected to redundancy removal processing to obtain a sparse spatiotemporal feature set; Data fusion is performed on the sparse spatiotemporal feature set to obtain the target detection result of the target road; wherein, the target detection result includes the identification result of several targets; the targets include vehicles and pedestrians; the identification result includes the target location and target size.

2. The road target detection method based on multi-sensor fusion according to claim 1, characterized in that, The method employs several sensors to collect data on the target road in real time, forming multimodal sensor data, including: Several sensors are used to collect data on the target road in real time to form initial sensor data; wherein, the initial sensor data includes point cloud data, image data and millimeter-wave radar data; The initial sensor data is preprocessed to obtain processed sensor data; The processed sensor data is timestamped to obtain multimodal sensor data; wherein, the multimodal sensor data includes sensor data corresponding to several sensors.

3. The road target detection method based on multi-sensor fusion according to claim 2, characterized in that, The step of analyzing the spatial-data connectivity relationships between the sensors based on their location information and the multimodal sensor data, and constructing a sensor adjacency matrix, includes: For every two sensors, calculate the sensor distance based on the position information of the two sensors; In the multimodal sensor data, sensor data corresponding to two of the sensors are determined, and data similarity is calculated based on the sensor data of the two sensors; The sensor correlation degree between the two sensors is calculated based on the sensor distance and the data similarity. Summarize the correlation of all sensors and construct a sensor adjacency matrix.

4. The road target detection method based on multi-sensor fusion according to claim 3, characterized in that, The sensor correlation degree is calculated according to the following formula: In the formula, A ij σ represents the sensor correlation degree between sensor i and sensor j; s p is the scale parameter for spatial distance; i p represents the position coordinates of sensor i; j h represents the position coordinates of sensor j; i h is the sensor data for sensor i; j This refers to the sensor data of sensor j.

5. The road target detection method based on multi-sensor fusion according to claim 4, characterized in that, Based on the sensing adjacency matrix, the spatiotemporal features of the multimodal sensor data are extracted using several computational paths in a spatiotemporal convolutional network to generate a spatiotemporal feature set, including: For each sensor data in the multimodal sensor data, a spatiotemporal convolution operation is performed on the sensor data based on the sensing adjacency matrix to obtain spatiotemporal convolution features; Perform temporal convolution operation on the spatiotemporal convolution features to obtain spatiotemporal features; Summarize the spatiotemporal features corresponding to all sensor data to generate a spatiotemporal feature set.

6. The road target detection method based on multi-sensor fusion according to claim 5, characterized in that, The redundancy analysis of each computation path is performed based on the neuron activation values ​​of each computation path, and several retained paths are selected, including: Statistically analyze the activation values ​​of neurons in all computational paths, and calculate the mean and standard deviation of the activation values; Calculate the dynamic activation value threshold based on the mean activation value and the standard deviation of the activation value; For each computation path, the contribution of the computation path to the result is calculated based on the neuron activation value of the computation path. The calculation paths whose contribution to the result is greater than or equal to the dynamic activation value threshold are determined as reserved paths.

7. The road target detection method based on multi-sensor fusion according to claim 6, characterized in that, The calculation of the contribution of the computation path to the results based on the neuron activation values ​​of the computation path includes: Calculate the product of the activation values ​​of all neurons in the computation path to obtain the activation value product; The norm of the product of the activation values ​​is determined as the degree of contribution of the computation path to the result.

8. The road target detection method based on multi-sensor fusion according to claim 7, characterized in that, The process of performing redundancy removal on the spatiotemporal feature set based on each of the retained paths to obtain a sparse spatiotemporal feature set includes: Construct a result contribution matrix based on the degree of contribution of each computation path; For each spatiotemporal feature in the spatiotemporal feature set, redundant information in the spatiotemporal feature is removed according to the contribution degree matrix of the result to obtain sparse spatiotemporal features; By summarizing all sparse spatiotemporal features, a sparse spatiotemporal feature set is obtained.

9. The road target detection method based on multi-sensor fusion according to claim 8, characterized in that, The step of fusing the sparse spatiotemporal feature set to obtain the target detection result of the target road includes: Calculate the feature similarity between every two sparse spatiotemporal features in the sparse spatiotemporal feature set; Calculate the spatial distance between every two sensors; For each sensor, the fusion weight coefficient of the sensor is calculated based on the spatial distance data of the sensor and the feature similarity data of the sparse spatiotemporal features corresponding to the sensor. The fusion weight coefficients of each sensor and the sparse spatiotemporal features corresponding to each sensor are weighted and integrated to calculate the fusion feature data; Based on the fused feature data, the identification results of several targets are determined; Based on the identification results of each target, the target detection results of the target road are determined.

10. A road target detection device based on multi-sensor fusion, characterized in that, include: The system includes a data acquisition module, a sensor correlation analysis module, a feature extraction module, a redundancy analysis module, a redundancy removal module, and a data fusion module. The data acquisition module is used to collect data on the target road in real time using several sensors, forming multimodal sensor data, and acquiring the position information of each sensor; wherein, each sensor is of a different type; The sensor association analysis module is used to analyze the spatial-data connectivity relationship between the sensors based on the location information of each sensor and the multimodal sensor data, and to construct a sensor adjacency matrix. The feature extraction module is used to extract the spatiotemporal features of the multimodal sensor data based on the sensing adjacency matrix and using several computational paths in the spatiotemporal convolutional network, generate a spatiotemporal feature set, and obtain the neuron activation values ​​of each computational path in real time. The redundancy analysis module is used to perform redundancy analysis on each of the computation paths based on the neuron activation values ​​of each computation path, and to select a number of retained paths. The redundancy removal module is used to perform redundancy removal processing on the spatiotemporal feature set based on each of the retained paths to obtain a sparse spatiotemporal feature set. The data fusion module is used to perform data fusion on the sparse spatiotemporal feature set to obtain the target detection result of the target road; wherein, the target detection result includes the identification result of several targets; the targets include vehicles and pedestrians; the identification result includes the target location and target size.

Images