Obstacle detection method and device based on multi-sensor data

Abstract

The present application discloses an obstacle detection method based on multi-sensor data, which obtains sensor data of more than two sensors, wherein the sensor data includes binocular visual image data and radar image data, obtains visual point cloud data in three-dimensional space based on binocular visual image data, divides the three-dimensional space where the visual point cloud data is located into at least one first voxel, obtains image features of each first voxel based on the visual point cloud data, obtains radar point cloud data based on radar image data, divides the three-dimensional space where the radar point cloud data is located into at least one second voxel, obtains radar features of each second voxel based on the radar point cloud image data, and sequentially performs feature fusion within the voxel, feature fusion between local voxels, and feature fusion between global voxels on the image features and radar features of each voxel to obtain fusion features of each voxel; obstacle detection is performed based on the fusion features of each voxel. The present application improves the reliability of obstacle detection.

Description

Technical Field

[0001] The present invention relates to the field of obstacle detection, and in particular, to an obstacle detection method based on multi-sensor data. Background Art

[0002] With the development of obstacle detection technology, more and more obstacle detection is achieved by integrating multiple sensor data, such as obstacle detection based on visual images collected by cameras and point cloud images collected by radars.

[0003] Existing obstacle detection based on multiple sensor data usually implements obstacle detection in a selective manner. For example, when obstacle detection is performed based on visual images collected by a camera and point cloud images collected by a radar, the visual image data is used to detect the lighting state. When the lighting state is normal, the visual image data is used to detect obstacles; when the lighting state is abnormal, the point cloud image data is used to determine the obstacle distance.

[0004] In actual applications, due to the complex environment, factors such as lighting, fog, and precipitation may cause the camera or radar to fail. When visual images and radar images are used alone for obstacle detection, they cannot complement each other when the detection effects of both are poor. Summary of the invention

[0005] The present invention provides an obstacle detection method based on multi-sensor data to improve the reliability of obstacle detection.

[0006] A first aspect of an embodiment of the present application provides an obstacle detection method based on multi-sensor data, the method comprising:

[0007] Get sensor data from more than two sensors,

[0008] For sensor data from any sensor, the three-dimensional space corresponding to the sensor data is divided into at least one voxel, and a feature of each voxel is acquired based on the sensor data to obtain a voxel feature corresponding to the sensor data from the sensor,

[0009] For any voxel, the voxel features corresponding to the sensor data from at least two sensors in the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fused features of the voxel, and the process is repeated until the voxel features of all voxels are fused.

[0010] Based on the fusion features of each voxel, obstacle detection is performed on each voxel.

[0011] Preferably, the feature fusion in the voxel is such that each voxel feature of the voxel is fused to obtain a first fused feature,

[0012] Preferably, the feature fusion between the local voxels enables the first fused features of the nearby voxels of the voxel to be fused to obtain the second fused features.

[0013] Preferably, the feature fusion between the global voxels enables the second fused features of the nearby voxels and the non-nearby voxels of the voxel to be fused to obtain the third fused feature.

[0014] Preferably, the feature fusion within the voxel includes:

[0015] For any voxel,

[0016] The voxel features of the voxel are spliced ​​in the channel dimension to obtain the spliced ​​feature vector.

[0017] Inputting the concatenated feature vectors into a plurality of sequentially connected fully connected layers, wherein the fully connected layers output the first fused feature of the voxel;

[0018] Preferably, the feature fusion between global voxels includes:

[0019] For any voxel,

[0020] The second fusion feature of the voxel is input into three different groups of fully connected layers to obtain the query value, key value, and feature value respectively.

[0021] Multiply each query value by all key values ​​and normalize them to get the weight value.

[0022] The feature values ​​are weighted and summed using the weight values ​​to obtain the third fused feature of the voxel.

[0023] Preferably, the feature fusion between local voxels includes:

[0024] For any voxel,

[0025] The first fusion feature of the voxel is input into a neural network model for local inter-voxel feature fusion, and the neural network model outputs the second fusion feature of the voxel.

[0026] Preferably, the neural network model comprises: a maximum pooling layer, a first residual network layer, an upsampling layer, and a second residual network layer connected in sequence.

[0027] The input end of the maximum pooling layer is directly connected to the output end of the upsampling layer, wherein the features of both ends of the direct connection have the same size, so that the features of the two ends are accumulated and input into the second residual network layer.

[0028] or,

[0029] Preferably, the neural network model comprises: at least one maximum pooling layer, at least one first residual network layer, at least one upsampling layer, and at least one second residual network layer.

[0030] A maximum pooling layer is connected to a first residual network layer to form a pooling residual unit, multiple pooling residual units are connected in sequence, an upsampling layer is connected to a second residual network layer to form an upsampling residual unit, and multiple upsampling residual units are connected in sequence.

[0031] The first residual network layer in the last pooled residual unit is connected to the upsampling layer in the first upsampling residual unit.

[0032] At least one of the input ends of each maximum pooling layer is directly connected to at least one of the output ends of each upsampling layer, wherein the features of the two directly connected ends have the same size, so that the features of the two ends are accumulated and input to the next layer.

[0033] Among them, the number of maximum pooling layers, residual network layers, and upsampling layers are used to determine the range of nearby voxels.

[0034] Preferably, the neural network model further comprises: at least one third residual network layer, each third residual network layer is connected in sequence,

[0035] The first residual network layer and the upsampling layer connected in sequence are further connected to each third residual network layer, wherein the first residual network layer is connected to the first third residual network layer in each third residual network layer, and the last third residual network layer in each third residual network layer is connected to the upsampling layer.

[0036] Preferably, the first residual network layer in the last pooled residual unit is connected to the first third residual network layer in each third residual network layer, and the last third residual network layer in each third residual network layer is connected to the upsampling layer in the first upsampling residual unit.

[0037] Preferably, the direct connection further includes at least one fourth residual network layer, and each fourth residual network layer is connected in sequence.

[0038] Preferably, the number of upsampling layers is smaller than the number of maximum pooling layers, so that the feature size of the second fused feature is smaller than the feature size of the first fused feature;

[0039] Preferably, the step of sequentially fusing the image feature of the voxel with the radar feature within the voxel, fusing the feature between local voxels, and fusing the feature between global voxels further comprises:

[0040] For any voxel, a deconvolution operation is performed on the third fused feature of the voxel so that the fused feature of the voxel has the same feature size as the first fused feature.

[0041] Preferably, the sensor data includes: binocular vision image data, and radar image data;

[0042] Preferably, for sensor data from any sensor, dividing the three-dimensional space corresponding to the sensor data into at least one voxel, and acquiring the feature of each voxel based on the sensor data to obtain the voxel feature corresponding to the sensor data from the sensor, includes:

[0043] Based on the binocular visual image data, visual point cloud data in a three-dimensional space is obtained, the three-dimensional space where the visual point cloud data is located is divided into at least one first voxel, and image features of each first voxel are obtained based on the visual point cloud data.

[0044] Based on the radar image data, radar point cloud data is acquired, the three-dimensional space where the radar point cloud data is located is divided into at least one second voxel, and a radar feature of each second voxel is acquired based on the radar point cloud image data;

[0045] Preferably, the voxel features corresponding to the sensor data from at least two sensors in the voxel are sequentially subjected to intra-voxel feature fusion, local voxel feature fusion, and global voxel feature fusion to obtain the fusion feature of the voxel, and the process is repeated until the voxel features of each voxel are fused, including:

[0046] For any voxel, the image feature and radar feature of the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fusion feature of the voxel. This process is repeated until the image feature and radar feature of each voxel are fused.

[0047] Preferably, the step of splicing the voxel features of the voxel in the channel dimension comprises:

[0048] The image feature of the voxel is connected to the radar feature.

[0049] In the case where the voxel has no image feature or no radar feature, the feature value in the corresponding channel is set to 0.

[0050] Preferably, acquiring the image feature of each first voxel based on the visual point cloud data comprises:

[0051] For any first voxel,

[0052] Collect each projection point in the visual point cloud data within the first voxel,

[0053] The feature vectors of the collected projection points are spliced ​​in the channel dimension.

[0054] Using the first feature extraction network, feature extraction is performed on the spliced ​​feature vector to obtain the image feature of the first voxel.

[0055] Among them, the eigenvector of the projection point is obtained as follows:

[0056] For any projection point, obtain the feature vector of the pixel point corresponding to the projection point in any visual feature map, and obtain the feature vector of the projection point.

[0057] Preferably, the visual feature map is obtained based on feature extraction of any one of the binocular visual images;

[0058] Preferably, the acquiring the radar feature of each second voxel based on the radar point cloud image data comprises:

[0059] For any second voxel,

[0060] Collect each radar point in the radar point cloud data within the second voxel,

[0061] The feature vectors of the collected radar points are spliced ​​in the channel dimension.

[0062] The second feature extraction network is used to extract features from the spliced ​​feature vectors to obtain the radar features of the second voxel.

[0063] Among them, the feature vector of the radar point is obtained as follows:

[0064] For any radar point, a feature vector of a radar point corresponding to the radar point in a radar feature map is obtained, wherein the radar feature map is obtained by performing feature extraction on radar point cloud image data.

[0065] Preferably, the collecting of each projection point in the visual point cloud data in the first voxel includes:

[0066] When the number of projection points in the first voxel is less than the set first value, the projection points are repeatedly collected so that the number of collected projection points is equal to the first value.

[0067] When the number of projection points in the first voxel is not less than the set first value, collecting each projection point in the first voxel so that the number of collected projection points is equal to the first value;

[0068] Preferably, the step of splicing the feature vectors of the collected projection points in the channel dimension includes:

[0069] The feature vectors of each collected projection point are used as row vectors, and the row vectors are adjacent to each other row by row, or,

[0070] The feature vectors of each collected projection point are respectively used as column vectors, and the column vectors are adjacent to each other column by column;

[0071] Preferably, the collecting of each radar point in the radar point cloud data within the second voxel comprises:

[0072] When the number of radar points in the second voxel is less than the set second value, the radar points are repeatedly collected so that the number of collected radar points is equal to the second value.

[0073] When the number of radar points in the second voxel is not less than the set second value, collecting each radar point in the second voxel so that the number of collected radar points is equal to the second value;

[0074] Preferably, the step of splicing the feature vectors of the collected radar points in the channel dimension includes:

[0075] The feature vectors of each radar point collected are used as row vectors, and the row vectors are adjacent to each other row by row, or,

[0076] The characteristic vectors of each collected radar point are respectively used as column vectors, and the column vectors are adjacent to each other column by column.

[0077] Preferably, the first voxel has the same size as the second voxel, and the three-dimensional space where the visual point cloud data is located is the same as the three-dimensional space where the radar point cloud data is located.

[0078] Preferably, the obstacle detection of each voxel based on the fusion features of each voxel includes:

[0079] For any voxel,

[0080] The fusion features of the voxel are input into the neural network model for obstacle detection to obtain the probability of the existence of an obstacle in the voxel.

[0081] When the probability that an obstacle exists in the voxel is greater than a set probability threshold, it is determined that an obstacle exists in the voxel.

[0082] A second aspect of the present application provides an obstacle detection device based on multi-sensor data, the device comprising:

[0083] A data acquisition module, used to acquire sensor data from more than two sensors,

[0084] A feature acquisition module is used to divide the three-dimensional space corresponding to the sensor data from any sensor into at least one voxel, and acquire the feature of each voxel based on the sensor data to obtain the voxel feature corresponding to the sensor data from the sensor,

[0085] The feature fusion module is used to perform feature fusion within the voxel, feature fusion between local voxels, and feature fusion between global voxels for each voxel feature corresponding to the sensor data from at least two sensors in the voxel, to obtain the fusion feature of the voxel, and to perform the fusion repeatedly until the voxel features of each voxel are fused.

[0086] The detection module is used to detect obstacles of each voxel based on the fusion features of each voxel.

[0087] A third aspect of the present application provides a computer storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the above-mentioned obstacle detection method based on multi-sensor data.

[0088] The obstacle detection method based on multi-sensor data provided by the present invention takes voxels as units, performs intra-voxel feature fusion, local voxel feature fusion, and global voxel feature fusion on voxel features in the voxel in sequence, and obtains voxel fusion features, so that each voxel contains global information from multiple sensors, which is helpful for performing obstacle detection on the voxels, avoids being limited to performing obstacle detection on single sensor data during the obstacle detection process, improves the complementarity of multi-sensor data during the obstacle detection process, and thus improves the reliability and accuracy of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A flowchart of an obstacle detection method based on multi-sensor data according to an embodiment of the present application is shown.

[0090] Figure 2 A schematic diagram of a neural network model for feature fusion between local voxels.

[0091] Figure 3 Another schematic diagram of a neural network model for feature fusion between local voxels.

[0092] Figure 4 A flowchart of an obstacle detection method based on multi-sensor data according to an embodiment of the present application is shown.

[0093] Figure 5 A schematic diagram of splicing projection point feature vectors in this embodiment.

[0094] Figure 6A schematic diagram of stitching the image feature of the first voxel and the radar feature of the second voxel corresponding to the first voxel in this embodiment.

[0095] Figure 7 A schematic diagram of a neural network model for feature fusion between local voxels in this embodiment.

[0096] Figure 8 A schematic diagram of feature fusion between global voxels in this embodiment.

[0097] Fig. 9 This is an example of obstacle prediction results in this embodiment.

[0098] Fig.10 A schematic diagram of an obstacle detection device based on multi-sensor data according to an embodiment of the present application.

[0099] Fig.11 Another schematic diagram of an obstacle detection device based on multi-sensor data according to an embodiment of the present application. DETAILED DESCRIPTION

[0100] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0101] The obstacle detection method based on multi-sensor data provided in the embodiment of the present application uses voxels as the granularity, fuses the features of two or more sensor data in each voxel, obtains the fusion features of each voxel, and performs obstacle detection on each voxel based on the fusion features of each voxel to improve the reliability and accuracy of obstacle detection.

[0102] See also Figure 1 As shown, Figure 1 This is a flow chart of an obstacle detection method based on multi-sensor data in an embodiment of the present application. The method includes:

[0103] Step 101, obtaining sensor data from two or more sensors,

[0104] For example, sensor data includes binocular vision image data and radar image data.

[0105] As an example, the sensor includes: a visual image collector and a radar. For example, the visual image collector may be a binocular camera and / or a three-eye camera, and the radar may be a millimeter wave radar and / or a laser radar.

[0106] Step 102: for sensor data from any sensor, divide the three-dimensional space corresponding to the sensor data into at least one voxel, and acquire a feature of each voxel based on the sensor data to obtain a voxel feature corresponding to the sensor data from the sensor.

[0107] Voxel is the abbreviation of volume pixel, which is the smallest unit in three-dimensional space segmentation. In this embodiment, the shape of the voxel can be a cuboid, and the size of the cuboid can be set according to the needs of the application environment. For the three-dimensional space corresponding to the sensor data from different sensors, the size of the voxel can be different or the same.

[0108] For example, based on binocular visual image data, visual point cloud data in a three-dimensional space is obtained, the three-dimensional space where the visual point cloud data is located is divided into at least one first voxel, and image features of each first voxel are obtained based on the visual point cloud data.

[0109] As an example, visual point cloud data is a set of projection points of visual image pixels projected into three-dimensional space. Each projection point corresponds to three-dimensional spatial position information and a feature vector used to characterize the visual image features of the projection point.

[0110] As an example, the feature vector of the projection point is obtained as follows:

[0111] For any projection point, obtain the feature vector of the pixel point corresponding to the projection point in the left eye visual feature map, or obtain the feature vector of the pixel point corresponding to the projection point in the right eye feature map, and obtain the feature vector of the projection point.

[0112] The left-eye feature map and the right-eye feature map are obtained by extracting features from the left-eye visual image and the right-eye visual image in the binocular visual image respectively;

[0113] As an example, the image feature of the first voxel is obtained as follows:

[0114] For any first voxel,

[0115] Collect each projection point in the visual point cloud data within the first voxel,

[0116] The feature vectors of the collected projection points are spliced ​​in the channel dimension.

[0117] The first feature extraction network is used to extract features from the spliced ​​feature vectors to obtain image features of the first voxel.

[0118] Based on the radar image data, radar point cloud data is obtained, the three-dimensional space where the radar point cloud data is located is divided into at least one second voxel, and a radar feature of each second voxel is obtained based on the radar point cloud image data.

[0119] For example, radar point cloud data is a set of radar points, each radar point corresponds to three-dimensional spatial position information and a feature vector used to characterize the image features of the radar point. The feature vector of the radar point can be obtained by extracting features from the radar point cloud image data.

[0120] As an example, the radar signature of the second voxel is obtained as follows:

[0121] For any second voxel,

[0122] Collect each radar point in the radar point cloud data within the second voxel,

[0123] The feature vectors of the collected radar points are spliced ​​in the channel dimension.

[0124] The second feature extraction network is used to extract features from the spliced ​​feature vectors to obtain the radar features of the second voxel.

[0125] Step 103: for any voxel, the voxel features corresponding to the sensor data from at least two sensors in the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fused features of the voxel, and the process is repeated until the voxel features of all voxels are fused.

[0126] For example, for any voxel, the image feature and radar feature of the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fusion feature of the voxel. This process is repeated until the image feature and radar feature of each voxel are fused.

[0127] As an example, feature fusion within a voxel causes the image feature within the voxel to be fused with the radar feature to obtain a first fused feature, feature fusion between local voxels causes the first fused features of nearby voxels of the voxel to be fused to obtain a second fused feature, and feature fusion between global voxels causes the second fused features of nearby voxels of the voxel and non-nearby voxels to be fused to obtain a third fused feature.

[0128] Among them, the first fusion feature characterizes the fusion features of the projection points and radar points within the voxel, the second fusion feature characterizes the fusion features of the projection points and radar points between voxels in the first range, and the third fusion feature characterizes the fusion features of the projection points and radar points between voxels in the second range, wherein the first range is composed of nearby voxels, and the second range is composed of nearby voxels and non-nearby voxels, that is, the first range is smaller than the second range.

[0129] As an example, feature fusion within a voxel, feature fusion between local voxels, and feature fusion between global voxels are respectively implemented by neural network models. For example, feature fusion within a voxel is implemented by multiple fully connected layers connected in sequence, feature fusion between global voxels is implemented by a neural network model with an attention mechanism, such as a transform model, and feature fusion between local voxels is implemented by a neural network model including at least one maximum pooling layer, at least one first residual network layer, at least one upsampling layer, and at least one second residual network layer.

[0130] As an example, feature fusion within a voxel is implemented as follows:

[0131] For any voxel,

[0132] The image feature of the voxel is spliced ​​with the radar feature in the channel dimension to obtain the spliced ​​feature vector.

[0133] Inputting the concatenated feature vectors into a plurality of sequentially connected fully connected layers, wherein the fully connected layers output the first fused feature of the voxel;

[0134] As an example, feature fusion between global voxels is implemented as follows:

[0135] For any voxel,

[0136] The second fusion feature of the voxel is input into three different groups of fully connected layers to obtain the query value, key value, and feature value respectively.

[0137] Multiply each query value by all key values ​​to get the weight value,

[0138] The feature values ​​are weighted and summed using the weight values ​​to obtain the third fused feature of the voxel.

[0139] As an example, feature fusion between local voxels is implemented as follows:

[0140] For any voxel,

[0141] The first fusion feature of the voxel is input into a neural network model for local inter-voxel feature fusion, and the neural network model outputs the second fusion feature of the voxel.

[0142] See also Figure 2 As shown, Figure 2 A schematic diagram of a neural network model for feature fusion between local voxels, including: a maximum pooling layer, a first residual network layer, an upsampling layer, and a second residual network layer connected in sequence. Given that the input end of the maximum pooling layer and the output end of the upsampling layer have the same feature size, in order to retain the original feature information, the input end of the maximum pooling layer is directly connected to the output end of the upsampling layer, so that the features of the two ends are accumulated and input into the second residual network layer. The neural network model may also include at least one third residual network layer, each third residual network layer is connected in sequence, and each third residual network layer is also connected between the first residual network layer and the upsampling layer connected in sequence, wherein the first residual network layer is connected to the first third residual network layer in each third residual network layer, and the last third residual network layer in each third residual network layer is connected to the upsampling layer. The direct connection also includes at least one fourth residual network layer, and each fourth residual network layer is connected in sequence.

[0143] See also Figure 3 As shown, Figure 3 Another schematic diagram of a neural network model for feature fusion between local voxels, including: at least one maximum pooling layer, at least one first residual network layer, at least one upsampling layer, and at least one second residual network layer, a maximum pooling layer is connected to a first residual network layer to form a pooling residual unit, multiple pooling residual units are connected in sequence, an upsampling layer is connected to a second residual network layer to form an upsampling residual unit, multiple upsampling residual units are connected in sequence, the first residual network layer in the last pooling residual unit is connected to the upsampling layer in the first upsampling residual unit, at least one of the input ends of each maximum pooling layer is directly connected to at least one of the output ends of each upsampling layer, wherein the two directly connected ends have features with the same size, so that the features of the two ends are accumulated and input to the next layer. The neural network model may also include: at least one third residual network layer, each third residual network layer is connected in sequence, the first residual network layer in the last pooling residual unit is connected to the first third residual network layer in each third residual network layer, the last third residual network layer in each third residual network layer is connected to the upsampling layer in the first upsampling residual unit, and the direct connection also includes at least one fourth residual network layer, and each fourth residual network layer is connected in sequence.

[0144] As an example, the number of upsampling layers is less than the number of maximum pooling layers, so that the feature size of the second fused feature is smaller than the feature size of the first fused feature; thus, the method may further include:

[0145] For any voxel, a deconvolution operation is performed on the third fused feature of the voxel so that the fused feature of the voxel has the same feature size as the first fused feature.

[0146] Step 104 : performing obstacle detection on each voxel based on the fusion features of each voxel.

[0147] As an example, for any voxel, the fusion features of the voxel are input into the neural network model for obstacle detection to obtain the probability of the existence of an obstacle in the voxel. When the probability of the existence of an obstacle in the voxel is greater than a set probability threshold, it is determined that an obstacle exists in the voxel.

[0148] It should be understood that, in this embodiment, the size of the first voxel may be the same as or different from the size of the second voxel, and the three-dimensional space where the visual point cloud data is located may be the same as or different from the three-dimensional space where the radar point cloud data is located. When the voxel sizes and / or the three-dimensional spaces where the data are located are different, when performing feature fusion, the first voxel and the second voxel may be corresponded to the same voxel. In addition, the neural network model used for feature fusion and the neural network model used for obstacle detection are both trained models.

[0149] The obstacle detection method for multi-sensor data provided in the embodiment of the present application sequentially fuses the voxel features of each voxel within the voxel, fuses the features between local voxels, and fuses the features between global voxels, and performs obstacle detection based on the fused features of each voxel, so that the features of the multi-sensor data are fused, thereby avoiding being limited to a single sensor data in the obstacle detection process, improving the complementarity of the multi-sensor data in the obstacle detection process, and thus improving the reliability and accuracy of the obstacle detection.

[0150] To facilitate the understanding of the present application, the following is an example of obstacle detection applied to drones and using multi-sensor data from binocular cameras and radars. It should be understood that the present application is not limited to drones, and any movable device that requires obstacle detection can be used, including but not limited to autonomous mobile robots, cleaning robots, vehicles, etc. Multi-sensor data includes but is not limited to binocular visual image data and radar image data.

[0151] See also Figure 4 As shown, Figure 4 This is a flow chart of an obstacle detection method based on multi-sensor data in an embodiment of the present application. The method includes:

[0152] Step 401, obtaining binocular vision image data and radar image data in the current three-dimensional space,

[0153] As an example, the drone collects binocular visual images in the current flight space through a binocular camera, and detects the current flight space through a radar such as a millimeter-wave radar to collect a point cloud image of the current environment. Preferably, the two collections are performed simultaneously.

[0154] Step 402, based on the binocular visual image, obtain visual point cloud data,

[0155] As an example, visual point cloud data can be obtained by:

[0156] Step 4021, extract features from the binocular vision images to obtain binocular vision feature maps.

[0157] As an example, the left eye visual image and the right eye visual image in the binocular visual image are respectively input into a feature extraction network for image feature extraction to obtain a left eye visual feature map and a right eye visual feature map, respectively, wherein the visual feature map represents the image feature information of each pixel point, and the image feature information can be represented by a feature vector. The feature extraction network can be a convolutional neural network.

[0158] Step 4022: perform depth estimation based on the binocular vision feature map to obtain a depth map of the binocular vision feature map.

[0159] As an example, a binocular estimation algorithm is used to estimate the left-eye visual feature map and the right-eye visual feature map in the binocular vision feature map to obtain a depth map, wherein the value of each feature point in the depth map represents the distance from the feature point to the camera, that is, the depth value; the size of the depth map is the same as that of any binocular image, and the feature points correspond to the pixels one by one.

[0160] Step 4023, according to the camera parameters, using the depth value of the feature point obtained by binocular depth estimation, each pixel point in any visual image is projected into a three-dimensional point, thereby obtaining visual point cloud data. Specifically, any 2D pixel point (u, v) in any visual image is projected into a 3D point (x, y, z) that satisfies:

[0161]

[0162]

[0163]

[0164] in, is the depth value of the feature point corresponding to the pixel point (u, v), , is the center coordinate of the camera imaging plane, It is the ratio of focal length to the pixel size of the image in the u and v directions.

[0165] As another example, a visual matching algorithm is used to perform parallax matching on the left eye visual image and the right eye visual image in the binocular visual image to obtain a depth map. The value of each pixel in the depth map represents the distance from the pixel to the camera. Then, according to the camera parameters, the obtained depth value is used to project each pixel in any binocular visual image into a three-dimensional point, and then feature extraction is performed on any binocular visual image in the binocular visual image to obtain the visual feature map of the binocular visual image.

[0166] Step 403 : divide the three-dimensional space formed by the visual point cloud data into a plurality of first voxels, and extract image features in each first voxel.

[0167] In this embodiment, the first voxel is a cube.

[0168] The total number of first voxels is W×H×S, where W is the number of voxels in the width direction of the three-dimensional space formed by the visual point cloud, H is the number of voxels in the height direction of the three-dimensional space formed by the visual point cloud, and S is the number of voxels in the depth direction of the three-dimensional space formed by the visual point cloud.

[0169] As an example, extracting image features within each voxel can be performed as follows:

[0170] The projection points in each first voxel are sampled, and the number of samples meets the first set value. If the number of projection points in the first voxel is less than the first set value, the sampling is repeated. For example, if there are 3 projection points in a first voxel and the first set value is 5, 2 projection points are repeatedly sampled; if the number of projection points in the first voxel is greater than the first set value, one of the projection points is selected for sampling.

[0171] For any sampled projection point, the feature vector of the pixel point corresponding to the projection point is used as the feature vector of the projection point, wherein the feature vector of the pixel point is composed of the feature vector of the pixel point in the left-eye visual feature map and the feature vector of the pixel point in the right-eye visual feature map. For example, the feature vector of the pixel point in the left-eye visual feature map and the feature vector of the pixel point in the right-eye visual feature map are concatenated into a feature vector, and these feature vectors are one-dimensional feature vectors.

[0172] The feature vectors of each projection point sampled in the first voxel are concatenated in the channel dimension to obtain a concatenated feature vector, see Figure 5 As shown, Figure 5It is a schematic diagram of the splicing of the feature vectors of the projection points. For example, the feature vector 1 of the projection point 1 is taken as the first column vector, the feature vector 2 of the projection point 2 is taken as the second column vector, and so on, the feature vector n of the projection point n is taken as the nth column vector, and the nth column vector is obtained as the feature vector after splicing; or, the feature vector 1 of the projection point 1 is taken as the first row vector, the feature vector 2 of the projection point 2 is taken as the second row vector, and so on, the feature vector n of the projection point n is taken as the nth row vector, and the nth row vector is obtained as the feature vector after splicing. It should be understood that the projection points have no order of precedence. During the splicing process, if the feature vectors are adjacent in columns, the column direction is the channel dimension, and if the feature vectors are adjacent in rows, the row direction is the channel dimension.

[0173] The concatenated feature vector is sent to a first feature extraction network, for example, a first multi-layer perceptron, to obtain the image feature of the first voxel. The image feature of the first voxel is a one-dimensional feature vector.

[0174] Step 404 , based on the radar image data, radar point cloud data is acquired, the three-dimensional space formed by the radar point cloud data is divided into a plurality of second voxels, and radar features in each second voxel are extracted.

[0175] Since the radar point in the radar image represents the three-dimensional spatial position information of the radar point, the three-dimensional spatial position information of each radar point in the radar point cloud data can be obtained. Based on the radar image, feature extraction is performed to obtain the image features of each radar point as the feature vector of each radar point.

[0176] In order to facilitate the information fusion of radar point cloud data and visual point cloud data, the code is simplified so that the three-dimensional space formed by the radar point cloud data is the same as the three-dimensional space formed by the visual point cloud data. The size of the second voxel divided in this step is the same as the size of the first voxel divided in step 403, and the number of voxels is the same, that is, the first voxel is the same as the second voxel, and the number of the first voxels is the same as the number of the second voxels.

[0177] The radar points in each second voxel are sampled, and the number of samples is a second set value. If the number of radar points in the second voxel is less than the second set value, the sampling is repeated. The feature vectors of the radar points sampled in the second voxel are spliced ​​in the channel dimension, and the splicing method can be the same as the splicing method of the projection point, and the spliced ​​feature vector is sent to the second feature extraction network, such as the second multi-layer perceptron, to obtain the radar feature of the second voxel. The image feature of the second voxel is a one-dimensional feature vector.

[0178] The above steps 402-403 and step 404 have no order of precedence and can be executed in parallel.

[0179] Step 405 , for the same voxel, perform feature fusion within the voxel, feature fusion between local voxels, and feature fusion between global voxels in sequence on the image feature and the radar feature within the voxel to obtain a voxel fusion feature.

[0180] As an example, the image feature of the first voxel and the radar feature of the second voxel are fused voxel by voxel to obtain a first fused feature, and then the first fused feature is fused between local voxels to obtain a second fused feature, and then the second fused feature is fused between global voxels to obtain a third fused feature, and finally the feature size of the third fused feature is restored to obtain a voxel fused feature. Among them, the feature fusion between local voxels is the feature fusion of each voxel in a small range, and the feature fusion between global voxels is the feature fusion of each voxel in a large range.

[0181] As an example, feature fusion within a voxel can be implemented as follows:

[0182] For any voxel, for example, for any first voxel,

[0183] The image feature of the first voxel and the radar feature of the second voxel corresponding to the first voxel are concatenated in the channel dimension to obtain a concatenated feature vector. For example, see Figure 6 As shown, Figure 6 A schematic diagram of splicing the image feature of the first voxel and the radar feature of the second voxel corresponding to the first voxel, the channel number of the image feature of the first voxel is C1, the channel number of the radar feature of the second voxel corresponding to the first voxel is C2, the radar feature of the second voxel is connected to the image feature of the first voxel, so that the channel number of the spliced ​​feature vector is C3, where C3 is the sum of C1 and C2. If there is no projection point or radar point in the first voxel, the feature value at the corresponding channel is filled with 0;

[0184] The concatenated feature vectors are sent to multiple fully connected layers connected in sequence to obtain the feature vector of the first voxel as the first fusion feature. The number of channels of the feature vector of the first voxel is C4;

[0185] The process is repeated until all first voxels are fused within the voxel to form a 4D feature volume, the feature size of which is W×H×S×C4.

[0186] After intra-voxel fusion, the feature vector of each voxel, i.e., the first fused feature, contains the features of the image and radar in the voxel.

[0187] As an example, feature fusion between local voxels can be achieved as follows:

[0188] For any voxel, for example, for any first voxel,

[0189] A maximum pooling operation, a convolution operation, and an upsampling operation are sequentially performed on the first fused feature of the first voxel, or a maximum pooling operation, multiple consecutive convolution operations, and an upsampling operation are sequentially performed on the first fused feature of the first voxel; and the first fused feature is accumulated with the feature obtained after the upsampling operation and then a convolution operation is performed to obtain a second fused feature of the first voxel, or the first fused feature is convoluted more than once and then accumulated with the feature obtained after the upsampling operation, and a convolution operation is performed on the accumulated feature to obtain the second fused feature of the first voxel.

[0190] The process is repeated until the first fusion features of all first voxels have been fused between local voxels.

[0191] As another example, feature fusion between local voxels can also be implemented as follows:

[0192] The first fused feature of the first voxel is sequentially subjected to a plurality of first operations, each of which includes a maximum pooling operation and a convolution operation as a first operation, and a plurality of second operations, each of which includes an upsampling operation and a convolution operation as a second operation, or at least one convolution operation is further performed between the plurality of first operations and the plurality of second operations, each operation being performed based on an operation result of a previous operation adjacent to the operation; and

[0193] At least one feature among the features obtained by the multiple first operations is accumulated with a feature having the same feature size as the feature among the features obtained after each upsampling operation, and the accumulated feature is then subjected to the convolution operation in the second operation; or, at least one feature among the features obtained by the multiple first operations is subjected to at least one convolution operation, and then accumulated with a feature having the same feature size as the feature among the features obtained after each upsampling operation, and the accumulated feature is subjected to the convolution operation in the second operation;

[0194] The process is repeated until the first fusion features of all first voxels have been fused between local voxels.

[0195] In the above example, the accumulated features have the same feature size, the maximum pooling operation makes the feature size smaller, the convolution operation keeps the feature size unchanged, the upsampling operation makes the feature size larger, and the second fused feature size is smaller than the first fused feature size, so as to help reduce the computational complexity of feature fusion between global voxels; the number of maximum pooling operations, convolution operations, and upsampling operations can control the range of local voxels. The more the number of maximum pooling operations, convolution operations, and upsampling operations, the larger the range of local voxels.

[0196] See also Figure 7 As shown, Figure 7 Schematic diagram of a neural network model for feature fusion between local voxels in this embodiment. The model includes: multiple maximum pooling layers for performing maximum pooling operations, multiple first residual network layers for performing convolution operations, multiple upsampling layers for performing upsampling operations, and multiple second residual network layers for performing convolution operations.

[0197] Among them, a maximum pooling layer is connected to a first residual network layer to form a pooling residual unit for performing a first operation, and multiple pooling residual units are connected in sequence to perform multiple first operations. An upsampling layer is connected to a second residual network layer to form an upsampling residual unit for performing a second operation, and multiple upsampling residual units are connected in sequence to perform multiple second operations.

[0198] The first residual network layer in the last pooled residual unit may also be connected to the first third residual network layer in at least one or more third residual network layers connected in sequence, and the last third residual network layer is connected to the upsampling layer 1 in the first upsampling residual unit.

[0199] The output end of the output feature having the same feature size as the output feature of the upsampling layer 1 among the output features of each first residual network layer has a direct connection (shortcut) 1 with the output end of the upsampling layer 1, and the direct connection path has at least one fourth residual network layer 1 connected in sequence, so that the output feature of the fourth residual network layer 1 is accumulated with the output feature of the upsampling layer 1 to retain the original information, and the accumulated feature is input to the second residual network layer 1, and the output feature of the second residual network layer 1 is input to the upsampling layer 2 connected thereto.

[0200] There is a direct connection 2 between the output end of the upsampling layer 2 and the output end of the output feature having the same feature size as the output feature of the upsampling layer 2 among the output features of each first residual network layer, and there is at least one fourth residual network layer 2 connected on the direct connection path, so that the output feature of the fourth residual network layer 2 is accumulated with the output feature of the upsampling layer 2 to retain the original information, and the accumulated feature is input to the second residual network layer 2, and the output feature of the second residual network layer 2 is input to the upsampling layer 3 connected thereto.

[0201] …

[0202] And so on.

[0203] There is a direct connection n between the output end of the upsampling layer n and the output ends of the output features with the same feature size as the output feature of the upsampling layer n among the features output by each first residual network layer. There is at least one or more connected fourth residual network layers n on this direct connection path, so that the output feature of the fourth residual network layer n is accumulated with the output feature of the upsampling layer n to retain the original information. The accumulated feature is input into the second residual network layer n, and the output feature of the second residual network layer n is input into the connected upsampling layer n+1.

[0204] It should be understood that the direct connection path in the model can be one or more of the n direct connection paths, and the number of direct residual network layers in each direct connection path can be the same or different.

[0205] As an example, the residual network layer uses 3D convolution. After passing through the residual network layer, the feature size remains unchanged. The Maxpooling layer makes the feature size smaller, and the upsampling layer uses linear interpolation or transposed convolution to make the feature size larger. The final output feature volume size of the model is w×h×s×C5. When the number of upsampling layers is less than the number of max pooling layers, the final output feature volume size is smaller than the input feature volume size, that is, the size of the second fusion feature is smaller than the feature size of the first fusion feature, w < W, h < H, s < S, which is beneficial to reducing the computational amount of feature fusion between global voxels.

[0206] After the feature fusion between local voxels, the feature vector of each voxel contains the image and radar features of the nearby voxels of that voxel.

[0207] See Figure 8 as shown Figure 8 is a schematic diagram of feature fusion between global voxels. As an example, the feature fusion between global voxels can be realized by an attention mechanism. Specifically,

[0208] For any first voxel,

[0209] The second fusion feature of the first voxel is respectively input into three groups of fully connected layers to obtain a query value, a key value, and a value,

[0210] Multiply each query value by all key values, and after a normalization operation such as a softmax operation, obtain a weight value w,

[0211] Use the weight value w to perform a weighted sum of all value values to obtain the third fusion feature of the first voxel,

[0212] Repeat until the second fusion features of all first voxels have undergone feature fusion between global voxels.

[0213] The feature size of the third fused feature is the same as the feature size of the second fused feature.

[0214] Since the neural network with attention mechanism can introduce a wide range of information, after the feature fusion of global voxels, the feature vector of each voxel contains the image and radar features of all voxels in 3D space.

[0215] For any first voxel,

[0216] The third fused feature of the first voxel is subjected to multiple 3D deconvolution operations to restore the feature size, and a voxel fused feature of the first voxel is obtained. The size of the voxel fused feature is W×H×S×C6, which is the same as the feature size of the first fused feature.

[0217] The process is repeated until the feature sizes of the third fused features of all the first voxels are restored;

[0218] Among them, multiple 3D deconvolution operations are implemented by multiple 3D deconvolution layers.

[0219] Step 406 , using the voxel fusion features of each voxel, through the neural network model for obstacle detection, the probability of the existence of an obstacle in each voxel is obtained, and whether there is an obstacle in each voxel is determined according to a set probability threshold.

[0220] As an example, for any first voxel, the voxel fusion feature of the first voxel is input into the third multi-layer perceptron to obtain the probability of the existence of an obstacle in the first voxel, and based on the probability threshold, it is determined whether there are obstacles in each voxel of the first voxel. If the probability of the existence of an obstacle in the first voxel is greater than the probability threshold, it is determined that there are obstacles in each voxel of the first voxel; otherwise, it is determined that there are no obstacles in each voxel of the first voxel.

[0221] See also Fig. 9 As shown, Fig. 9 This is an example of obstacle prediction results, where green voxels indicate no obstacles and blue voxels indicate obstacles.

[0222] This embodiment projects the image pixels into three-dimensional space points, and takes voxels as units, and sequentially fuses the image features and radar features within the voxels, the local voxels, and the global voxels, so that each voxel contains global image and radar information, which helps to better judge whether there are obstacles within the voxel, and enables the drone to avoid obstacles under different lighting and weather conditions. On the other hand, in the process of feature fusion between local voxels, the feature size of the feature body is compressed, which reduces the amount of calculation for subsequent global voxel fusion, and helps to apply the obstacle perception function on drone chips with lower computing power.

[0223] See also Fig.10 As shown, Fig.10 This is a schematic diagram of an obstacle detection device based on multi-sensor data in an embodiment of the present application. The device includes:

[0224] A data acquisition module is used to acquire sensor data from two or more sensors. For example, the sensor data includes binocular vision image data and radar image data.

[0225] A feature extraction module is used to divide the three-dimensional space corresponding to the sensor data from any sensor into at least one voxel, and obtain the feature of each voxel based on the sensor data to obtain the voxel feature corresponding to the sensor data from the sensor,

[0226] For example, the feature extraction module includes:

[0227] An image feature acquisition module is used to acquire visual point cloud data in a three-dimensional space based on binocular visual image data, divide the three-dimensional space where the visual point cloud data is located into at least one first voxel, and acquire an image feature of each first voxel based on the visual point cloud data.

[0228] A radar feature acquisition module is used to acquire radar point cloud data based on radar image data, divide the three-dimensional space where the radar point cloud data is located into at least one second voxel, and acquire a radar feature of each second voxel based on the radar point cloud image data.

[0229] The feature fusion module is used to perform feature fusion within the voxel, feature fusion between local voxels, and feature fusion between global voxels for each voxel feature corresponding to the sensor data from at least two sensors in the voxel, such as image feature and radar feature, in sequence, to obtain the fusion feature of the voxel, and perform the process repeatedly until all voxel features are fused.

[0230] The detection module is used to detect obstacles of each voxel based on the fusion features of each voxel.

[0231] As an example, the feature fusion module includes:

[0232] The intra-voxel feature fusion submodule is used to fuse the voxel features in each voxel, such as image features, with radar features to obtain a first fused feature.

[0233] The local voxel feature fusion submodule is used to fuse the first fusion features of each voxel with its neighboring voxels to obtain the second fusion feature.

[0234] The global inter-voxel feature fusion submodule is used to fuse the second fusion features of each voxel, its attached voxels and non-nearby voxels to obtain a third fusion feature.

[0235] For example, the intra-voxel feature fusion submodule includes multiple fully connected layers connected in sequence, and the local inter-voxel feature fusion submodule includes Figure 2 or Figure 3 The neural network model shown, the global inter-voxel feature fusion submodule includes a neural network model with an attention mechanism,

[0236] As an example, the feature fusion module further includes:

[0237] The size restoration submodule is used to perform a deconvolution operation on the third fused feature of each voxel so that the fused feature of each voxel has the same feature size as its first fused feature. For example, the size restoration submodule includes multiple 3D deconvolution layers.

[0238] See also Fig.11 As shown, Fig.11 Another schematic diagram of an obstacle detection device based on multi-sensor data according to an embodiment of the present application. The device comprises: a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the obstacle detection method based on multi-sensor data according to an embodiment of the present application.

[0239] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0240] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0241] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the obstacle detection method based on multi-sensor data in the embodiment of the present application are implemented.

[0242] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0243] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0244] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An obstacle detection method based on multi-sensor data, characterized in that: The method includes: Get sensor data from more than two sensors, For sensor data from any sensor, the three-dimensional space corresponding to the sensor data is divided into at least one voxel, and a feature of each voxel is acquired based on the sensor data to obtain a voxel feature corresponding to the sensor data from the sensor, For any voxel, the voxel features corresponding to the sensor data from at least two sensors in the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fused features of the voxel, and the process is repeated until the voxel features of all voxels are fused. Based on the fusion features of each voxel, obstacle detection is performed on each voxel; in, The feature fusion in the voxel enables the voxel features of the voxel to be fused to obtain a first fused feature, The feature fusion between the local voxels enables the first fusion features of the nearby voxels of the voxel to be fused to obtain the second fusion feature, The feature fusion between the global voxels enables the second fused features of the nearby voxels and the non-nearby voxels of the voxel to be fused to obtain a third fused feature; The feature fusion within the voxel includes: For any voxel, The voxel features of the voxel are spliced ​​in the channel dimension to obtain the spliced ​​feature vector. Inputting the concatenated feature vectors into a plurality of sequentially connected fully connected layers, wherein the fully connected layers output the first fused feature of the voxel; The feature fusion between global voxels includes: For any voxel, The second fusion feature of the voxel is input into three different groups of fully connected layers to obtain the query value, key value, and feature value respectively. Multiply each query value by all key values ​​and normalize them to get the weight value. The feature values ​​are weighted and summed using the weight value to obtain the third fusion feature of the voxel; The feature fusion between local voxels includes: For any voxel, The first fused feature of the voxel is input into a neural network model for local inter-voxel feature fusion, and the neural network model outputs the second fused feature of the voxel.

2. The obstacle detection method according to claim 1, characterized in that: The neural network model includes: a maximum pooling layer, a first residual network layer, an upsampling layer, and a second residual network layer connected in sequence. The input end of the maximum pooling layer is directly connected to the output end of the upsampling layer, wherein the features of both ends of the direct connection have the same size, so that the features of the two ends are accumulated and input into the second residual network layer; Among them, the number of maximum pooling layers, residual network layers, and upsampling layers are used to determine the range of nearby voxels.

3. The obstacle detection method according to claim 1, characterized in that: The neural network model includes: at least one maximum pooling layer, at least one first residual network layer, at least one upsampling layer, and at least one second residual network layer. A maximum pooling layer is connected to a first residual network layer to form a pooling residual unit, multiple pooling residual units are connected in sequence, an upsampling layer is connected to a second residual network layer to form an upsampling residual unit, and multiple upsampling residual units are connected in sequence. The first residual network layer in the last pooled residual unit is connected to the upsampling layer in the first upsampling residual unit. At least one of the input ends of each maximum pooling layer is directly connected to at least one of the output ends of each upsampling layer, wherein the features of the two ends of the direct connection have the same size, so that the features of the two ends are accumulated and input to the next layer. Among them, the number of maximum pooling layers, residual network layers, and upsampling layers are used to determine the range of nearby voxels.

4. The obstacle detection method according to claim 2 or 3, characterized in that: The neural network model further includes: at least one third residual network layer, each third residual network layer is connected in sequence, The first residual network layer and the upsampling layer connected in sequence are further connected to each third residual network layer, wherein the first residual network layer is connected to the first third residual network layer in each third residual network layer, and the last third residual network layer in each third residual network layer is connected to the upsampling layer. The first residual network layer in the last pooled residual unit is connected to the first third residual network layer in each third residual network layer, and the last third residual network layer in each third residual network layer is connected to the upsampling layer in the first upsampling residual unit. The direct connection also includes at least one fourth residual network layer, and each fourth residual network layer is connected in sequence.

5. The obstacle detection method according to claim 4, characterized in that: The number of upsampling layers is smaller than the number of maximum pooling layers, so that the feature size of the second fused feature is smaller than the feature size of the first fused feature; The step of sequentially fusing the image feature of the voxel with the radar feature within the voxel, fusing the feature between local voxels, and fusing the feature between global voxels further includes: For any voxel, a deconvolution operation is performed on the third fused feature of the voxel so that the fused feature of the voxel has the same feature size as the first fused feature.

6. The obstacle detection method according to claim 1, characterized in that: The sensor data includes: binocular vision image data and radar image data; For sensor data from any sensor, dividing the three-dimensional space corresponding to the sensor data into at least one voxel, and acquiring the feature of each voxel based on the sensor data to obtain the voxel feature corresponding to the sensor data from the sensor, includes: Based on the binocular visual image data, visual point cloud data in a three-dimensional space is obtained, the three-dimensional space where the visual point cloud data is located is divided into at least one first voxel, and image features of each first voxel are obtained based on the visual point cloud data. Based on the radar image data, radar point cloud data is acquired, the three-dimensional space where the radar point cloud data is located is divided into at least one second voxel, and a radar feature of each second voxel is acquired based on the radar point cloud image data.

7. The obstacle detection method according to claim 1, characterized in that: The step of sequentially performing intra-voxel feature fusion, local voxel feature fusion, and global voxel feature fusion on each voxel feature corresponding to the sensor data from at least two sensors in the voxel to obtain the fusion feature of the voxel, and repeatedly performing the step until the voxel features of each voxel are fused, includes: For any voxel, the image feature and radar feature of the voxel are sequentially fused within the voxel, between local voxels, and between global voxels to obtain the fusion feature of the voxel. This process is repeated until the image feature and radar feature of each voxel are fused. The step of splicing the voxel features of the voxel in the channel dimension includes: The image feature of the voxel is connected to the radar feature. In the case where the voxel has no image feature or no radar feature, the feature value in the corresponding channel is set to 0.

8. The obstacle detection method according to claim 6, characterized in that: The step of acquiring the image feature of each first voxel based on the visual point cloud data comprises: For any first voxel, Collect each projection point in the visual point cloud data within the first voxel, The feature vectors of the collected projection points are spliced ​​in the channel dimension. Using the first feature extraction network, feature extraction is performed on the spliced ​​feature vector to obtain the image feature of the first voxel. Among them, the eigenvector of the projection point is obtained as follows: For any projection point, obtain the feature vector of the pixel point corresponding to the projection point in any visual feature map, and obtain the feature vector of the projection point. The visual feature map is obtained based on feature extraction of any one of the binocular visual images; The acquiring the radar feature of each second voxel based on the radar point cloud image data comprises: For any second voxel, Collect each radar point in the radar point cloud data within the second voxel, The feature vectors of the collected radar points are spliced ​​in the channel dimension. The second feature extraction network is used to extract features from the spliced ​​feature vectors to obtain the radar features of the second voxel. Among them, the feature vector of the radar point is obtained as follows: For any radar point, a feature vector of a radar point corresponding to the radar point in a radar feature map is obtained, wherein the radar feature map is obtained by performing feature extraction on radar point cloud image data.

9. The obstacle detection method according to claim 8, characterized in that: The collecting of each projection point in the visual point cloud data in the first voxel includes: When the number of projection points in the first voxel is less than the set first value, the projection points are repeatedly collected so that the number of collected projection points is equal to the first value. When the number of projection points in the first voxel is not less than the set first value, collecting each projection point in the first voxel so that the number of collected projection points is equal to the first value; The step of splicing the feature vectors of the collected projection points in the channel dimension includes: The feature vectors of each collected projection point are used as row vectors, and the row vectors are adjacent to each other row by row, or, The feature vectors of each collected projection point are respectively used as column vectors, and the column vectors are adjacent to each other column by column; The collecting of each radar point in the radar point cloud data within the second voxel includes: When the number of radar points in the second voxel is less than the set second value, the radar points are repeatedly collected so that the number of collected radar points is equal to the second value. When the number of radar points in the second voxel is not less than the set second value, collecting each radar point in the second voxel so that the number of collected radar points is equal to the second value; The step of splicing the feature vectors of the collected radar points in the channel dimension includes: The feature vectors of each radar point collected are used as row vectors, and the row vectors are adjacent to each other row by row, or, The characteristic vectors of each collected radar point are respectively used as column vectors, and the column vectors are adjacent to each other column by column.

10. The obstacle detection method according to claim 6, characterized in that: The first voxel has the same size as the second voxel, the three-dimensional space where the visual point cloud data is located is the same as the three-dimensional space where the radar point cloud data is located, The obstacle detection of each voxel based on the fusion features of each voxel includes: For any voxel, The fusion features of the voxel are input into the neural network model for obstacle detection to obtain the probability of the existence of an obstacle in the voxel. When the probability that an obstacle exists in the voxel is greater than a set probability threshold, it is determined that an obstacle exists in the voxel.

11. An obstacle detection device based on multi-sensor data, characterized in that: The device includes: A data acquisition module, used to acquire sensor data from more than two sensors, A feature acquisition module is used to divide the three-dimensional space corresponding to the sensor data from any sensor into at least one voxel, and acquire the feature of each voxel based on the sensor data to obtain the voxel feature corresponding to the sensor data from the sensor, The feature fusion module is used to perform feature fusion within the voxel, feature fusion between local voxels, and feature fusion between global voxels for each voxel feature corresponding to the sensor data from at least two sensors in the voxel, to obtain the fusion feature of the voxel, and to perform the fusion repeatedly until the voxel features of each voxel are fused. The detection module is used to detect obstacles at each voxel based on the fusion features of each voxel. in, The feature fusion in the voxel enables the voxel features of the voxel to be fused to obtain a first fused feature, The feature fusion between the local voxels enables the first fusion features of the nearby voxels of the voxel to be fused to obtain the second fusion feature, The feature fusion between the global voxels enables the second fused features of the nearby voxels and the non-nearby voxels of the voxel to be fused to obtain a third fused feature; The feature fusion within the voxel includes: For any voxel, The voxel features of the voxel are spliced ​​in the channel dimension to obtain the spliced ​​feature vector. Inputting the concatenated feature vectors into a plurality of sequentially connected fully connected layers, wherein the fully connected layers output the first fused feature of the voxel; The feature fusion between global voxels includes: For any voxel, The second fusion feature of the voxel is input into three different groups of fully connected layers to obtain the query value, key value, and feature value respectively. Multiply each query value by all key values ​​and normalize them to get the weight value. The feature values ​​are weighted and summed using the weight value to obtain the third fusion feature of the voxel; The feature fusion between local voxels includes: For any voxel, The first fused feature of the voxel is input into a neural network model for local inter-voxel feature fusion, and the neural network model outputs the second fused feature of the voxel.

Images