Vehicle detection method and system based on graph neural network and multi-sensor fusion

Abstract

The application discloses a vehicle detection method, system and equipment based on a graph neural network and multi-sensor fusion, wherein the method comprises the following steps: acquiring single-frame point cloud and single-frame image data of a region of interest in front of a vehicle; extracting global image features from the single-frame image data, and then performing feature enhancement on the single-frame point cloud by using the global image features to obtain a first single-frame point cloud; performing context feature extraction on the first single-frame point cloud by using a graph neural network to obtain a corresponding deep feature image; identifying and acquiring all target vehicle information in the deep feature image, and then marking and visualizing the all target vehicle information in the single-frame point cloud and the single-frame image data. The application can solve the noise interference problem that may exist in the point cloud data by combining the global image features and the attention fusion mechanism to perform the feature enhancement operation on the point cloud, and can improve the vehicle detection precision by fully mining the point cloud features by using the graph neural network.

Description

Technical Field

[0001] This invention relates to the field of vehicle detection technology, specifically to a vehicle detection method, system, and device based on graph neural networks and multi-sensor fusion. Background Technology

[0002] Vehicle detection is a crucial function in the perception technology of intelligent driving vehicles. Its accuracy affects the accuracy of prediction, decision-making, and control in autonomous vehicles, thereby impacting their driving safety. However, current LiDAR-based vehicle detection algorithms typically use 2D or 3D convolution as feature extraction methods, which cannot fully exploit point cloud features. Furthermore, the lack of image semantic information to aid analysis, coupled with significant interference from background point cloud features, leads to low vehicle detection accuracy. Summary of the Invention

[0003] This invention provides a vehicle detection method, system, and device based on graph neural networks and multi-sensor fusion to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] Firstly, a vehicle detection method based on graph neural networks and multi-sensor fusion is provided, the method comprising:

[0005] Acquire single-frame point cloud and single-frame image data that fall within the region of interest in front of the vehicle;

[0006] Global image features are extracted from the single-frame image data, and then the global image features are used to enhance the features of the single-frame point cloud to obtain the first single-frame point cloud.

[0007] The first single-frame point cloud is subjected to contextual feature extraction using a graph neural network to obtain the corresponding deep feature image;

[0008] All target vehicle information is identified and obtained from the deep feature image, and then marked and visualized in the single-frame point cloud and the single-frame image data based on all target vehicle information.

[0009] Furthermore, acquiring single-frame point cloud and single-frame image data falling within the region of interest in front of the vehicle includes:

[0010] Acquire a single frame of raw point cloud data of the front of the vehicle collected by the LiDAR, and simultaneously acquire a single frame of raw image data of the front of the vehicle collected by the RGB camera.

[0011] The origin of the lidar coordinate system coincides with the origin of the pixel coordinate system, and the region of interest is constructed based on the coincident origin.

[0012] Extract all point cloud data falling within the region of interest from the single-frame raw point cloud to form a single-frame point cloud;

[0013] Extract single-frame image data that falls within the region of interest from the original single-frame image data.

[0014] Further, the step of using the global image features to perform feature enhancement on the single-frame point cloud to obtain the first single-frame point cloud includes:

[0015] Transform each point cloud data in the single-frame point cloud into a pixel coordinate system;

[0016] Based on the global image features, each point cloud data after conversion is assigned a corresponding image feature, and then the feature enhancement of each point cloud data after conversion is performed by combining the attention fusion mechanism, thereby obtaining the first single frame point cloud.

[0017] Further, assigning corresponding image features to each converted point cloud data based on the global image features includes:

[0018] For any transformed point cloud data, determine whether the global image features contain image features corresponding to the position of the transformed point cloud data;

[0019] If so, the image features are assigned to the converted point cloud data;

[0020] If not, select four converted point cloud data that are adjacent to the converted point cloud data from the converted single-frame point cloud, and obtain four image features corresponding to the positions of the four converted point cloud data from the global image features. Then, use the bilinear interpolation algorithm to perform interpolation operations on the four image features, and finally assign the image features obtained from the interpolation operation to the converted point cloud data.

[0021] Furthermore, the feature enhancement of each transformed point cloud data using the attention fusion mechanism includes:

[0022] For any transformed point cloud data, the first fully connected layer is used to extract features from the transformed point cloud data to obtain the first feature data;

[0023] The second fully connected layer is used to extract image features corresponding to the transformed point cloud data to obtain second feature data.

[0024] After adjusting the first feature data and the second feature data to have the same number of channels, the features are added together to obtain the third feature data;

[0025] The third feature data is extracted using a third fully connected layer, and the feature extraction results are normalized using an activation function to obtain attention weights.

[0026] The image features corresponding to the transformed point cloud data are multiplied by the attention weights, and then the multiplication result is concatenated with the transformed point cloud data to obtain the enhanced point cloud data.

[0027] Furthermore, the step of using a graph neural network to extract contextual features from the first single-frame point cloud to obtain the corresponding deep feature image includes:

[0028] The region of interest is divided into several voxels, and the single-frame point cloud is distributed within the several voxels;

[0029] Using the single-frame point cloud and the first single-frame point cloud, calculate the features of several voxels corresponding to the several voxels, and then use the K nearest neighbor algorithm to construct several sub-graphs corresponding to the several voxels.

[0030] The graph neural network is used to extract features from the sub-graphs, and then the feature extraction results are converted into two-dimensional feature images.

[0031] The two-dimensional feature image is convolved to obtain a deep feature image.

[0032] Further, the step of calculating the several voxel features corresponding to the several voxels using the single-frame point cloud and the first single-frame point cloud includes:

[0033] For any voxel, multiple point cloud data distributed within the voxel are obtained from the single-frame point cloud, and multiple enhanced point cloud data associated with the multiple point cloud data are obtained from the first single-frame point cloud.

[0034] The multiple enhanced point cloud data are converted to the lidar coordinate system and then stitched together with the multiple point cloud data to obtain multiple stitched point cloud data.

[0035] The multiple stitched point cloud data are averaged to obtain the voxel features corresponding to the voxels.

[0036] Furthermore, the step of constructing several subgraphs corresponding to the several voxels using the K-nearest neighbor algorithm includes:

[0037] For any voxel, based on the voxel features corresponding to the voxels, the K nearest neighbor algorithm is used to search for the first set of voxels that are adjacent to the voxel. Then, the voxel is connected to each voxel in the first set of voxels to obtain the first set of side lengths.

[0038] The K-nearest neighbor algorithm is used to search for a second set of voxels that are adjacent to the voxels in addition to the first set of voxels. Then, the voxels are connected to each voxel in the second set of voxels to obtain the second set of side lengths.

[0039] The K-nearest neighbor algorithm is used to search for a third set of voxels that is adjacent to the voxel, excluding the first set of voxels and the second set of voxels. Then, the voxel is connected to each voxel in the third set of voxels to obtain the third set of side lengths.

[0040] Based on the first set of side lengths, the second set of side lengths, and the third set of side lengths, construct the subgraph corresponding to the voxel.

[0041] Secondly, a vehicle detection system based on graph neural networks and multi-sensor fusion is provided, the system comprising:

[0042] The data acquisition module is used to acquire single-frame point cloud and single-frame image data of the area of ​​interest in front of the vehicle;

[0043] The point cloud enhancement module is used to extract global image features from the single-frame image data, and then use the global image features to enhance the single-frame point cloud to obtain the first single-frame point cloud.

[0044] The feature extraction module is used to extract contextual features from the first single-frame point cloud using a graph neural network to obtain the corresponding deep feature image.

[0045] The target recognition module is used to identify and acquire all target vehicle information in the deep feature image, and then mark and visualize the single-frame point cloud and the single-frame image data according to all target vehicle information.

[0046] Thirdly, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the vehicle detection method based on graph neural networks and multi-sensor fusion as described in the first aspect.

[0047] The present invention has at least the following beneficial effects: by constructing regions of interest to selectively extract from the acquired raw point cloud and raw image data, the computation speed can be improved; by assigning associated image features to each point cloud data and further combining attention fusion mechanism to perform feature enhancement operations on the point cloud data, it helps to solve the noise interference problem that may exist in the point cloud data; finally, by using graph neural networks to perform multi-scale feature extraction operations on the enhanced point cloud, the point cloud features can be fully explored, thereby improving the vehicle detection accuracy. Attached Figure Description

[0048] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0049] Figure 1 This is a flowchart illustrating a vehicle detection method based on graph neural networks and multi-sensor fusion in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram illustrating the visualization effect of the target vehicle marking in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the composition of a vehicle detection system based on graph neural network and multi-sensor fusion in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the hardware structure of the computer device in an embodiment of this disclosure. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," "third," "fourth," etc., used in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed and are inherent to these processes, methods, products, or apparatuses.

[0055] Please refer to Figure 1 , Figure 1 This is a schematic flowchart of a vehicle detection method based on graph neural networks and multi-sensor fusion provided by an embodiment of the present invention. The method includes the following:

[0056] Step S110: Acquire single-frame point cloud and single-frame image data of the area of ​​interest in front of the vehicle;

[0057] Step S120: Extract global image features from the single-frame image data, and then use the global image features to perform feature enhancement on the single-frame point cloud to obtain the first single-frame point cloud;

[0058] Step S130: Use a graph neural network to extract contextual features from the first single-frame point cloud to obtain the corresponding deep feature image;

[0059] Step S140: Identify and obtain all target vehicle information in the deep feature image, and then mark and visualize the single-frame point cloud and the single-frame image data according to all target vehicle information.

[0060] In this embodiment of the invention, an RGB camera and a LiDAR are installed on the vehicle, and the RGB camera and the LiDAR are configured to collect data only facing forward of the vehicle. The specific implementation process of the above step S110 includes the following:

[0061] Step S111: Obtain the single-frame raw image data captured by the RGB camera, and store the single-frame raw image data in the data format of H1×W1×3, where H1 is the height of the single-frame raw image data, W1 is the width of the single-frame raw image data, and 3 refers to the R (red) channel value, G (green) channel value and B (blue) channel value in the single-frame raw image data.

[0062] Step S112: Obtain the single-frame original point cloud acquired by the lidar, and store the single-frame original point cloud in an N×4 data format, where N is the number of all point cloud data contained in the single-frame original point cloud, and 4 represents the X-axis coordinate value, Y-axis coordinate value, and Z-axis coordinate value in the lidar coordinate system carried by each point cloud data, as well as its corresponding reflection intensity; it should be noted that the single-frame original point cloud and the single-frame original image data are time-synchronized.

[0063] Step S113: Since the pixel coordinate system where the original single-frame image data is located coincides with the lidar coordinate system where the original single-frame point cloud is located, the region of interest is established with the coincident origin as the reference point, so that the distribution range of the region of interest in the X-axis direction is [0, 70.4m], the distribution range in the Y-axis direction is [-40m, 40m] and the distribution range in the Z-axis direction is [-3m, 1m]. That is, the region of interest is actually a cuboid with a length L of 70.4 meters, a width W of 80 meters and a height H of 4 meters.

[0064] Step S114: Since the region of interest is a three-dimensional region, a plane corresponding to the shooting angle of the RGB camera is selected from the region of interest, and then the single-frame image data that can be covered by the plane is extracted from the single-frame original image data with a larger data coverage.

[0065] Step S115: Select all point cloud data that the region of interest can cover from the single-frame original point cloud with a larger data distribution range, and then output all the point cloud data as a single-frame point cloud.

[0066] In this embodiment of the invention, the feature extraction process mentioned in step S120 above includes the following:

[0067] Step S121: Call the pre-trained convolutional neural network model, which includes a residual network (i.e., a ResNet-50 network) and a feature pyramid network.

[0068] Step S122: Extract multiple low-level image features at different scales from the single-frame image data using the residual network, and then fuse the multiple low-level image features at different scales using the feature pyramid network to obtain global image features. The global image features can reflect richer image semantic information contained in the single-frame image data.

[0069] It should be noted that by building the residual network in the convolutional neural network model, the problems of convergence difficulties and network performance degradation that occur as the number of network layers increases can be solved.

[0070] Based on this, the feature enhancement process mentioned in step S120 above includes the following:

[0071] Step S123: Based on the intrinsic and extrinsic parameter matrices of the RGB camera obtained through the existing sensor calibration method, each point cloud data in the single-frame point cloud is mapped from the lidar coordinate system to the pixel coordinate system for representation. Specifically, for any point cloud data in the single-frame point cloud, the coordinate information of the point cloud data in the lidar coordinate system is recorded as the first coordinate information. First, the first coordinate information is multiplied by the extrinsic parameter matrix to obtain the coordinate information of the point cloud data in the camera coordinate system, which is recorded as the second coordinate information. Then, the second coordinate information is multiplied by the intrinsic parameter matrix to obtain the coordinate information of the point cloud data in the pixel coordinate system.

[0072] The intrinsic parameter matrix describes parameters related to the characteristics of the RGB camera itself, such as the pixel size and focal length of the RGB camera, while the extrinsic parameter matrix includes rotation and translation matrices.

[0073] Step S124: Based on the global image features, assign associated image features to each point cloud data after mapping;

[0074] Step S125: Combining the image features associated with each mapped point cloud data, perform feature enhancement operations on each mapped point cloud data through an attention fusion mechanism to obtain the enhanced first single-frame point cloud.

[0075] In this embodiment of the invention, since the raster map formed by the mapped single-frame point cloud may have a different size than the feature map formed by the global image features, the specific implementation process of the above step S124 includes the following:

[0076] Step S124.1: For any mapped point cloud data, identify whether there is an image feature in the global image features whose location coincides with the location of the mapped point cloud data; if it exists, directly assign the image feature to the mapped point cloud data; if it does not exist, proceed to step S124.2.

[0077] Step S124.2: Search in the mapped single-frame point cloud to obtain four mapped point cloud data that are closer to the mapped point cloud data. At the same time, it is limited that there should be four image features in the global image features, wherein the location of each image feature corresponds to the location of a mapped point cloud data found in the search.

[0078] Step S124.3: Perform interpolation operation on the four image features associated with the four mapped point cloud data through bilinear interpolation algorithm to obtain new image features, and then assign the new image features to the mapped point cloud data.

[0079] Repeat steps S124.1 to S124.3 until each mapped point cloud data contained in the mapped single-frame point cloud is assigned an associated image feature.

[0080] More specifically, the coordinate information of the four mapped point cloud data in the pixel coordinate system is preferentially set as (u1,v1), (u1,v2), (u2,v1), and (u2,v2), respectively. Simultaneously, the four image features associated with the four mapped point cloud data are set as F(u1,v1), F(u1,v2), F(u2,v1), and F(u2,v2), respectively. The implementation process of step S124.3 includes the following:

[0081] Step S124.3.1: Perform linear interpolation along the U-axis direction of the pixel coordinate system to obtain the image feature F(u', v1) at the position (u', v1). The corresponding linear interpolation formula is:

[0082]

[0083] Step S124.3.2: Perform linear interpolation along the U-axis direction of the pixel coordinate system to obtain the image feature F(u', v2) at the position (u', v2). The corresponding linear interpolation formula is:

[0084]

[0085] Step S124.3.3: Perform linear interpolation along the V-axis direction of the pixel coordinate system to obtain the image feature F(u0, v0) at the position (u0, v0). The corresponding linear interpolation formula is:

[0086]

[0087] Among them, (u', v1) is a point on the line v = v1, (u', v2) is a point on the line v = v2, and u1 < u' < u2. (u0, v0) is the coordinate information of the mapped point cloud data in the pixel coordinate system, and F(u0, v0) is the new image feature.

[0088] In the embodiment of the present invention, due to environmental influences such as object occlusion and light intensity during vehicle driving, in order to eliminate the possible noise interference problem in the single-frame original point cloud collected by the lidar, the specific implementation process of the above step S125 is as follows:

[0089] Step S125.1: For any mapped point cloud data, input the mapped point cloud data into the first fully connected layer to perform feature extraction operations to obtain first feature data;

[0090] Step S125.2: Input the image feature associated with the mapped point cloud data into the second fully connected layer to perform feature extraction operations to obtain second feature data;

[0091] Step S125.3: Perform a channel number adjustment operation on the first feature data and the second feature data. Specifically, if it is determined that the channel number of the first feature data is greater than the channel number of the second feature data, simply increase the dimensionality of the second feature data until its channel number is equal to the channel number of the first feature data, and then execute step S125.4; or, if it is determined that the channel number of the first feature data is less than the channel number of the second feature data, simply increase the dimensionality of the first feature data until its channel number is equal to the channel number of the second feature data, and then execute step S125.4; or, if it is determined that the channel number of the first feature data is equal to the channel number of the second feature data, directly execute step S125.4.

[0092] Step S125.4: Perform a feature addition operation on the first feature data and the second feature data with the same number of current channels to obtain the third feature data;

[0093] Step S125.5: Input the third feature data into the third fully connected layer to perform feature extraction, and then normalize the feature extraction results through an activation function (preferably the sigmoid function in this invention) to obtain attention weights;

[0094] Step S125.6: Multiply the attention weights with the image features associated with the mapped point cloud data, and then concatenate the result of the multiplication with the mapped point cloud data to obtain the enhanced point cloud data.

[0095] Repeat steps S125.1 to S125.6 until the feature enhancement operation for each mapped point cloud data contained in the mapped single-frame point cloud is completed, thereby obtaining the enhanced first single-frame point cloud.

[0096] In this embodiment of the invention, the specific implementation process of step S130 includes the following:

[0097] Step S131: Divide the region of interest along the X-axis, Y-axis, and Z-axis of the lidar coordinate system to obtain several voxels of equal size. The corresponding calculation formula is as follows:

[0098]

[0099] Where M is the number of the aforementioned voxels, and V L The length of each voxel, V W For the width of each voxel, V H The height of each voxel, and the present invention preferably sets V L =0.05m, V W=0.05m and V H =0.1m; It should be noted that all point cloud data contained in the single frame point cloud still fall within the aforementioned voxels;

[0100] Step S132: Combine the first single-frame point cloud and the single-frame point cloud to calculate the several voxel features corresponding to the several voxels.

[0101] Step S133: Based on the features of the voxels corresponding to the voxels, construct several sub-graphs corresponding to the voxels using the K nearest neighbor algorithm.

[0102] Step S134: Perform feature extraction operations on the several subgraphs using a graph neural network to obtain several feature data;

[0103] Step S135: Convert the plurality of feature data into a three-dimensional feature image represented in three-dimensional space and denoted as [W',H',D',C'], then convert the three-dimensional feature image into a two-dimensional feature image by multiplying the dimensions and denoted as [W',H',D'×C'], and finally perform a two-dimensional convolution operation on the two-dimensional feature image to obtain a deep feature image and denoted as [W',H',C”];

[0104] Wherein, W' is the width of the three-dimensional feature image, and the widths of the three-dimensional feature image, the two-dimensional feature image, and the deep feature image are the same; H' is the height of the three-dimensional feature image, and the heights of the three-dimensional feature image, the two-dimensional feature image, and the deep feature image are the same; D' is the depth of the three-dimensional feature image; C' is the number of channels of the three-dimensional feature image; and C” is the number of channels of the deep feature image.

[0105] More specifically, the implementation process of step S132 above includes the following:

[0106] Step S132.1: For any voxel among the plurality of voxels, extract all point cloud data falling inside the voxel from the single frame point cloud, and record them as N1 point cloud data.

[0107] Step S132.2: Extract N enhanced point cloud data that are directly associated with the N1 point cloud data from the first single-frame point cloud;

[0108] Step S132.3: Based on the intrinsic parameter matrix and extrinsic parameter matrix obtained in step S123 above, the N enhanced point cloud data are mapped from the pixel coordinate system to the lidar coordinate system for representation. Specifically, for any one of the N enhanced point cloud data, the coordinate information of the enhanced point cloud data in the pixel coordinate system is recorded as the third coordinate information. First, the third coordinate information is multiplied by the inverse matrix of the intrinsic parameter matrix to obtain the coordinate information of the enhanced point cloud data in the camera coordinate system and recorded as the fourth coordinate information. Then, the fourth coordinate information is multiplied by the inverse matrix of the extrinsic parameter matrix to obtain the coordinate information of the enhanced point cloud data in the lidar coordinate system.

[0109] Step S132.4: Perform a stitching operation between the N mapped enhanced point cloud data and the N1 point cloud data to obtain N1 stitched point cloud data.

[0110] Step S132.5: Perform a mean averaging operation on the N1 stitched point cloud data to obtain the voxel features corresponding to the voxels. Specifically, since each stitched point cloud data carries X-axis, Y-axis, and Z-axis coordinate values ​​in the lidar coordinate system, the N1 X-axis coordinate values ​​carried in the N1 stitched point cloud data are averaged to obtain the first X-axis coordinate value, the N1 Y-axis coordinate values ​​carried in the N1 stitched point cloud data are averaged to obtain the first Y-axis coordinate value, and the N1 Z-axis coordinate values ​​carried in the N1 stitched point cloud data are averaged to obtain the first Z-axis coordinate value. Then, the first X-axis coordinate value, the first Y-axis coordinate value, and the first Z-axis coordinate value are output as the voxel features.

[0111] Repeat steps S132.1 to S132.5 until the voxel characteristics corresponding to each voxel in the plurality of voxels are calculated.

[0112] More specifically, the implementation process of step S133 above includes the following:

[0113] Step S133.1: For any one of the several voxels, define the voxel as a vertex voxel, and calculate the voxel feature distance between the vertex voxel and the other voxels among the several voxels;

[0114] Step S133.2: Search the voxels using the K nearest neighbor algorithm to obtain 9 voxels that are closer to the vertex voxel and use them as the first voxel set at the first scale. Then connect each of the 9 voxels to the vertex voxel to obtain 9 side lengths and use them as the first side length set.

[0115] Step S133.3: Automatically ignore the 9 voxels, and continue to search among the voxels using the K nearest neighbor algorithm to obtain 14 voxels that are closer to the vertex voxel and use them as the second voxel set at the second scale. Then, connect each of the 14 voxels to the vertex voxel to obtain 14 side lengths and use them as the second side length set.

[0116] Step S133.4: Automatically ignore the 9 voxels and the 14 voxels, and continue to search among the voxels using the K nearest neighbor algorithm to obtain 20 voxels that are closer to the vertex voxel and use them as the third voxel set at the third scale. Then, connect each of the 20 voxels to the vertex voxel to obtain 20 side lengths and use them as the third side length set.

[0117] Step S133.5: By combining the first set of side lengths, the second set of side lengths, and the third set of side lengths, the subgraph corresponding to the vertex voxel can be constructed.

[0118] Repeat steps S133.1 to S133.5 until a subgraph corresponding to each voxel in the plurality of voxels is constructed.

[0119] More specifically, the implementation process of step S134 above includes the following:

[0120] Step S134.1: Call the pre-trained graph neural network. The graph neural network includes a feature extraction network and an aggregation function. The feature extraction network is internally configured with a one-dimensional convolutional layer, a ReLU (Rectified Linear Unit) layer, a BN (Batch Normalization) layer and a self-attention layer connected in sequence according to the data transmission order. The aggregation function is preferably a maximum value function.

[0121] Step S134.2: For any one of the subgraphs, obtain the voxel features corresponding to the vertex voxels associated with the subgraph and record them as vertex voxel features; obtain the 9 voxel features corresponding to the 9 voxels at the first scale of the subgraph; obtain the 14 voxel features corresponding to the 14 voxels at the second scale of the subgraph; and obtain the 20 voxel features corresponding to the 20 voxels at the third scale of the subgraph.

[0122] Step S134.3: Input the vertex voxel features and the 9 voxel features into the feature extraction network to obtain 9 edge features generated after the vertex voxel is connected to the 9 voxels respectively. Then, the 9 edge features are aggregated by the aggregation function to obtain the first vertex voxel features at the first scale of the subgraph.

[0123] Step S134.4: Input the vertex voxel features and the 14 voxel features into the feature extraction network to obtain 14 edge features generated after connecting the vertex voxels with the 14 voxels respectively. Then, aggregate the 14 edge features through the aggregation function to obtain the second vertex voxel features at the second scale of the subgraph.

[0124] Step S134.5: Input the vertex voxel features and the 20 voxel features into the feature extraction network to obtain 20 edge features generated after connecting the vertex voxels with the 20 voxels respectively. Then, aggregate the 20 edge features through the aggregation function to obtain the third vertex voxel features at the third scale of the subgraph.

[0125] Step S134.6: Output the first vertex voxel feature, the second vertex voxel feature, and the third vertex voxel feature as the feature data corresponding to the subgraph;

[0126] Repeat steps S134.2 to S134.6 until the feature data corresponding to each subgraph in the plurality of subgraphs is output.

[0127] In this embodiment of the invention, the target vehicle identification process mentioned in step S140 above includes the following:

[0128] Step S141: Call the pre-trained target recognition network. The target recognition network has a first convolutional layer and a second convolutional layer with a kernel size of 1×1. The first convolutional layer mainly serves as the category prediction branch of the target recognition network, and the second convolutional layer mainly serves as the parameter regression branch of the target recognition network.

[0129] Step S142: Input the deep feature image into the target recognition network to perform category prediction and parameter regression for each position in the deep feature image. That is, predict all target vehicle positions through the first convolutional layer and generate anchor boxes with a heading angle of 0 or a heading angle of 0.5π accordingly. Then, predict the offset of all anchor boxes through the second convolutional layer to generate all candidate boxes accordingly.

[0130] Step S143: Use the non-maximum suppression algorithm to extract candidate boxes with higher prediction scores from all candidate boxes. Each extracted candidate box reflects information about a target vehicle, specifically including the center position, size, and orientation of the target vehicle.

[0131] For the image visualization operation mentioned in step S140 above, please refer to... Figure 2 As shown, Figure (a) is a single-frame image data in which all target vehicles are marked with candidate boxes, and Figure (b) is a single-frame point cloud in which all target vehicles are marked with candidate boxes.

[0132] In this embodiment of the invention, by constructing a region of interest to selectively extract from the acquired raw point cloud and raw image data, the computation speed can be improved; by assigning associated image features to each point cloud data and further combining the attention fusion mechanism to perform feature enhancement operations on the point cloud data, it is helpful to solve the noise interference problem that may exist in the point cloud data; finally, by using a graph neural network to perform multi-scale feature extraction operations on the enhanced point cloud, the point cloud features can be fully explored, thereby improving the vehicle detection accuracy.

[0133] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the composition of a vehicle detection system based on graph neural networks and multi-sensor fusion, provided by an embodiment of the present invention. The system includes:

[0134] Data acquisition module 210 is used to acquire single-frame image data and single-frame point cloud data of the area of ​​interest in front of the vehicle;

[0135] The point cloud enhancement module 220 is used to acquire the global image features corresponding to the single frame image data, and then combine the global image features to perform feature enhancement operations on the single frame point cloud to obtain the first single frame point cloud.

[0136] The feature extraction module 230 is used to perform context feature extraction on the first single-frame point cloud through a graph neural network to obtain the deep feature image corresponding to the first single-frame point cloud.

[0137] The target recognition module 240 is used to identify and acquire all target vehicle information in the deep feature image, and then mark and visualize the single-frame image data and the single-frame point cloud based on all target vehicle information.

[0138] The content of the above method embodiments is applicable to this system embodiment. The functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are the same as those in the above method embodiments. Therefore, they will not be repeated here.

[0139] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the vehicle detection method based on graph neural networks and multi-sensor fusion described in the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium on which a device (e.g., a computer, mobile phone, etc.) stores or transmits information in a readable form, and can be a read-only memory, a disk, or an optical disk, etc.

[0140] also, Figure 4 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. The computer device includes components such as a processor 320, a memory 330, an input unit 340, and a display unit 350. Those skilled in the art will understand that... Figure 4 The illustrated device structure is not intended to limit all devices and may include more or fewer components than shown, or combine certain components. The memory 330 can be used to store the computer program 310 and various functional modules. The processor 320 runs the computer program 310 stored in the memory 330, thereby performing various functional applications and data processing of the device. The memory can be internal memory or external memory, or include both internal and external memory. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory 330 disclosed in the embodiments of this invention includes, but is not limited to, these types of memory. The memory 330 disclosed in the embodiments of this invention is only an example and not a limitation.

[0141] Input unit 340 is used to receive signal input and user-input keywords. Input unit 340 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 350 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 350 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 320 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, performing various functions and processing data by running or executing software programs and / or modules stored in memory 330, and calling data stored in memory.

[0142] As one embodiment, the computer device includes a processor 320, a memory 330, and a computer program 310, wherein the computer program 310 is stored in the memory 330 and configured to be executed by the processor 320, and the computer program 310 is configured to perform the vehicle detection method based on graph neural networks and multi-sensor fusion in the above embodiment.

[0143] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A vehicle detection method based on graph neural networks and multi-sensor fusion, characterized in that, The method includes: Acquire single-frame point cloud and single-frame image data that fall within the region of interest in front of the vehicle; Global image features are extracted from the single-frame image data, and then the global image features are used to enhance the features of the single-frame point cloud to obtain the first single-frame point cloud. The first single-frame point cloud is subjected to contextual feature extraction using a graph neural network to obtain the corresponding deep feature image; All target vehicle information is identified and obtained from the deep feature image, and then marked and visualized in the single frame point cloud and the single frame image data based on all target vehicle information; The step of using the global image features to perform feature enhancement on the single-frame point cloud to obtain the first single-frame point cloud includes: Transform each point cloud data in the single-frame point cloud into a pixel coordinate system; Based on the global image features, each point cloud data after conversion is assigned a corresponding image feature, and then the attention fusion mechanism is combined to enhance the features of each point cloud data after conversion, thereby obtaining the first single frame point cloud. The step of assigning corresponding image features to each point cloud data after conversion based on the global image features includes: For any transformed point cloud data, determine whether the global image features contain image features corresponding to the position of the transformed point cloud data; If so, the image features are assigned to the converted point cloud data; If not, select four converted point cloud data that are adjacent to the converted point cloud data from the converted single-frame point cloud, and obtain four image features corresponding to the positions of the four converted point cloud data from the global image features. Then, use the bilinear interpolation algorithm to perform interpolation on the four image features, and finally assign the image features obtained by the interpolation operation to the converted point cloud data. The step of using a graph neural network to extract contextual features from the first single-frame point cloud to obtain the corresponding deep feature image includes: The region of interest is divided into several voxels, and the single-frame point cloud is distributed within the several voxels; Using the single-frame point cloud and the first single-frame point cloud, calculate the features of several voxels corresponding to the several voxels, and then use the K nearest neighbor algorithm to construct several sub-graphs corresponding to the several voxels. The graph neural network is used to extract features from the sub-graphs, and then the feature extraction results are converted into two-dimensional feature images. The two-dimensional feature image is convolved to obtain a deep feature image.

2. The vehicle detection method based on graph neural networks and multi-sensor fusion according to claim 1, characterized in that, The acquisition of single-frame point cloud and single-frame image data falling within the region of interest in front of the vehicle includes: Acquire a single frame of raw point cloud data of the front of the vehicle collected by the LiDAR, and simultaneously acquire a single frame of raw image data of the front of the vehicle collected by the RGB camera. The origin of the lidar coordinate system coincides with the origin of the pixel coordinate system, and the region of interest is constructed based on the coincident origin. Extract all point cloud data falling within the region of interest from the single-frame raw point cloud to form a single-frame point cloud; Extract single-frame image data that falls within the region of interest from the original single-frame image data.

3. The vehicle detection method based on graph neural networks and multi-sensor fusion according to claim 1, characterized in that, The feature enhancement of each point cloud data after transformation by combining the attention fusion mechanism includes: For any transformed point cloud data, the first fully connected layer is used to extract features from the transformed point cloud data to obtain the first feature data; The second fully connected layer is used to extract image features corresponding to the transformed point cloud data to obtain second feature data. After adjusting the first feature data and the second feature data to have the same number of channels, the features are added together to obtain the third feature data; The third feature data is extracted using a third fully connected layer, and the feature extraction results are normalized using an activation function to obtain attention weights. The image features corresponding to the transformed point cloud data are multiplied by the attention weights, and then the multiplication result is concatenated with the transformed point cloud data to obtain the enhanced point cloud data.

4. The vehicle detection method based on graph neural networks and multi-sensor fusion according to claim 1, characterized in that, The step of calculating the features of several voxels corresponding to several voxels using the single-frame point cloud and the first single-frame point cloud includes: For any voxel, multiple point cloud data distributed within the voxel are obtained from the single-frame point cloud, and multiple enhanced point cloud data associated with the multiple point cloud data are obtained from the first single-frame point cloud. The multiple enhanced point cloud data are converted to the lidar coordinate system and then stitched together with the corresponding multiple point cloud data to obtain multiple stitched point cloud data. The multiple stitched point cloud data are averaged to obtain the voxel features corresponding to the voxels.

5. The vehicle detection method based on graph neural networks and multi-sensor fusion according to claim 1, characterized in that, The construction of several subgraphs corresponding to the several voxels using the K-nearest neighbor algorithm includes: For any voxel, based on the voxel features corresponding to the voxels, the K nearest neighbor algorithm is used to search for the first set of voxels that are adjacent to the voxel. Then, the voxel is connected to each voxel in the first set of voxels to obtain the first set of side lengths. The K-nearest neighbor algorithm is used to search for a second set of voxels that are adjacent to the voxels, excluding the first set of voxels. Then, the voxels are connected to each voxel in the second set of voxels to obtain the second set of side lengths. The K-nearest neighbor algorithm is used to search for a third set of voxels that is adjacent to the voxel, excluding the first set of voxels and the second set of voxels. Then, the voxel is connected to each voxel in the third set of voxels to obtain the third set of side lengths. Based on the first set of side lengths, the second set of side lengths, and the third set of side lengths, construct the subgraph corresponding to the voxel.

6. A vehicle detection system based on graph neural networks and multi-sensor fusion, characterized in that, The system includes: The data acquisition module is used to acquire single-frame point cloud and single-frame image data of the area of ​​interest in front of the vehicle; The point cloud enhancement module is used to extract global image features from the single-frame image data, and then use the global image features to enhance the single-frame point cloud to obtain the first single-frame point cloud. The feature extraction module is used to extract contextual features from the first single-frame point cloud using a graph neural network to obtain the corresponding deep feature image. The target recognition module is used to identify and acquire all target vehicle information in the deep feature image, and then mark and visualize the single-frame point cloud and the single-frame image data according to all target vehicle information. The step of using the global image features to perform feature enhancement on the single-frame point cloud to obtain the first single-frame point cloud includes: Transform each point cloud data in the single-frame point cloud into a pixel coordinate system; Based on the global image features, each point cloud data after conversion is assigned a corresponding image feature, and then the attention fusion mechanism is combined to enhance the features of each point cloud data after conversion, thereby obtaining the first single frame point cloud. The step of assigning corresponding image features to each point cloud data after conversion based on the global image features includes: For any transformed point cloud data, determine whether the global image features contain image features corresponding to the position of the transformed point cloud data; If so, the image features are assigned to the converted point cloud data; If not, select four converted point cloud data that are adjacent to the converted point cloud data from the converted single-frame point cloud, and obtain four image features corresponding to the positions of the four converted point cloud data from the global image features. Then, use the bilinear interpolation algorithm to perform interpolation on the four image features, and finally assign the image features obtained by the interpolation operation to the converted point cloud data. The step of using a graph neural network to extract contextual features from the first single-frame point cloud to obtain the corresponding deep feature image includes: The region of interest is divided into several voxels, and the single-frame point cloud is distributed within the several voxels; Using the single-frame point cloud and the first single-frame point cloud, calculate the features of several voxels corresponding to the several voxels, and then use the K nearest neighbor algorithm to construct several sub-graphs corresponding to the several voxels. The graph neural network is used to extract features from the sub-graphs, and then the feature extraction results are converted into two-dimensional feature images. The two-dimensional feature image is convolved to obtain a deep feature image.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the computer program to implement the vehicle detection method based on graph neural networks and multi-sensor fusion as described in any one of claims 1 to 5.

Images