Automobile automatic obstacle avoidance system based on real-time 3D target detection

Abstract

The invention relates to an automobile automatic obstacle avoidance system based on real-time 3D target detection, the system comprises a point cloud data collection module, an image data collection module, a LoGoNet network and an early warning and feedback module, and the LoGoNet network is used for realizing real-time target detection of an obstacle, detecting whether there is an obstacle, and judging the category of the obstacle. The system is a local-to-global multi-modal data fusion network, can carry out real-time 3D target detection automobile automatic obstacle avoidance, solves the inherent limitation of a single mode, carries out local-to-global multi-modal fusion by using the complementary characteristics of an image and a laser radar, provides a more complete three-dimensional environment perception capability through multi-source fusion, and improves the real-time 3D target detection efficiency. And the vehicle is helped to better perceive the surrounding environment.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and in particular to an automatic obstacle avoidance system for an automobile based on real-time 3D target detection. Background Art

[0002] The autonomous driving obstacle avoidance system is a system that can help autonomous vehicles perceive and avoid obstacles. It uses various sensors, algorithms and decision-making systems to monitor and analyze the environment around the vehicle, and automatically controls the vehicle's steering, acceleration and braking to avoid potential obstacles. The goal of the system is to ensure that the vehicle can safely avoid collisions with other vehicles, pedestrians, buildings, animals or other obstacles during driving. Among them, the machine vision-based autonomous driving obstacle avoidance system refers to a system that uses image recognition and analysis technology to perceive and avoid obstacles. It uses cameras or other visual sensors to collect real-time images, and performs image processing and analysis through computer vision algorithms to detect and identify obstacles around the vehicle.

[0003] In the actual use of existing autonomous driving obstacle avoidance systems, there are a large number of obstacles and traffic conditions in complex scenes. Traditional methods may not be able to effectively process and analyze this information, resulting in performance degradation. Secondly, traditional methods may have accuracy and reliability issues in environmental perception and obstacle detection. The limitations of a single sensor and sensor noise may lead to incomplete or inaccurate information, which affects the accuracy and safety of obstacle avoidance decisions and limits the system's adaptive capabilities and decision-making effectiveness.

[0004] With the continuous progress of deep learning and computer vision technology, 3D object detection has become a core component in autonomous driving technology. According to the type of input data used, 3D object detection technology can be mainly divided into three types: image-based methods, lidar-based methods, and multimodal data fusion methods. Although image data is rich in color and texture information, they usually do not contain enough depth information and are very sensitive to lighting conditions, which limits the performance of image-based detection methods. On the other hand, although lidar overcomes the lack of depth information in image data by providing accurate distance measurement and detailed geometric data. However, lidar data also has its limitations, including low resolution, sparse point cloud distribution, and lack of texture information, especially at long distances and in occluded areas, the information provided by lidar is very limited, which seriously affects the detection effect of distant objects. Therefore, the sparsity and disorder of lidar data cannot provide enough context to distinguish distant areas, resulting in suboptimal performance. Existing multimodal data fusion methods mainly perform global fusion, fusing image features and point cloud features in the entire scene, lacking fine-grained regional-level local information, and easily misjudging the target detection object, resulting in unsatisfactory fusion performance. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above technologies. The present invention provides an automatic obstacle avoidance system for automobiles based on real-time 3D target detection. The system is a local to global multimodal data fusion network that can perform automatic obstacle avoidance for automobiles with real-time 3D target detection, solving the inherent limitations of a single modality, and utilizing the complementary characteristics of images and laser radars to perform local to global multimodal fusion. Through multi-source fusion, a more complete three-dimensional environmental perception capability is provided to help vehicles better perceive the surrounding environment.

[0006] To achieve the above object, the technical solution of the present invention is: An automatic obstacle avoidance system for a vehicle based on real-time 3D target detection, the system comprising: Point cloud data collection module: including laser radar, used to collect three-dimensional point cloud data of the surrounding environment; Image data collection module: including a depth camera module for capturing two-dimensional images of the surrounding environment; LoGoNet network: used to realize real-time target detection of obstacles; connected with the point cloud data collection module and the image data collection module, it processes the collected point cloud data and image data in real time, detects whether there are obstacles, and determines the category of obstacles; Early warning and feedback module: obtain the detection result of the LoGoNet network, set the first distance threshold, the second distance threshold and the first height threshold, and the second height threshold. If there is an obstacle in the forward area and it is a dynamic target, when the distance of the obstacle detected in front of the vehicle's driving route is not greater than the first distance threshold, start to avoid the obstacle; if there is an obstacle in the forward area and it is a static target, when the height of the obstacle detected in front of the vehicle's driving route is greater than the first height threshold and the distance is not greater than the first distance threshold, re-plan the route to avoid the obstacle; if there is an obstacle in the forward area and it is a static target, when the height of the obstacle detected in front of the vehicle's driving route is between the second height threshold and the first height threshold and the distance is not greater than the second distance threshold, re-plan the route to avoid the obstacle; The first distance threshold is greater than the second distance threshold, and the first height threshold is greater than the second height threshold; The LoGoNet network includes a global fusion module GoF, a local fusion module LoF, and a feature dynamic aggregation module FDA. It takes the point cloud data P obtained by the laser radar and the image data I obtained by the depth camera module as input, including a point cloud branch and an image branch. For the point cloud branch, given the input point cloud, a 3D voxel-based skeleton network is used to generate voxel features. ; Then use the region proposal network RPN to generate the initial bounding box proposal , n is the number of bounding box proposals; For the image branch, the image data I is processed using the 2D pixel skeleton to obtain dense semantic image features. ; Voxel features and dense semantic image features Input into the global fusion module GoF, dense semantic image features and bounding box proposals Input to the local fusion module LoF; obtain the features of the local grid area of ​​interest through the local fusion module LoF processing and local mesh fusion features , obtain the global fusion feature through the global fusion module GoF ; Will , , The input is fed into the feature dynamic aggregation module FDA for cross-modal fusion, and the target detection result is output. The target detection result includes the category to which the obstacle target belongs and the position and size of the obstacle target in three-dimensional space; The processing process of the feature dynamic aggregation module FDA is: , , Aggregate according to the following formula to obtain the aggregated features : = + +

[0007] Will After being input into CBAM attention for processing, it is then combined with Perform residual connection, then input it into the residual connection block G-RCB composed of GLU and residual, and output the final classified image containing the bounding box.

[0008] Furthermore, the global fusion module GoF is used to obtain global features, and the process is as follows: the input voxel features and dense semantic image features , by The spatial positions of all points are averaged to obtain the averaged features of each voxel The point centroid : Next, for each point centroid Assign a voxel grid index and match the relevant voxel features through the hash table; use the camera projection matrix M' and the point centroid Calculate the reference point on the image plane :

[0009] Among them, M' is used to project points in three-dimensional space onto the two-dimensional image plane; is matrix multiplication; at the same time, through the voxel features The sampling offset is obtained by linear projection through the linear layer and attention weights ; Dense semantic image features Learn the offset , according to the formula Generate image features ; At the reference point Based on the reference point A set of surrounding image features According to formula (3), clustering is performed to generate clustered image features. , (3) in, and is the learnable weight, is the number of attention heads, K is the total number of sampling points; and They represent the sampling offset and attention weight of the kth sampling point in the mth attention head respectively, O is a trainable parameter; h is a trainable bias vector; Each voxel feature Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then with the averaged feature of each voxel Splicing to get fused voxel features ; Finally, Perform ROI pooling to generate global fusion features .

[0010] Furthermore, the local fusion module LoF includes a grid point dynamic fusion module GDF, which is used to dynamically fuse point cloud features and image features, and perform local multimodal feature fusion on image features; The process of the grid point dynamic fusion module GDF is: given each bounding box proposal , which is divided into a regular voxel grid , where j is the voxel grid index; take the center point As the corresponding voxel grid The grid points are used to propose bounding boxes using the position information encoder PIE Encode the location information of each bounding box proposal Create a mesh feature ; Perform PIE processing on all bounding box proposals to obtain local grid area of ​​interest features = , is the encoded local original point cloud position index; Features in the region of interest on the local grid Linear projection is performed on the sample to obtain the sampling offset and attention weights , dense semantic image features Learn the offset ,according to Generate image features , According to formula (3), clustering is performed to generate clustered image features. ; (3) in, and is the learnable weight, is the number of attention heads, K is the total number of sampling points; O is a trainable parameter; h is a trainable bias vector; The local grid region of interest features Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then combined with the local grid area of ​​interest features. Splicing to get dynamic fusion features of grid points ; Then Perform ROI pooling to obtain local grid fusion features , and obtain the output of the local fusion module.

[0011] Furthermore, the position information encoder PIE is implemented using a multi-layer perceptron MLP.

[0012] Furthermore, the CBAM attention includes a channel attention module and a spatial attention module. The input of the CBAM attention is used as Query, Key, and Value respectively. The three are arranged into a matrix and input into the channel attention module to emphasize the features in different channels. The formula of the channel attention module is expressed as:

[0013] in, It refers to the sigmoid function; , are the parameters obtained through MLP training, , , R is the hidden activation size, r is the reduction rate, C is the number of channels; b is the bias parameter; and They are two different spatial context descriptors generated by compressing features in the spatial dimension through average pooling and maximum pooling respectively; is the channel attention feature; The spatial attention module is used to emphasize the spatial region in the feature, and its formula is expressed as: =

[0014] in, Indicates filter 7 7 convolution operations, and They are two 2D maps generated by applying average pooling and maximum pooling in the channel dimension respectively; [;] represents the concatenation operation in the channel dimension; is the spatial attention feature; First, the channel attention feature Multiply it with the input of CBAM attention to get the refined feature F', and use F' as the input of the spatial attention module; then use the spatial attention feature Multiply it with F' to get the final adaptively refined features.

[0015] Furthermore, the structure of the G-RCB is expressed by the formula:

[0016] in, is the output of the linear gating unit GLU, is the input of the linear gating unit GLU, W is the trainable weight, is the overall output of G-RCB.

[0017] Furthermore, the three-dimensional point cloud data collected by the point cloud data collection module is enhanced as follows: (1.1) The point cloud data is collected in the road scene of daily life. The point cloud data in a collection scene is a point cloud set. All the collected point cloud sets constitute a point cloud data set. At the same time, at least 5 RGB images are randomly taken using the depth camera module in each point cloud scene. The RGB images taken in all point cloud scenes constitute an image data set. The point cloud set in a point cloud scene and the corresponding RGB image constitute a sample data, and all sample data constitute a data set. (1.2) Set the detection distance of the X and Y axes to [-75.2m, 75.2m], the detection distance of the Z axis to [-2m, 4m], and the voxel size to (0.1m, 0.1m, 0.15m); XYZ represents the spatial coordinate axis of the point cloud; (1.3) Normalize the collected point cloud data; (1.4) Then the point cloud data is Random rotation with the Z axis as the rotation axis is performed within the range to simulate the scene of the vehicle in different directions; (1.5) Then globally scale the point cloud data using a scaling factor in the range of [0.95, 1.05] to simulate the observation effects at different distances; (1.6) Finally, random noise is added to the point cloud data to simulate the sensor noise in the real world and obtain the enhanced point cloud dataset.

[0018] Furthermore, the training process of the LoGoNet network is: In the first stage, the point cloud branch is trained 20 times. When training the point cloud branch, the weight of the image branch is frozen, and the overall loss function L is composed of the RPN loss L RPN , confidence loss L conf and box regression loss L reg composition, L=L RPN +L conf +ɑL reg Among them, ɑ is a hyperparameter for balancing different losses; In the second stage, the entire LoGoNet network is trained for 60 epochs, and the batch size is set to 8 batches per GPU; In the third stage, the entire LoGoNet network is trained for 80 epochs, and the batch size is set to 2 batches per GPU; When the training gradient value of the training set is within the range of ±1e-5, the optimal training parameters are obtained. Then, the training is completed when the target detection accuracy is not less than 95% on the test set and the validation set.

[0019] Preferably, α takes a value of 0.5-1.

[0020] Furthermore, the depth camera module includes four cameras: front view, rear view, side view, and side rear view blind spot compensation. The front view is composed of three cameras: one 2-megapixel lens with a long-range field of view FOV of 30°, one 2-megapixel lens with a medium-range field of view FOV of 60°, and a 5-megapixel lens with a short-range field of view FOV of 130°; the side view is composed of two 2-megapixel FOV100° lenses, the side rear view blind spot compensation is composed of two 2-megapixel FOV100° lenses, and the rear view is composed of one 2-megapixel FOV100° lens.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses the LoGoNet network to perform lidar-camera fusion processing on point cloud data and image data at both global and local levels. Local fusion focuses on the detailed information around each box, while global fusion considers the context of the entire scene. This local-to-global fusion method can capture environmental information of different scales, and at the same time, combine local and global information to effectively improve target detection accuracy. The local-to-global fusion can adapt to different detection tasks and scenarios, especially in complex scenes, and can reduce false detections and missed detections.

[0022] The system of the present invention realizes a local-to-global fusion network, and uses feature dynamic aggregation (FDA) to provide finer-grained regional-level information for objects at different distances and retain the original position information at a finer granularity. The local fusion module LoF combined with the position information encoder (PIE) effectively fuses features by independently processing channel and spatial dimensions through CBAM attention.

[0023] The system of the present invention has stronger real-time performance and higher precision, and can realize automatic obstacle avoidance of automobiles with real-time 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a schematic diagram of the model structure of the LoGoNet network according to an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the specific structure of the global fusion module (GoF).

[0026] Figure 3 This is a schematic diagram of the specific structure of the local fusion module (LoF).

[0027] Figure 4 Schematic diagram of the specific structure of the characteristic dynamic aggregation module (FDA).

[0028] Figure 5 Schematic diagram of the specific structure of CBAM attention. DETAILED DESCRIPTION

[0029] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0030] Embodiment 1: The automatic obstacle avoidance system for automobiles based on real-time 3D target detection in this embodiment includes: Point cloud data collection module: It is composed of laser radar and installed on the vehicle to collect 3D point cloud data of the surrounding environment. The laser radar determines the distance and shape of the target obstacle by emitting laser pulses and measuring the time it takes for the laser pulses to be reflected back.

[0031] Image data collection module: includes a depth camera module, which consists of T cameras and is installed on the vehicle to capture two-dimensional images of the surrounding environment. The two-dimensional image data provides rich texture and color information, which helps to identify the appearance and details of objects.

[0032] LoGoNet network module: used to realize real-time target detection of obstacles; connected with the point cloud data collection module and the image data collection module, its main function is to process the collected point cloud data and image data in real time, detect whether there are obstacles, and determine the category of obstacles for obstacle avoidance; Early warning and feedback module: upload the results of obstacle detection. If there is an obstacle in the forward area, re-plan the route. If there is an obstacle in the forward area and it is a dynamic target, leave at least 20 meters to start obstacle avoidance; if it is identified as a static target, there is an obstacle less than 3 meters and higher than 0.3 meters 10 meters ahead of the car's driving route, re-plan the route. If there is an obstacle higher than 3 meters within 20 meters ahead, re-plan the route. At this time, the first distance threshold is 20m, the second distance threshold is 10m, the first height threshold is 3m, and the second height threshold is 0.3m.

[0033] The overall architecture of the LoGoNet network is as follows Figure 1 As shown, it includes the global fusion module (GoF), the local fusion module (LoF), and the feature dynamic aggregation module (FDA). The specific process is as follows: First, input the point cloud data from Lidar (laser radar) and image data from T cameras .in is the spatial coordinate of the i-th point, f i is an additional feature containing the intensity or extension of each point, N is the number of points in the point cloud, and are the height and width of the input image, is the image taken by the jth camera, and T is the number of cameras.

[0034] The point cloud data provides the location information of the object to be identified, and at the same time provides the structure of the surrounding environment, which is helpful for vehicle positioning and environmental modeling. At the same time, the input image data provides rich texture and color information, which helps to identify the appearance and details of the object.

[0035] Then, for the point cloud branch, given the input point cloud, a 3D voxel-based skeleton network is used to generate voxel features ,in is the number of channels for each voxel feature, (X,Y,Z) is the grid size for each voxel feature. Then, a region proposal network (RPN) is used to generate the initial bounding box proposal , n is the number of grid blocks, i.e., the number of bounding box proposals.

[0036] Next, for the image branch, the original multi-camera images are processed using a 2D pixel skeleton (such as VIT) to obtain dense semantic image features ,in is the number of channels of image features.

[0037] Next, the voxel features and dense semantic image features Input into the global fusion module GoF, dense semantic image features and bounding box proposals Input to the local fusion module LoF; obtain the features of the local grid area of ​​interest through the local fusion module LoF processing and local mesh fusion features , obtain the global fusion feature through the global fusion module GoF ; Will , , The input is entered into the feature dynamic aggregation module FDA for cross-modal fusion, and then the final target detection result is output; the target detection result configures a 3D bounding box for all detected obstacle targets in the recognition scene, and the bounding box defines the position and size of the obstacle target in the three-dimensional space. At the same time, each obstacle target will also be identified according to its category.

[0038] Voxel features and dense semantic image features Input into the global fusion module (GoF), first calculate the centroid of each voxel and map it to the image plane to get the reference point , and then adaptively fuse the point cloud features and image features through a set of attention heads and weights learned by the linear layer, and finally obtain the global fusion feature .

[0039] Bounding Box Proposals and image features Input into the local fusion module (LoF), Obtaining the features of the local grid region of interest through the position encoder PIE , and The grid point dynamic fusion module (GDF) dynamically fuses point cloud features and image features at the proposal level to provide more local and fine-grained geometric information and ultimately generate local fusion features. .

[0040] The specific structure of the global fusion module (GoF) is as follows: Figure 2 As shown in Figure 2, the main purpose is to obtain global fusion features, which is mainly composed of the centroid dynamic fusion module (CDF). The main process is as follows: Input voxel features and image features ,in is the voxel index, is the non-empty voxel feature vector, is the number of non-empty voxel feature vectors. The spatial positions of all points are averaged to obtain the averaged features of each voxel The point centroid :

[0041] in is the spatial coordinate, |P( )| is a voxel The points within.

[0042] Next, a voxel grid index is assigned to each calculated voxel feature point centroid and the related voxel features are matched through a hash table. Then, using the camera projection matrix M', each calculated voxel point centroid is used according to the following formula Calculate the reference point on the image plane :

[0043] Among them, M' is a 3*4 matrix used to project points in three-dimensional space onto the two-dimensional image plane; is matrix multiplication. At the same time, by The sampling offset is obtained by linear projection through the linear layer and attention weights .

[0044] Dense semantic image features Learn the offset according to Generate image features ; At the reference point Based on the reference point A set of surrounding image features According to formula (3), clustering is performed to generate clustered image features. , (3) in, and is the learnable weight, is the number of attention heads, K is the total number of sampling points; and denote the sampling offset and attention weight of the kth sampling point in the mth attention head, respectively, and O is a trainable parameter to ensure that the obtained and voxel features have similarity; h is a trainable bias vector, making More robust; Each voxel feature Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then with the averaged feature of each voxel Splicing to get fused voxel features , N is the total number of voxel features, is the number of voxel feature channels; Input into CBAM attention, and obtain voxel features that are easier to fuse after CBAM attention processing ; Finally, Perform ROI pooling to generate global fusion features , which is the global proposal feature, is used for subsequent proposal refinement. This fusion method fully considers the geometric shape information of point cloud features and can better adapt to large-scale point clouds.

[0045] The local fusion module (LoF) can provide more local and fine-grained geometric information in the multimodal fusion process. The specific structure is as follows: Figure 3As shown, the grid point dynamic fusion module (GDF) is used to dynamically fuse point cloud features and image features in the proposal layer. In addition to using the position information of the original point cloud in each voxel grid, the model can use these encoded local grid region of interest (ROI) features. Adaptively absorb relevant image features into local proposals and perform local multimodal feature fusion on image features.

[0046] The process of the grid point dynamic fusion module is: given each bounding box proposal , which is divided into a regular voxel grid , where j is the voxel grid index; take the center point As the corresponding voxel grid The grid points are used to encode the location information of the bounding box proposal using the position information encoder PIE, and for each bounding box proposal Create a mesh feature ; Each mesh feature The acquisition process is:

[0047] in is the center point of each grid With bounding box proposal The centroid The relative position of Each voxel grid The number of points in, τ is a constant offset; the position information encoder PIE is implemented using a multi-layer perceptron MLP; Perform PIE processing on all bounding box proposals to obtain local grid area of ​​interest features = , is the encoded local original point cloud position index, indicating the number of voxel grids; the feature of the region of interest in the local grid The sampling offset is obtained by linear projection through the linear layer and attention weights .

[0048] Dense semantic image features Learn the offset according to Generate image features (This part is not shown in the figure), According to formula (3), clustering is performed to generate clustered image features. ; (3) in, and is the learnable weight, is the number of attention heads, K is the total number of sampling points; O is a trainable parameter; h is a trainable bias vector; The local grid region of interest features Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then combined with the local grid area of ​​interest features. Splicing to get dynamic fusion features of grid points ; Then Perform ROI pooling to obtain local grid fusion features , which is the output of the local fusion module.

[0049] The local fusion of the present invention helps to solve the occlusion problem and can analyze the local features of the object in more detail, while the global fusion helps to understand the relationship between objects. Both modules introduce CBAM attention to obtain voxel features that are easier to fuse. , and then get Cascading with K and V is conducive to obtaining more effective local and global fusion features.

[0050] The specific structure of the feature dynamic aggregation module FDA is as follows Figure 4 As shown in Figure 2, the specific process is as follows: After passing through the LOF and GOF modules, three features are obtained, namely , , These features are generated independently, with poor information interactivity and aggregation. They are fused using the feature dynamic aggregation (FDA) module, which introduces CBAM attention to adaptively establish the relationship between different grid points. First, the three are spliced ​​to obtain the aggregated feature for: = + +

[0051] Then, after CBAM attention processing, Perform residual connection, then input it into the residual connection block G-RCB composed of GLU and residual, and output the final classified image containing the bounding box.

[0052] The structure of the G-RCB is as follows Figure 4 As shown in , G-RCB is a residual structure as a whole:

[0053] in, is the output of the linear gating unit GLU, is the input of the linear gating unit GLU, and W is a trainable weight.

[0054] The present invention adds a linear gating unit GLU, which is mainly composed of an input layer, an intermediate layer, and an output layer. The intermediate layer is composed of two convolutional layers and a sigmoid function. The following is the output formula of the intermediate layer:

[0055] in is the input, W, V are trainable parameter matrices, b, c are the corresponding bias items, Represented as a sigmoid function, The corresponding elements are multiplied. Finally, the output layer uses the softmax function, and h is input into the output layer to obtain .

[0056] The present invention uses a residual connection block (G-RCB), in which GLU can learn which information is important and which can be ignored, which helps the model to be more robust in the face of noise and outliers, while improving parameter efficiency while achieving similar or better performance, helping to reduce overfitting of the model and improving model performance.

[0057] The structure of CBAM attention is as follows Figure 5 As shown in the figure, the CBAM attention includes a channel attention module and a spatial attention module. The input of the CBAM attention is used as Query, Key, and Value respectively. The three are arranged into a matrix and input into the channel attention module to emphasize the features in different channels. The formula of the channel attention module is expressed as:

[0058] in, It refers to the sigmoid function; , are the parameters obtained through MLP training, , , R is the hidden activation size, r is the reduction rate, C is the number of channels; b is the bias parameter, The value of and Stay consistent; and They are two different spatial context descriptors generated by compressing features in the spatial dimension through average pooling and maximum pooling respectively; is the channel attention feature; The spatial attention module is used to emphasize the spatial region in the feature, and its formula is expressed as: =

[0059] in, Indicates filter 7 7 convolution operations, and They are two 2D maps generated by applying average pooling and maximum pooling in the channel dimension respectively; [;] represents the concatenation operation in the channel dimension; is the spatial attention feature; b is the parameter.

[0060] First, the channel attention feature Multiply it with the input of CBAM attention to get the refined feature F', and use F' as the input of the spatial attention module; then use the spatial attention feature Multiply it with F' to get the final adaptively refined features.

[0061] Embodiment 2: The automatic obstacle avoidance system for automobiles based on real-time 3D target detection in this embodiment includes the following steps: (1) Obtain a point cloud dataset and perform enhancement processing on the point cloud dataset.

[0062] (1.1) Use the point cloud data collection module to collect point cloud data. The collection scene is the road scene of daily life. The point cloud data in a collection scene is a point cloud set. All the collected point cloud sets constitute a point cloud data set. At the same time, use the depth camera module to randomly shoot at least 5 RGB images in each point cloud scene. The RGB images shot in all point cloud scenes constitute an image data set. The point cloud set in a point cloud scene and the corresponding RGB image constitute a sample data. All sample data constitute a data set. The data set is randomly divided into a training set, a validation set, and a test set. In this embodiment, the number of training set point cloud sets is 800, the number of validation set point cloud sets is 200, and the number of test set point cloud sets is 150. Each point cloud set has about 200 frames.

[0063] (1.2) Set the point cloud data, set the detection distance of the X and Y axes to [-75.2m, 75.2m], and the detection distance of the Z axis to [-2m, 4m]. Then set the voxel size to (0.1m, 0.1m, 0.15m), where XYZ represents the spatial coordinate axes of the point cloud.

[0064] (1.3) The collected point cloud data is center-normalized, that is, the mean is subtracted and divided by the maximum distance of the point from the origin.

[0065] (1.4) Then rotate the point cloud data. Since it is daily road data, only random rotation with the Z axis as the rotation axis is performed. ] range, so as to simulate the scenes of vehicles in different directions.

[0066] (1.5) The point cloud data is then scaled to simulate the observation effect at different distances, and a scaling factor in the range of [0.95, 1.05] is used for global scaling.

[0067] (1.6) Finally, random noise is added to the point cloud data to simulate the sensor noise in the real world and obtain the enhanced point cloud dataset.

[0068] (2) Obtain the image dataset corresponding to the point cloud dataset and perform color enhancement processing.

[0069] (2.1) Extract the RGB image corresponding to each point cloud set.

[0070] (2.2) Convert the image format to the standard format 1024*1024 to obtain a standard format image; (2.3) Use a Gaussian function to perform a low-pass filter on the acquired standard format image to obtain a filtered image; (2.4) Convert the acquired standard format image and the filtered image to the logarithmic domain and perform subtraction (i.e., subtract the low-frequency components in the image) to obtain a reflection image in the logarithmic domain; (2.5) Perform image summation on the pixel level on the logarithmic domain reflection image to obtain the result; (2.6) Color restoration: (2.6.1) At the channel level, for each channel, add the pixel values ​​of the channel in the original image to get the sum of the channel, calculate the maximum value of the sum of all channels, and use it as the denominator of the normalization factor. For the sum of each channel, divide it by the denominator of the normalization factor to get the normalization factor; (2.6.2) Normalize the weight matrix (the weight matrix is ​​set manually according to different applications. Here, a uniform weight matrix is ​​used, that is, the weight of each channel is equal), convert it to the logarithmic domain, then multiply it by the nonlinear factor of color restoration (here taken as 2.0), and then divide it by the normalization factor to obtain the image color gain result.

[0071] (2.6.3) Recombine (multiply) the result obtained in (2.5) according to the weight matrix and the result of the image color gain to obtain the image result after color restoration; (2.7) Image restoration: The color restored image result obtained in (2.6.3) is multiplied by the gain of the image pixel value change range, and then added to the offset of the image pixel value change range to obtain the final result. The gain of the image pixel value change range is a multiplication factor used to adjust the brightness and contrast of the image, and is applied to each pixel value of the color restored image result. The offset of the image pixel value change range is a constant value used to shift the image after the multiplication factor, and is added to each pixel value of the color restored image result. The final image restoration result is obtained by multiplying the color restored image result by the multiplication factor and adding the offset to the multiplication result.

[0072] (3) Build the LoGoNet network, train it, and then test and verify it. The trained LoGoNet network can detect obstacles and accurately determine their distance.

[0073] A three-stage training strategy is adopted: First, the point cloud branch is trained for 20 times in the first stage. Then, in the second stage, the entire LoGoNet is trained for 60 epochs with a batch size of 8 per GPU. In the third stage, the entire LoGoNet is trained for 80 epochs with a batch size of 2 per GPU.

[0074] When training the point cloud branch, the weights of the image branch are frozen, and the overall loss function L is composed of the RPN loss L RPN , confidence loss L conf and box regression loss L reg composition, L=L RPN +L conf +ɑL reg Among them, ɑ is a hyperparameter for balancing different losses, and is set to 0.5 in this embodiment.

[0075] When the training gradient value of the training set is within the range of ±1e-5, the optimal training parameters are obtained. Then, the test and verification are performed on the test set and the validation set. If the results of the two are the same and the target detection accuracy is not less than 95%, the training ends and a real-time end-to-end target detection model is obtained.

[0076] (4) The point cloud data to be identified and the corresponding image data are input into the trained real-time end-to-end object detection model to detect obstacles and avoid them.

[0077] (4.1) The car in this embodiment is equipped with a solid-state laser radar, which is located in the middle of the top of the car and has a detection range of 200 meters from the center of the car.

[0078] (4.2) The depth camera module of this embodiment is: the front view is composed of three cameras: 1 2 million pixel (2M) FOV (field of view) 30° lens for long-distance monitoring; 1 2 million pixel FOV60° lens for medium-distance monitoring; 1 5 million pixel (5M) FOV130° lens for short-distance monitoring; the side view is composed of 2 2 million pixel FOV100° lenses, the side rear view blind spot is also composed of 2 2 million pixel FOV100° lenses, and the rear view is composed of 1 2 million pixel FOV100° lens. The above combination of cameras can have 50-200m long-distance detection capability, 20-120m medium-distance detection capability, and 10-50m close-distance detection capability.

[0079] (4.3) If the target is detected as a dynamic obstacle (pedestrian, car, cyclist), a safe distance of 20 meters is reserved to start obstacle avoidance.

[0080] (4.4) If it is determined that there are no obstacles within 10 meters of the vehicle’s route, the vehicle continues to move forward.

[0081] (4.5) If an obstacle smaller than 3 meters and higher than 0.3 meters is identified within 10 meters ahead of the vehicle’s route, the route will be replanned. If an obstacle higher than 3 meters is identified within 20 meters ahead, the route will be replanned.

[0082] The system of this embodiment is applied to the collected data and compared with the current mainstream single-modal methods based on images and point clouds, as well as multi-modal methods that fuse images and point clouds. There are three categories in the data set: cars (Car), pedestrians (Pedestrian) and cyclists (Cyclist), and the difficulty levels are divided into simple, medium and difficult. The overall performance comparison results are shown in Table 1, where AP3D refers to the average precision of three-dimensional detection, and '- ' means that the results are not reported or the source code is not public. "L" and "I" represent point clouds and images, respectively, and the best results are marked in bold.

[0083]

[0084] Any matters not described in the present invention are applicable to the prior art.

Claims

1. An automatic obstacle avoidance system for automobiles based on real-time 3D target detection, characterized in that: The system comprises: Point cloud data collection module: including laser radar, used to collect three-dimensional point cloud data of the surrounding environment; Image data collection module: including a depth camera module for capturing two-dimensional images of the surrounding environment; LoGoNet network: used to realize real-time target detection of obstacles; connected with the point cloud data collection module and the image data collection module, it processes the collected point cloud data and image data in real time, detects whether there are obstacles, and determines the category of obstacles; Early warning and feedback module: obtain the detection result of the LoGoNet network, set the first distance threshold, the second distance threshold and the first height threshold, and the second height threshold. If there is an obstacle in the forward area and it is a dynamic target, when the distance of the obstacle detected in front of the vehicle's driving route is not greater than the first distance threshold, start to avoid the obstacle; if there is an obstacle in the forward area and it is a static target, when the height of the obstacle detected in front of the vehicle's driving route is greater than the first height threshold and the distance is not greater than the first distance threshold, re-plan the route to avoid the obstacle; if there is an obstacle in the forward area and it is a static target, when the height of the obstacle detected in front of the vehicle's driving route is between the second height threshold and the first height threshold and the distance is not greater than the second distance threshold, re-plan the route to avoid the obstacle; The first distance threshold is greater than the second distance threshold, and the first height threshold is greater than the second height threshold; The LoGoNet network includes a global fusion module GoF, a local fusion module LoF, and a feature dynamic aggregation module FDA. It takes the point cloud data P obtained by the laser radar and the image data I obtained by the depth camera module as input, including a point cloud branch and an image branch. For the point cloud branch, given the input point cloud, a 3D voxel-based skeleton network is used to generate voxel features. ; Then use the region proposal network RPN to generate the initial bounding box proposal , n is the number of bounding box proposals; For the image branch, the image data I is processed using the 2D pixel skeleton to obtain dense semantic image features. ; Voxel features and dense semantic image features Input into the global fusion module GoF, dense semantic image features and bounding box proposals Input to the local fusion module LoF; obtain the features of the local grid area of ​​interest through the local fusion module LoF processing and local mesh fusion features , obtain the global fusion feature through the global fusion module GoF ; Will , , The input is fed into the feature dynamic aggregation module FDA for cross-modal fusion, and the target detection result is output. The target detection result includes the category to which the obstacle target belongs and the position and size of the obstacle target in three-dimensional space; The processing process of the feature dynamic aggregation module FDA is: , , Aggregate according to the following formula to obtain the aggregated features : = + + ， Will After being input into CBAM attention for processing, it is then combined with Perform residual connection, then input it into the residual connection block G-RCB composed of GLU and residual, and output the final classified image containing the bounding box.

2. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 1, characterized in that: The global fusion module GoF is used to obtain global features. The process is as follows: input voxel features and dense semantic image features , by The spatial positions of all points are averaged to obtain the averaged features of each voxel The point centroid : Next, for each point centroid Assign a voxel grid index and match the associated voxel feature through the hash table; Using the camera projection matrix M' and the point centroid Calculate the reference point on the image plane : ， Among them, M' is used to project the points in the three-dimensional space onto the two-dimensional image plane; • is the matrix multiplication; and at the same time, The sampling offset is obtained by linear projection through the linear layer and attention weights ; Dense semantic image features Learn the offset , according to the formula Generate image features ; At the reference point Based on the reference point A set of surrounding image features According to formula (3), clustering is performed to generate clustered image features. , （3） in, and is the learnable weight, M is the number of attention heads, and K is the total number of sampling points; and They represent the sampling offset and attention weight of the kth sampling point in the mth attention head, respectively. O is a trainable parameter; h is a trainable bias vector; Each voxel feature Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then with the averaged feature of each voxel Splicing to get fused voxel features ; Finally, Perform ROI pooling to generate global fusion features .

3. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 1, characterized in that: The local fusion module LoF includes a grid point dynamic fusion module GDF, which is used to dynamically fuse point cloud features and image features and perform local multimodal feature fusion on image features; The process of the grid point dynamic fusion module GDF is: given each bounding box proposal , which is divided into a regular voxel grid , where j is the voxel grid index; take the center point As the corresponding voxel grid The grid points are used to propose bounding boxes using the position information encoder PIE Encode the location information of each bounding box proposal Create a mesh feature ; Perform PIE processing on all bounding box proposals to obtain local grid area of ​​interest features = , is the encoded local original point cloud position index; Features in the region of interest on the local grid Linear projection is performed on the sample offset and attention weights , dense semantic image features Learn the offset ,according to Generate image features , According to formula (3), clustering is performed to generate clustered image features. ; （3） in, and is the learnable weight, M is the number of attention heads, K is the total number of sampling points; O is the trainable parameter; h is the trainable bias vector; The local grid region of interest features Represented as a query , the image features will be aggregated Represented as a key K and a value V, where After the input is processed by CBAM attention, it is cascaded with K and V, and then combined with the local grid area of ​​interest features. Splicing to get dynamic fusion features of grid points ; Then Perform ROI pooling to obtain local grid fusion features , and obtain the output of the local fusion module.

4. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 3 is characterized in that: The position information encoder PIE is implemented using a multi-layer perceptron MLP.

5. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to any one of claims 1 to 4, characterized in that: The CBAM attention includes a channel attention module and a spatial attention module. The input of the CBAM attention is used as Query, Key, and Value respectively. The three are arranged into a matrix and input into the channel attention module to emphasize the features in different channels. The formula of the channel attention module is expressed as: ， in, It refers to the sigmoid function; , are the parameters obtained through MLP training, , , R is the hidden activation size, r is the reduction rate, C is the number of channels; b is the bias parameter; and They are two different spatial context descriptors generated by compressing features in the spatial dimension through average pooling and maximum pooling respectively; is the channel attention feature; The spatial attention module is used to emphasize the spatial region in the feature, and its formula is expressed as: = ， in, Indicates filter 7 7 convolution operations, and They are two 2D maps generated by applying average pooling and maximum pooling in the channel dimension respectively; [;] represents the concatenation operation in the channel dimension; is the spatial attention feature; First, the channel attention feature Multiply it with the input of CBAM attention to get the refined feature F', and use F' as the input of the spatial attention module; then use the spatial attention feature Multiply it with F' to get the final adaptively refined features.

6. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 1, characterized in that: The structure of the G-RCB is expressed by the formula: ， in, is the output of the linear gating unit GLU, is the input of the linear gating unit GLU, W is the trainable weight, is the overall output of G-RCB.

7. The vehicle automatic obstacle avoidance system based on real-time 3D target detection according to claim 1, characterized in that: The three-dimensional point cloud data collected by the point cloud data collection module undergoes the following enhancement processing: (1.1) The point cloud data is collected in the road scene of daily life. The point cloud data in a collection scene is a point cloud set. All the collected point cloud sets constitute a point cloud data set. At the same time, at least 5 RGB images are randomly taken using the depth camera module in each point cloud scene. The RGB images taken in all point cloud scenes constitute an image data set. The point cloud set in a point cloud scene and the corresponding RGB image constitute a sample data, and all sample data constitute a data set. (1.2) Set the detection distance of the X and Y axes to [-75.2m, 75.2m], the detection distance of the Z axis to [-2m, 4m], and the voxel size to (0.1m, 0.1m, 0.15m); XYZ represents the spatial coordinate axis of the point cloud; (1.3) Normalize the collected point cloud data; (1.4) Then the point cloud data is Random rotation with the Z axis as the rotation axis is performed within the range to simulate the scene of the vehicle in different directions; (1.5) Then globally scale the point cloud data using a scaling factor in the range of [0.95, 1.05] to simulate the observation effects at different distances; (1.6) Finally, random noise is added to the point cloud data to simulate the sensor noise in the real world and obtain the enhanced point cloud dataset.

8. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 1, characterized in that: The training process of the LoGoNet network is: In the first stage, the point cloud branch is trained 20 times. When training the point cloud branch, the weight of the image branch is frozen, and the overall loss function L is composed of the RPN loss L RPN , confidence loss L conf and box regression loss L reg composition, L=L RPN +L conf +ɑL reg Among them, ɑ is a hyperparameter for balancing different losses; In the second stage, the entire LoGoNet network is trained for 60 epochs, and the batch size is set to 8 batches per GPU; In the third stage, the entire LoGoNet network is trained for 80 epochs, and the batch size is set to 2 batches per GPU; When the training gradient value of the training set is within the range of ±1e-5, the optimal training parameters are obtained. Then, the training is completed when the target detection accuracy is not less than 95% on the test set and the validation set.

9. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 8, characterized in that: The value of ɑ is 0.5-1.

10. The automatic obstacle avoidance system for automobiles based on real-time 3D target detection according to claim 1, characterized in that: The depth camera module includes four cameras: front view, rear view, side view, and side rear view blind spot compensation. The front view is composed of three cameras: one 2-megapixel lens with a long-range field of view FOV of 30°, one 2-megapixel lens with a medium-range field of view FOV of 60°, and a 5-megapixel lens with a short-range field of view FOV of 130°; the side view is composed of two 2-megapixel FOV100° lenses, the side rear view blind spot compensation is composed of two 2-megapixel FOV100° lenses, and the rear view is composed of one 2-megapixel FOV100° lens.

Images