A target perception method and system based on multi-sensor image result fusion

By combining the improved YOLOv5s and Complex-YOLOv4 network models with the CBAM attention mechanism, the problem of perception accuracy and efficiency of multi-sensor fusion strategies in adverse weather conditions was solved, achieving high-precision, real-time target detection.

CN117557911BActive Publication Date: 2024-11-22HARBIN INST OF TECH AT WEIHAI
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202311739155.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-15
Publication Date
2024-11-22
Estimated Expiration
2043-12-15

AI Technical Summary

Technical Problem

Existing multi-sensor fusion strategies suffer from limited perception accuracy and low operating efficiency in adverse weather conditions, making it difficult to accurately and in real-time detect target objects.

Method used

An improved YOLOv5s and Complex-YOLOv4 network model is used to extract features from camera images and LiDAR 3D point cloud data. The CBAM attention mechanism module is combined to allocate attention weights in space and channels. The detection target is optimized by feature fusion and non-maximum suppression method, so as to achieve the fusion of 2D and 3D detection results.

Benefits of technology

It improves the accuracy and robustness of target detection, enabling high-precision, real-time target detection in harsh weather and low-visibility light conditions, and adapting to multi-target detection in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117557911B_ABST
    Figure CN117557911B_ABST
Patent Text Reader

Abstract

The present invention provides a target perception method and system based on multi-sensor image result fusion, which relates to the fields of electronic information engineering and image processing technology, and is to solve the problems of limited perception accuracy and low operating efficiency of the prior art in severe weather environments. The method of the present invention extracts features from camera image data through an improved YOLOv5s network model; the model adopts deep separable convolution, and when the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer, and the plug-and-play CBAM attention mechanism module is used to distribute the weights of attention in space and channels during feature extraction; at the same time, through the improved Complex-YOLOv4 network model, feature extraction is performed on the three-dimensional point cloud data of the laser radar, the model adopts deep separable convolution, and the plug-and-play CBAM attention mechanism module is used to amplify the sensitivity in space and channels; the recognition results of the two are fused to obtain the final detected target information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of electronic information engineering and image processing technology, and in particular to a target perception method and system based on multi-sensor image result fusion. Background Art

[0002] With the rapid development of computer vision, countries around the world are accelerating the innovation and practical application of robots. For trackless intelligent inspection robots in park scenarios, how to quickly perceive the surrounding environment information has become the cornerstone for the robot to make safe and accurate routes.

[0003] At present, most sensors used for environmental perception are cameras that can provide rich image texture and color information. However, as a passive sensor, cameras are greatly restricted by light. Compared with cameras, lidar is an active ranging sensor that provides rich point cloud information and is less affected by the environment. In road environments, especially in closed and semi-closed parks, environmental perception tasks are somewhat complex. Not only are there many objects involved in traffic, but the weather environment is complex and changeable, and the lighting conditions cannot be maintained constantly. The fusion of lidar and cameras can make up for the shortcomings of a single camera perception strategy, but the existing multi-sensor fusion strategy still has limited perception accuracy and low operating efficiency in severe weather environments, making it difficult to accurately and real-time detect target objects. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] The existing multi-sensor fusion strategies have limited perception accuracy and low operating efficiency in severe weather environments, making it difficult to detect target objects accurately and in real time.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention provides a target perception method based on multi-sensor image result fusion, comprising the following steps:

[0008] S1, collect environmental data through camera and laser radar to obtain camera image data and laser radar three-dimensional point cloud data;

[0009] S2, extracting features from camera image data through the improved YOLOv5s network model to obtain camera image target recognition results;

[0010] The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction.

[0011] S3, through the improved Complex-YOLOv4 network model, the feature extraction of the laser radar 3D point cloud data is carried out to obtain the recognition result of the target object;

[0012] The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel.

[0013] S4: Fuse the camera image detection result with the three-dimensional space detection result to obtain the final detected target information.

[0014] Furthermore, the improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer;

[0015] The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module adopts a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module.

[0016] Furthermore, the improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer;

[0017] The Backbone layer processes the input image and uses the plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel to obtain the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the feature map received from the Backbone layer and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer.

[0018] Furthermore, the CBAM attention mechanism module includes a spatial attention mechanism module (SpatialAttention Module) and a channel attention mechanism module (Channel Attention Module). In the channel attention mechanism module, the feature map is first subjected to global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through a fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, that is:

[0019]

[0020] In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c Represents channel attention features.

[0021] Furthermore, the spatial attention mechanism module firstly performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function, and finally multiplies this weight with the feature map to complete the spatial attention weighting, that is:

[0022]

[0023] In the formula, f 7×7 represents a filter of size 7×7, M s Represents spatial attention features.

[0024] Furthermore, S3 specifically includes the following steps:

[0025] Step S3-1: Mapping the collected 3D point cloud data to a 2D image plane according to the color value corresponding to each point, so as to realize the 2D conversion of the point cloud data, that is, converting the 3D point cloud data into an RGB-Map;

[0026] Step S3-2: Use the Mosaic method to augment the RGB-Map data;

[0027] Step S3-3: inputting the augmented data and the original data into the improved Complex-YOLOv4 network model, performing feature extraction on the three-dimensional point cloud data of the laser radar, and obtaining a two-dimensional bounding box of the detection target;

[0028] Step S3-4: Using the non-maximum suppression (NMS) method to remove redundant detection boxes of the target object, only retaining the detection results with the highest confidence, and using the extended boundary generation network (E-RPN) to generate a 3D candidate box for the retained bounding box with the highest confidence;

[0029] Step S3-5: Perform target detection on the reconverted three-dimensional candidate box, and obtain the recognition result of the target object by performing target classification and bounding box regression in the three-dimensional candidate box.

[0030] A target perception system based on multi-sensor image result fusion, comprising:

[0031] The data acquisition module is used to collect environmental data through cameras and laser radars to obtain camera image data and laser radar three-dimensional point cloud data;

[0032] The camera image target recognition module is used to extract features from camera image data through an improved YOLOv5s network model to obtain camera image target recognition results;

[0033] The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction.

[0034] The LiDAR 3D point cloud target recognition module is used to extract features from the LiDAR 3D point cloud data through the improved Complex-YOLOv4 network model to obtain the recognition result of the target object;

[0035] The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel.

[0036] The detection module is used to fuse the camera image detection results with the three-dimensional space detection results to obtain the final detection target information.

[0037] Furthermore, in the camera image target recognition module, the improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer;

[0038] The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module adopts a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module.

[0039] Furthermore, in the laser radar three-dimensional point cloud target recognition module, the improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer;

[0040] The Backbone layer processes the input image and uses the plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel to obtain the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the feature map received from the Backbone layer and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer.

[0041] Furthermore, the CBAM attention mechanism module includes a spatial attention mechanism module (SpatialAttention Module) and a channel attention mechanism module (Channel Attention Module). In the channel attention mechanism module, the feature map is first subjected to global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through a fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, that is:

[0042]

[0043] In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c Represents channel attention features;

[0044] The spatial attention mechanism module first performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function. Finally, this weight is multiplied by the feature map to complete the spatial attention weighting, that is:

[0045]

[0046] In the formula, f 7×7 represents a filter of size 7×7, M s Represents spatial attention features.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention discloses a target perception method and system based on multi-sensor image result fusion. By combining two-dimensional and three-dimensional detectors and designing the architecture of two-dimensional and three-dimensional detection models, the utilization rate of feature information in the image is increased, the image processing branch is improved, and the detection accuracy is increased. In the improved YOLOv5s model for two-dimensional detection, the traditional convolution layer is replaced by a depth-separable convolution, and the number of parameters of the model is reduced to reduce the reasoning time; secondly, the features of feature maps of multiple scales are fused using a multi-layer feature fusion strategy, and when output to the Head part, the features are downsampled and multi-dimensionally spliced ​​with the features extracted from the Neck part structure; finally, the CBAM attention mechanism module is used to perform weight distribution of attention in space and channels during feature extraction to increase the detection accuracy. In the improved Complex-YOLOv4 model for three-dimensional detection, NMS and E-RPN are also used to further optimize the detection target, improve the accurate selection of three-dimensional candidate boxes, thereby achieving high accuracy and robustness of target detection, and improving the detection accuracy and operation efficiency of the model for multi-target detection in complex scenes.

[0049] The present invention has high detection accuracy for targets of interest on the road in bad weather and low visible light conditions at night, and has good real-time performance, thereby realizing target detection on complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a target perception method based on multi-sensor image result fusion in an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the improved YOLOv5s network model structure in an embodiment of the present invention;

[0052] Figure 3 It is a structural diagram of an improved Complex-YOLOv4 network model in an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of the CSP module structure in an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the structure of the SPPF module in an embodiment of the present invention;

[0055] Figure 6 Schematic diagram of the ResX module structure in an embodiment of the present invention;

[0056] Figure 7 Schematic diagram of the structure of the CBAM attention mechanism module in an embodiment of the present invention;

[0057] Figure 8 Schematic diagram of a channel attention mechanism module in an embodiment of the present invention;

[0058] Fig. 9 Schematic diagram of a spatial attention mechanism module in an embodiment of the present invention;

[0059] Fig.10 It is a curve diagram of the average loss change of YOLOv5s in an embodiment of the present invention;

[0060] Fig.11 It is a YOLOv5s verification accuracy graph in an embodiment of the present invention;

[0061] Fig.12 It is a YOLOv5s verification recall rate graph in an embodiment of the present invention;

[0062] Fig.13 It is a curve diagram of the average loss change of Complex-YOLOv4 in an embodiment of the present invention;

[0063] Fig.14 It is a Complex-YOLOv4 verification accuracy graph in an embodiment of the present invention;

[0064] Fig.15 It is a verification recall rate graph of Complex-YOLOv4 in an embodiment of the present invention;

[0065] Fig.16 is a schematic diagram of a three-dimensional detector network structure in an embodiment of the present invention;

[0066] Fig.17 A schematic diagram of target detection steps for image result fusion in an embodiment of the present invention;

[0067] Fig.18 is the detection result in low-visibility haze weather in an embodiment of the present invention;

[0068] Fig.19 This is the detection result in a low-visibility night scene in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In the description of the present invention, it should be noted that the terms "first", "second", and "third" mentioned in the embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0071] Specific implementation plan 1: Figure 1 As shown, the present invention provides a target perception method based on multi-sensor image result fusion, comprising the following steps:

[0072] S1, collect environmental data through camera and laser radar to obtain camera image data and laser radar three-dimensional point cloud data;

[0073] S2, extracting features from camera image data through the improved YOLOv5s network model to obtain camera image target recognition results;

[0074] The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction.

[0075] S3, through the improved Complex-YOLOv4 network model, the feature extraction of the laser radar 3D point cloud data is carried out to obtain the recognition result of the target object;

[0076] The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel.

[0077] S4: Fuse the camera image detection result with the three-dimensional space detection result to obtain the final detected target information.

[0078] Specific implementation plan 2: Figure 2 As shown, the improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer;

[0079] The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module adopts a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module.

[0080] The Backbone layer includes the focus module Focus, the first cross-stage binary module CBS in the Backbone layer, the first cross-stage partial module CSP1_1 containing a residual component in the Backbone layer, and the second cross-stage

[0081] The binary module CBS, the cross-stage partial module CSP1_2, the Backbone layer contains two residual components

[0082] The third cross-stage binary module CBS, the cross-stage partial module CSP1_3 with three residual components in the Backbone layer, the fourth cross-stage binary module CBS in the Backbone layer, the second cross-stage partial module CSP1_1 with one residual component in the Backbone layer, and the fast spatial pyramid pooling module SPPF.

[0083] The Neck layer includes a first cross-stage binary module CBS in the Neck layer, a first upsampling module Upsample in the Neck layer, a first concatenation module Concat in the Neck layer, a first cross-stage partial module CSP2_1 containing a residual component in the Neck layer, a second cross-stage binary module CBS in the Neck layer, a second upsampling module Upsample in the Neck layer, a second concatenation module Concat in the Neck layer, a second cross-stage partial module CSP2_1 containing a residual component in the Neck layer, which are arranged in sequence.

[0084] The third cross-stage binary module CBS in the Neck layer, the third concatenation module Concat in the Neck layer, the third cross-stage partial module CSP2_1 containing a residual component in the Neck layer, the fourth cross-stage binary module CBS in the Neck layer,

[0085] The fourth concatenation module Concat and the Neck layer contain a fifth cross-stage partial module CSP2_1 with a residual component.

[0086] A feature fusion module, including a first downsampling module Downsample of the feature fusion module, a first attention mechanism module CBAM of the feature fusion module, a second downsampling module Downsample of the feature fusion module, a second attention mechanism module CBAM of the feature fusion module, and a third attention mechanism module CBAM of the feature fusion module;

[0087] The Head layer includes a first detection module Detect in the Head layer, a second detection module Detect in the Head layer, and a third detection module Detect in the Head layer.

[0088] The CSP and CBS modules in the Backbone layer and Neck layer do not use the SPP module, but a SPPF module. Figure 4 As shown in the figure, the ResX module is mainly used in CSP to extract feature information on the feature map. The CSP module extracts features through two branches. Branch one extracts features through the CBS module and the ResX module, and branch two extracts features through the CBS module. The features of the two branches are then concat-joined and output through a CBS module. Figure 5 As shown in the figure, for ResX in the Backbone layer, Shortcut is set to True, which effectively enhances the depth of the feature extraction part, and Shortcut in the Neck part is set to False. The CBS module mainly includes convolutional layers, batch normalization layers, and SiLU activation functions. Figure 6 As shown in the figure, the SPPF module extracts feature information of different scales through a CBS module and three consecutive maximum pooling layers, and uses the Concat layer to concatenate the features of different scales of the four features, integrating the feature information between different levels, and finally outputs the features through a CBS module.

[0089] The rest of this embodiment is the same as the first specific embodiment.

[0090] Specific implementation plan three: Figure 3 As shown, the improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer;

[0091] The Backbone layer processes the input image and uses the plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel to obtain the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the feature map received from the Backbone layer and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer.

[0092] The Backbone layer includes CBM (consisting of Conv+BN+Mish activation function, Conv is the convolution layer, BN is the batch normalization layer, and Mish is the activation function in the convolutional neural network), one layer of residual module ResBody1 in the Backbone layer, two layers of residual module ResBody2 in the Backbone layer, the first eight layers of residual module ResBody8 in the Backbone layer, the first attention mechanism module CBAM in the Backbone layer, the second eight layers of residual module ResBody8 in the Backbone layer, the second attention mechanism module CBAM in the Backbone layer, four layers of residual module ResBody4 in the Backbone layer, and the third attention mechanism module CBAM in the Backbone layer. Among them, each layer of ResBody reduces the image pixels by half, and its main function is to extract the feature information of the image data.

[0093] The Neck layer includes the first three-layer convolution module CBL*3 in the Neck layer, the spatial pyramid pooling module SPP, the second three-layer convolution module CBL*3 in the Neck layer, the first splicing module Concat in the Neck layer, the first five-layer convolution module CBL*5 in the Neck layer, the first single-layer convolution module CBL in the Neck layer, the first upsampling module Upsample in the Neck layer, the first downsampling module Downsample in the Neck layer, the second single-layer convolution module CBL in the Neck layer, the second splicing module Concat in the Neck layer, the second five-layer convolution module CBL*5 in the Neck layer, the third splicing module Concat in the Neck layer, the third five-layer convolution module CBL*5 in the Neck layer, the third single-layer convolution module CBL in the Neck layer, the second upsampling module Upsample in the Neck layer, the second downsampling module Downsample in the Neck layer, the fourth single-layer convolution module CBL in the Neck layer, the fourth splicing module Concat in the Neck layer, and the fourth five-layer convolution module CBL*5 in the Neck layer.

[0094] The Head layer includes a first target detection head YOLO Head1 in the Head layer, a second target detection head YOLO Head2 in the Head layer, and a third target detection head YOLO Head3 in the Head layer, which are arranged sequentially.

[0095] The first eight-layer residual module ResBody8 in the Backbone layer transmits the feature map processed by the two-layer residual module ResBody2 in the Backbone layer to the second eight-layer residual module ResBody8 in the Backbone layer and the first attention mechanism module CBAM in the Backbone layer;

[0096] The second eight-layer residual module ResBody8 in the Backbone layer transmits the feature map processed by the first eight-layer residual module ResBody8 in the Backbone layer to the four-layer residual module ResBody4 in the Backbone layer and the second attention mechanism module CBAM in the Backbone layer;

[0097] The second eight-layer residual module ResBody8 in the Backbone layer transmits the feature map processed by the first eight-layer residual module ResBody8 in the Backbone layer to the four-layer residual module ResBody4 in the Backbone layer and the second attention mechanism module CBAM in the Backbone layer;

[0098] The first attention mechanism module CBAM in the Backbone layer transmits the feature map processed by the first eight-layer residual module ResBody8 in the Backbone layer to the fourth single-layer convolution module CBL in the Neck layer;

[0099] The second attention mechanism module CBAM in the Backbone layer transmits the feature map processed by the second eight-layer residual module ResBody8 in the Backbone layer to the second single-layer convolution module CBL in the Neck layer;

[0100] The third attention mechanism module CBAM in the Backbone layer transmits the feature map processed by the fourth residual module ResBody4 in the Backbone layer to the first three-layer convolution module CBL*3 in the Neck layer;

[0101] The second three-layer convolution module CBL*3 in the Neck layer transmits the feature map processed by the spatial pyramid pooling module SPP in the Neck layer to the first concatenation module Concat in the Neck layer and the first three-layer convolution module CBL in the Neck layer respectively;

[0102] The first concatenation module Concat in the Neck layer transmits the feature map processed by the second three-layer convolution module CBL*3 in the Neck layer and the first downsampling module Downsample in the Neck layer to the first five-layer convolution module CBL*5 in the Neck layer, and then transmits it to the first target detection head YOLO Head1 in the Head layer;

[0103] The second concatenation module Concat in the Neck layer transmits the feature map processed by the second convolution module CBL in the Neck layer and the first upsampling module Upsample in the Neck layer to the second five-layer convolution module CBL*5 in the Neck layer;

[0104] The second five-layer convolution module CBL*5 in the Neck layer transmits the feature map concatenated by the second concatenation module Concat in the Neck layer to the third concatenation module Concat in the Neck layer and the third but single convolution module CBL in the Neck layer respectively;

[0105] The third concatenation module Concat in the Neck layer transmits the feature map processed by the second five-layer convolution module CBL*5 in the Neck layer and the second downsampling module Downsample in the Neck layer to the third five-layer convolution module CBL*5 in the Neck layer, and then transmits the feature map to the first downsampling module Downsample in the Neck layer and the second target detection head YOLO Head2 in the Head layer respectively;

[0106] The fourth concatenation module Concat in the Neck layer concatenates the feature maps processed by the second up-sampling module Upsample in the Neck layer and the fourth single-layer convolution module CBL in the Neck layer, and transmits them to the fourth five-layer convolution module CBL*5 in the Neck layer, and then transmits the feature maps to the second down-sampling module Downsample in the Neck layer, the second down-sampling module Downsample in the Neck layer, and the third target detection head YOLO Head3 in the Head layer. The rest of this implementation is the same as the specific implementation plan one.

[0107] Specific implementation plan 4: Figure 7 As shown, the CBAM attention mechanism module includes a spatial attention mechanism module (Spatial Attention Module) and a channel attention mechanism module (Channel Attention Module), such as Figure 8 As shown in the figure, in the channel attention mechanism module, the feature map first undergoes global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through the fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, that is:

[0108]

[0109] In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c The other aspects of this embodiment are the same as those of the first embodiment.

[0110] The channel attention mechanism module assigns different weights to different channel features, assigning larger weights to important channel features and smaller weights to unimportant channel features, so as to achieve weighted weighting of different channel features, which means that the channel dimension remains unchanged and the spatial dimension is compressed. This module focuses on the meaningful information in the input image, emphasizing important features while suppressing unimportant features.

[0111] Specific implementation plan five: Fig. 9 As shown in the figure, the spatial attention mechanism module firstly performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function. Finally, this weight is multiplied by the feature map to complete the spatial attention weighting, that is:

[0112]

[0113] In the formula, f 7×7 represents a filter of size 7×7, M s Indicates the spatial attention feature. The rest of this implementation is the same as the specific implementation scheme four.

[0114] The spatial attention mechanism module pays more attention to the spatial feature information. The spatial attention mechanism module indicates that the spatial dimension remains unchanged and the channel dimension is compressed. The module focuses on the location information of the target.

[0115] By jointly using channel and spatial attention modules, the CBAM attention mechanism fully extracts important features in space and channels, effectively suppresses the loss of two important features, better captures global context and local detail information, and improves the model's representation learning ability.

[0116] Specific implementation plan six: Fig.16 As shown, S3 specifically includes the following steps:

[0117] Step S3-1: Mapping the collected 3D point cloud data to a 2D image plane according to the color value corresponding to each point, so as to realize the 2D conversion of the point cloud data, that is, converting the 3D point cloud data into an RGB-Map;

[0118] Step S3-2: Use the Mosaic method to augment the RGB-Map data;

[0119] Step S3-3: inputting the augmented data and the original data into the improved Complex-YOLOv4 network model, performing feature extraction on the three-dimensional point cloud data of the laser radar, and obtaining a two-dimensional bounding box of the detection target;

[0120] Step S3-4: Using the non-maximum suppression (NMS) method to remove redundant detection boxes of the target object, only retaining the detection results with the highest confidence, and using the extended boundary generation network (E-RPN) to generate a 3D candidate box for the retained bounding box with the highest confidence;

[0121] Step S3-5: Target detection is performed on the reconverted three-dimensional candidate frame, and the target object recognition result is obtained by performing target classification and bounding box regression in the three-dimensional candidate frame. The rest of this embodiment is the same as the first embodiment.

[0122] Specific implementation scheme seven: A target perception system based on multi-sensor image result fusion, characterized by comprising:

[0123] The data acquisition module is used to collect environmental data through cameras and laser radars to obtain camera image data and laser radar three-dimensional point cloud data;

[0124] The camera image target recognition module is used to extract features from camera image data through an improved YOLOv5s network model to obtain camera image target recognition results;

[0125] The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction.

[0126] The LiDAR 3D point cloud target recognition module is used to extract features from the LiDAR 3D point cloud data through the improved Complex-YOLOv4 network model to obtain the recognition result of the target object;

[0127] The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel.

[0128] The detection module is used to fuse the camera image detection results with the three-dimensional space detection results to obtain the final detection target information.

[0129] Specific implementation scheme eight: In the camera image target recognition module, the improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer;

[0130] The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module uses a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module. The rest of this implementation plan is the same as Specific Implementation Plan Seven.

[0131] Specific implementation scheme nine: In the laser radar three-dimensional point cloud target recognition module, the improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer;

[0132] The Backbone layer processes the input image and uses the plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel to obtain the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the feature map received from the Backbone layer and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer. The rest of this implementation plan is the same as the specific implementation plan seven.

[0133] Specific implementation scheme ten: The CBAM attention mechanism module includes a spatial attention mechanism module (SpatialAttention Module) and a channel attention mechanism module (Channel Attention Module). In the channel attention mechanism module, the feature map is first subjected to global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through a fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, that is:

[0134]

[0135] In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c Represents channel attention features;

[0136] The spatial attention mechanism module first performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function. Finally, this weight is multiplied by the feature map to complete the spatial attention weighting, that is:

[0137]

[0138] In the formula, f 7×7 represents a filter of size 7×7, M s Indicates the spatial attention feature. The rest of this implementation is the same as the specific implementation plan seven.

[0139] Example 1

[0140] In order to verify the effect of the present invention, the following examples are used to illustrate the effect of the present invention.

[0141] First, the improved YOLOv5s network model of the present invention was trained using the ApolloScape dataset and the augmented dataset; the training set was divided into a training set and a validation set using the 8:2 principle, the initial learning rate was set to 0.001, the Epoch was set to 300, and the training time lasted 12.5 hours. Fig.10 As shown in the figure, the horizontal axis represents the Epoch of training the model with all the data in the training set, and the vertical axis represents the training loss value avg_Loss. When the training reaches 300K, the training loss value gradually tends to be flat, and the training loss is 0.0064 at this time.

[0142] The model reliability of the trained improved YOLOv5s network is verified.

[0143] like Fig.11 As shown in the figure, the horizontal axis represents the Epoch of the model training with all the data in the training set, and the vertical axis represents the accuracy of the model training. It can be seen that as the Epoch increases, the accuracy of the verified model also gradually increases, and finally stabilizes at 0.992; Fig.12 As shown in the figure, the horizontal axis represents the Epoch of the model training with all the data in the training set, and the vertical axis represents the recall rate of the model training. It can be seen that with the increase of Epoch, the recall rate of the verified model is also gradually increasing, and finally stabilized at 0.956; the verification results of the model training show that the model accuracy and recall rate are both high, both exceeding 0.95, indicating that the improved YOLOv5s network model has high reliability.

[0144] According to the trained improved YOLOv5s network, a two-dimensional detector based on the improved YOLOv5s network is established.

[0145] The performance of the two-dimensional detector based on the improved YOLOv5s network was tested on the ApolloScape dataset. The test results showed that the FPS reached 92.1, which has high real-time performance and can greatly save running time and improve overall efficiency.

[0146] Then, the ApolloScape dataset and the augmented dataset were used to train the improved Complex-YOLOv4 network model of the present invention. The training set was divided into a training set and a validation set using the 8:2 principle. The Epoch was set to 100 and the initial learning rate was set to 0.0001. Fig.13 As shown in the figure, the horizontal axis represents the number of batches required to complete a network model training, and the vertical axis represents the training loss value Avg_Loss. After 60K iterations, the loss curve begins to gradually flatten out.

[0147] Verify the reliability of the model of the trained improved Complex-YOLOv4 network, such as Fig.14 As shown in the figure, the horizontal axis represents the Epoch of the model training with all the data in the training set, and the vertical axis represents the accuracy of the model training. It can be seen that as the Epoch increases, the accuracy of the verified model also gradually increases, and finally stabilizes at 0.981; Fig.15 As shown in the figure, the horizontal axis represents the Epoch of training the model with all the data in the training set, and the vertical axis represents the accuracy of the model training. It can be seen that with the increase of Epoch, the recall rate of the verified model is also gradually increasing, and finally stabilized at 0.976. The verification results of the model training show that the model accuracy and recall rate are both high, both exceeding 0.95, indicating that the improved Complex-YOLOv4 network model has high reliability and can be applied to subsequent high-standard target detection.

[0148] According to the trained improved Complex-YOLOv4 network model, a three-dimensional detector based on the improved Complex-YOLOv4 network is established. The performance of the three-dimensional detector based on the improved Complex-YOLOv4 network is tested on the ApolloScape dataset. The test results show that the FPS reaches 39, indicating that the three-dimensional detector has high real-time performance.

[0149] Finally, the established two-dimensional detector based on the improved YOLOv5s network model and the three-dimensional detector based on the improved Complex-YOLOv4 network model are fused; Fig.17As shown in the figure, the camera sensor transmits two-dimensional image data, and the improved YOLOv5s network model obtains the detection result of the target object in the two-dimensional space; the laser sensor transmits three-dimensional point cloud data, and the improved Complex YOLOv4 network model obtains the detection result of the target object in the three-dimensional space, that is, the three-dimensional detection frame, and projects this target detection result into the two-dimensional image; the detection results of the two are fused to the bounding box by using the spatial domain IoU matching method in image fusion, so that the bounding boxes of the target detection overlap to the greatest extent, and the result with the highest overlap is the final target detection result.

[0150] In haze weather, the detector based on multi-sensor fusion of this embodiment is used to perform target recognition, such as Fig.18 As shown in the figure, the experimental verification results in low-visibility haze scenes show that the target detection results of the strategy have no missed detection or false detection, and can adapt to the target recognition task in low-visibility haze scenes, and the FPS can reach up to 26.54, which can meet the real-time requirements of perceiving road information in low-visibility haze environments.

[0151] In night scenes, detection is easily affected by low visibility or vehicle lighting. The detector based on multi-sensor fusion in this embodiment is used for target recognition, such as Fig.19 As shown in the figure, a good recognition effect is achieved, and the missed detection and false detection rates are low. It has good detection accuracy and the FPS can reach up to 25.59, which can meet the real-time requirements of road information perception in low-visibility night environments.

[0152] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A target perception method based on multi-sensor image result fusion, characterized in that: The steps include: S1, collect environmental data through camera and laser radar to obtain camera image data and laser radar three-dimensional point cloud data; S2, extracting features from camera image data through the improved YOLOv5s network model to obtain camera image target recognition results; The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction. The improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer; The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module uses a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module; A feature fusion module, including a first downsampling module Downsample of the feature fusion module, a first attention mechanism module CBAM of the feature fusion module, a second downsampling module Downsample of the feature fusion module, a second attention mechanism module CBAM of the feature fusion module, and a third attention mechanism module CBAM of the feature fusion module; S3, through the improved Complex-YOLOv4 network model, feature extraction is performed on the laser radar three-dimensional point cloud data to obtain the recognition result of the target object; including the following steps: Step S3-1: Mapping the collected 3D point cloud data to a 2D image plane according to the color value corresponding to each point, so as to realize the 2D conversion of the point cloud data, that is, converting the 3D point cloud data into an RGB-Map; Step S3-2: Use the Mosaic method to augment the RGB-Map data; Step S3-3: inputting the augmented data and the original data into the improved Complex-YOLOv4 network model, performing feature extraction on the three-dimensional point cloud data of the laser radar, and obtaining a two-dimensional bounding box of the detection target; Step S3-4: Using the non-maximum suppression (NMS) method to remove redundant detection boxes of the target object, only retaining the detection results with the highest confidence, and using the extended boundary generation network (E-RPN) to generate a 3D candidate box for the retained bounding box with the highest confidence; Step S3-5: performing target detection on the reconverted three-dimensional candidate box, and obtaining the recognition result of the target object by performing target classification and bounding box regression in the three-dimensional candidate box; The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel. The improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer; The Backbone layer processes the input image, introduces three plug-and-play CBAM attention mechanism modules to amplify the sensitivity in space and channel, obtains the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer adds two downsampling modules to perform multi-scale feature fusion on the feature map of the Backbone layer received, and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer; S4: Fuse the camera image detection result with the three-dimensional space detection result to obtain the final detected target information.

2. The target perception method based on multi-sensor image result fusion according to claim 1 is characterized in that: The CBAM attention mechanism module includes a spatial attention mechanism module and a channel attention mechanism module. In the channel attention mechanism module, the feature map is first subjected to global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through a fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, namely: In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c Represents channel attention features.

3. The target perception method based on multi-sensor image result fusion according to claim 2 is characterized in that: The spatial attention mechanism module first performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function. Finally, this weight is multiplied by the feature map to complete the spatial attention weighting, that is: In the formula, f 7×7 represents a filter of size 7×7, M s Represents spatial attention features.

4. A target perception system based on multi-sensor image result fusion, characterized in that: include: The data acquisition module is used to collect environmental data through cameras and laser radars to obtain camera image data and laser radar three-dimensional point cloud data; The camera image target recognition module is used to extract features from camera image data through an improved YOLOv5s network model to obtain camera image target recognition results; The improved YOLOv5s network model adopts depth-separable convolution. When the image is output to the Head layer, the feature map is downsampled and multi-dimensionally spliced ​​with the features extracted by the Neck layer. At the same time, the plug-and-play CBAM attention mechanism module is used to allocate the weights of spatial and channel attention during feature extraction. The improved YOLOv5s network model includes a Backbone layer, a Neck layer, a feature fusion module and a Head layer; The Backbone layer processes the input image and extracts features from the input image to obtain a feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer performs multi-scale feature fusion on the received feature map and transmits the fused feature map to the feature fusion module and the Head layer; the feature fusion module uses a multi-layer feature fusion strategy to perform multi-dimensional splicing of features of the received feature map, and uses the CBAM attention mechanism module to allocate spatial and channel attention weights to the feature map, and transmits the processed feature map to the Neck layer and the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer and the feature fusion module; A feature fusion module, including a first downsampling module Downsample of the feature fusion module, a first attention mechanism module CBAM of the feature fusion module, a second downsampling module Downsample of the feature fusion module, a second attention mechanism module CBAM of the feature fusion module, and a third attention mechanism module CBAM of the feature fusion module; The LiDAR 3D point cloud target recognition module is used to extract features from the LiDAR 3D point cloud data through an improved Complex-YOLOv4 network model to obtain the recognition result of the target object; it includes the following steps: Step S3-1: Mapping the collected 3D point cloud data to a 2D image plane according to the color value corresponding to each point, so as to realize the 2D conversion of the point cloud data, that is, converting the 3D point cloud data into an RGB-Map; Step S3-2: Use the Mosaic method to augment the RGB-Map data; Step S3-3: inputting the augmented data and the original data into the improved Complex-YOLOv4 network model, performing feature extraction on the three-dimensional point cloud data of the laser radar, and obtaining a two-dimensional bounding box of the detection target; Step S3-4: Using the non-maximum suppression (NMS) method to remove redundant detection boxes of the target object, only retaining the detection results with the highest confidence, and using the extended boundary generation network (E-RPN) to generate a 3D candidate box for the retained bounding box with the highest confidence; Step S3-5: performing target detection on the reconverted three-dimensional candidate box, and obtaining the recognition result of the target object by performing target classification and bounding box regression in the three-dimensional candidate box; The improved Complex-YOLOv4 network model adopts deep separable convolution and uses a plug-and-play CBAM attention mechanism module to amplify the sensitivity in space and channel. The improved Complex-YOLOv4 network model includes a Backbone layer, a Neck layer and a Head layer; The Backbone layer processes the input image, introduces three plug-and-play CBAM attention mechanism modules to amplify the sensitivity in space and channel, obtains the feature map of the input image, and transmits the feature map to the Neck layer; the Neck layer adds two downsampling modules to perform multi-scale feature fusion on the feature map of the Backbone layer received, and transmits it to the Head layer; the Head layer performs multi-scale target detection on the feature map extracted by the Neck layer; The detection module is used to fuse the camera image detection results with the three-dimensional space detection results to obtain the final detection target information.

5. The target perception system based on multi-sensor image result fusion according to claim 4 is characterized in that: The CBAM attention mechanism module includes a spatial attention mechanism module and a channel attention mechanism module. In the channel attention mechanism module, the feature map is first subjected to global average pooling and global maximum pooling, and then the two outputs are connected, and then an output weight is obtained through a fully connected layer, and this weight is multiplied by the feature map, thereby completing the weighting of different channel features, namely: In the formula, MLP represents multi-layer perceptron, F represents the input feature map, M c Represents channel attention features; The spatial attention mechanism module first performs two convolution operations on the input in the vertical and horizontal directions, then adds the results in these two directions, and then obtains the weight of the spatial position through the Sigmoid activation function. Finally, this weight is multiplied by the feature map to complete the spatial attention weighting, that is: In the formula, f 7×7 represents a filter of size 7×7, M s Represents spatial attention features.

Citation Information

Patent Citations

  • Lightweight target detection method

    CN114120019A

  • Student classroom behavior detection method based on ESRGAN and improved YOLOv5s

    CN114241422A

  • Target detection method based on improved YOLOv5 and binocular stereo vision

    CN114565900A

  • Unmanned aerial vehicle aerial image river floating object detection method based on improved YOLOv5

    CN115115934A

  • YOLOv5 improvement-based substation equipment nut defect identification and detection method

    CN116342596A