BEV-based multi-sensor fusion scrap steel material pile feature extraction method

Abstract

The invention relates to a BEV-based multi-sensor fusion scrap steel material pile feature extraction method, and belongs to the technical field of computer vision and robots. According to the method, firstly, data acquired by a laser radar and a camera are preprocessed, noise is removed, and key information is reserved; and then, the preprocessed data is converted to a unified BEV view angle, so that subsequent feature extraction and fusion are facilitated. Then, a Point Net network and an improved ResNet network are used for extracting features of the point cloud data and the image data respectively; and finally, through a Deformable Attention mechanism and a multi-head attention mechanism, the two kinds of features are aligned and fused, and a fused BEV feature is generated. According to the method, the problems of data misalignment, insufficient feature extraction and the like in a waste steel material pile scene are solved, and the precision and robustness of identification, classification and positioning of the waste steel material pile are improved, so that technical support is provided for automation of waste steel material treatment.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and robotics, and relates to a BEV-based multi-sensor fusion scrap steel pile feature extraction method. Background Art

[0002] In the field of intelligent unmanned industrial automation, the development of industrial automation technology relies on computer vision and sensor fusion technology. LiDAR provides high-precision three-dimensional point cloud data, and the camera captures two-dimensional images of the environment. These data can provide rich environmental information through fusion. However, when processing such multi-source data, traditional algorithms face problems such as data misalignment, unstable perception results, and insufficient feature extraction. In addition, in many industrial scenarios, the complexity of the object being processed can also bring instability to the algorithm. For example, in a scrap steel yard, the environment is complex and changeable, and the shapes, sizes, positions, and occlusions of the steel materials are diverse, which will bring huge challenges to the automatic grasping task.

[0003] At present, the problems faced by perception tasks in intelligent unmanned scrap steel plants mainly include the following aspects: data fusion is difficult. The fusion of multi-sensor data requires the alignment and synchronization of different types of data (point clouds and images), which is particularly difficult in complex environments; low recognition accuracy. The dynamic changes in the environment, the diversity of steel materials, and occlusion make it difficult for traditional perception and recognition algorithms to ensure high accuracy and high robustness; poor environmental adaptability. The environment of the scrap steel yard is complex and changeable. Traditional algorithms have poor adaptability under different environmental conditions and are difficult to meet actual application needs.

[0004] Existing data fusion methods: In the actual scene of scrap steel piles, the lidar point cloud and image data in the perception task are usually processed separately and then simply fused. This processing method easily leads to data misalignment because the point cloud and image data come from different perspectives and sensors. Data misalignment will lead to inaccurate information after fusion, affecting the identification and grasping operations of scrap steel.

[0005] Existing feature extraction methods: In the scrap steel pile scenario, traditional methods often rely on simple convolutional neural networks (CNN) or traditional feature fusion methods for feature extraction and fusion. These methods are limited in their effectiveness in extracting complex features of scrap steel, resulting in low accuracy in recognition and classification, and are difficult to adapt to the environment of scrap steel piles with various shapes and complex stacking. Summary of the invention

[0006] In view of this, an object of the present invention is to provide a scrap steel pile feature extraction method based on BEV using multi-sensor fusion.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-sensor fusion scrap steel pile feature extraction method based on BEV, the method comprising the following steps:

[0009] Data preprocessing, including:

[0010] Acquire the three-dimensional point cloud data of the scrap steel pile from the LiDAR sensor, and perform statistical filtering and voxel filtering to remove noise and retain parameter information of the scrap steel pile;

[0011] Obtain two-dimensional image data of the scrap steel pile from the camera, and perform distortion correction and color enhancement to improve image quality and information content;

[0012] Perspective transformation, including:

[0013] Project the pre-processed point cloud data onto a two-dimensional plane to form a BEV perspective and obtain the distribution of scrap steel piles on the ground;

[0014] The image data is converted from the image coordinate system to the world coordinate system using the camera intrinsic matrix and extrinsic matrix, and mapped to the BEV perspective;

[0015] Multi-sensor data feature extraction, including:

[0016] Use the PointNet network to extract point cloud data features and obtain the feature representation of each point;

[0017] The improved ResNet network is used to extract image data features, and the details and features of the scrap steel pile image are better captured by adjusting the convolution kernel size and increasing the number of convolution layers.

[0018] Feature alignment and fusion, including:

[0019] Set up unified BEV Queries for feature alignment in BEV space;

[0020] Through the Deformable Attention mechanism, an offset is learned for each query point to adjust the query point position;

[0021] A multi-head attention mechanism is used for feature fusion to generate fused BEV features, which can better capture the complex information in the scrap steel pile scene, thereby realizing the identification, classification and positioning of the scrap steel pile.

[0022] Furthermore, the improved ResNet network includes:

[0023] Input layer: The size of input image data I is H×W×3, where H and W are the height and width of the image, representing pixels, and 3 represents the three color channels of the image;

[0024] Initial convolutional layer: Adjust the kernel size of the initial convolutional layer from 7×7 to 3×3 to capture details; increase the number of initial convolutional layers to extract low-level features:

[0025] F0=Conv 3×3,64,s＝1 (I)

[0026] F0 represents the feature map obtained after the first layer of convolution operation using a 3×3 convolution kernel, 64 output channels, and a step size of 1.

[0027] Batch normalization and activation function: Batch normalization and ReLU activation are performed on the convolution output; the formula is as follows:

[0028] F1 = ReLU(BN(F0))

[0029] Among them, BN stands for batch normalization, ReLU is the activation function, and the formulas are:

[0030] ReLU(x)=max(0,x)

[0031]

[0032] Among them, μ B is the mean of F0; is the variance of F0; ò is a small constant to prevent division by zero; γ, β are trainable parameters;

[0033] Max pooling layer: Use a 3×3 max pooling layer with a stride of 2 to reduce the feature map size, expressed as:

[0034]

[0035] Among them, F2(i,j,k) represents the value of the output feature map F2 at the (i,j)th position and the kth channel, and F1(2i+m,2j+n,k) represents the value of the input feature map F1 at the corresponding position and channel; m and n are the offsets of the local window, m,n∈{0,1,2};

[0036] Residual block: In ResNet, the residual block is an important component; the input feature map is F2, and the output feature map after the residual block is F img , the residual block consists of two convolutional layers and a skip connection; the specific process is:

[0037] The first convolution layer:

[0038] F conv1 =ReLU(BN(Conv 3×3,C,s＝1 (F2)))

[0039] Where C represents the number of channels of the input data, and s represents the step size:

[0040] The second convolution layer:

[0041] F conv2 =BN(Conv 3×3,C,s＝1 (F conv1 ))

[0042] Skip connections:

[0043] F img =F2+F conv2

[0044] That is, we get the image feature F img .

[0045] Furthermore, the Deformable Attention mechanism includes the following steps:

[0046] Set up unified BEV Queries for feature alignment in BEV space;

[0047] Through the Deformable Attention mechanism, an offset is learned for each query point to adjust the query point position;

[0048] Use the multi-head attention mechanism to perform feature fusion and generate fused BEV features.

[0049] Furthermore, the multi-sensor data feature extraction is replaced by a feature extraction method of an Inception network or a DenseNet model.

[0050] Furthermore, the use of a multi-head attention mechanism for feature fusion is replaced by the use of convolutional fusion or graph convolutional network GCN for feature fusion.

[0051] The beneficial effects of the present invention are:

[0052] (1) For data fusion, in complex environments such as scrap steel piles, the spatial relationships between objects and scenes are very important. The BEV perspective can clearly show these relationships, which helps to more accurately identify and classify different objects and improve overall perception capabilities. This solution provides a unified reference framework by converting point cloud and image data to the same BEV bird's-eye view. In this way, data from different sensors can be compared and fused in the same coordinate system, effectively solving the problem of data misalignment.

[0053] (2) Regarding the accuracy issue, existing technologies often rely on simple convolutional neural networks (CNN) or traditional feature fusion methods during feature extraction and fusion, resulting in limited accuracy in steel recognition and classification. These methods cannot fully mine the delicate features in multi-source data and are difficult to meet high-precision requirements. BEV-based perspective conversion helps to obtain more consistent and accurate spatial information, and in feature extraction, this solution mines more delicate features from multi-source data through improved ResNet and attention mechanisms. The accuracy of steel recognition and classification is significantly improved, meeting the high-precision requirements in industrial scenarios.

[0054] In order to solve the problem of data misalignment, this method uses a bird's eye view (BEV) transformation to unify the point cloud and image data into one perspective. This transformation ensures that the data is aligned in the same coordinate system, thus eliminating the data misalignment problem caused by perspective differences.

[0055] In order to solve the limitations of traditional methods in extracting complex features of scrap steel, this method uses the attention mechanism of Transformer to mine more delicate features from multi-source data. Based on the perspective conversion of BEV and combined with the multi-head attention mechanism of Transformer, deep feature fusion is achieved. This method can make full use of the complementary information of LiDAR and camera data to extract high-quality features.

[0056] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 It is a flow chart of the present invention;

[0059] Figure 2 Flowchart for feature extraction of image data. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0062] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] 1. Data preprocessing

[0064] See also Figure 1 and Figure 2 In the scrap steel pile perception task, data preprocessing is the basic link of the entire system. Its purpose is to preliminarily process and clean the raw data obtained from the lidar and camera to facilitate subsequent perception and classification tasks. The specific steps include the acquisition and filtering of point cloud data, and the acquisition and correction of image data.

[0065] (1) Point cloud data acquisition and filtering

[0066] The 3D point cloud data is obtained from the LiDAR sensor. This data contains detailed 3D structural information of the environment, including the shape and location of the scrap pile. However, the point cloud data is usually noisy and needs to be filtered.

[0067] Statistical Outlier Removal: Remove outliers (noise points) that are too far away by calculating the average neighborhood distance of each point. Assume that the average distance of the k nearest neighbors of a point in the point cloud is d i , after statistical filtering, the point cloud data is:

[0068] P filtered ={(x i ,y i ,z i )|d i <μ+α·σ}(1)

[0069] Among them, μ and σ are the mean and standard deviation of the average neighborhood distance, respectively, and α is the adjustment parameter.

[0070] Voxel Grid Filter: By dividing the space into voxel grids, a representative point is selected from each voxel to reduce the amount of point cloud data. Assume that the point cloud data is P raw ={(x i ,y i ,z i )}, after voxel filtering, the point cloud data becomes:

[0071]

[0072] Where N is the number of points in a voxel.

[0073] In actual scrap steel piles, the above-mentioned point cloud filtering process can effectively remove environmental noise, retain key steel pile information, and provide clear three-dimensional data for subsequent perception.

[0074] (2) Image data acquisition and correction

[0075] The two-dimensional image data obtained from the camera provides the color and texture information of the environment. However, the raw image data usually has lens distortion and needs to be corrected.

[0076] Undistortion: Use the camera internal parameters and distortion parameters to correct the image distortion and restore the real image. Assume that the image distortion parameters are k1, k2, p1, p2, and the original image point (u d ,v d ) The corrected point is (u,v), and the correction formula is:

[0077]

[0078] in, (u0,v0) is the center of the image.

[0079] Color Enhancement: Enhances the visual effect of an image by adjusting its brightness, contrast, and color saturation. Assuming the original image pixel value is, the color enhanced image is, and the color enhancement formula is:

[0080] I enhanced =Enhance(I raw )(4)

[0081] The specific operations of Enhance include brightness and contrast adjustment, namely:

[0082]

[0083] Among them, α and β are contrast and brightness adjustment parameters respectively.

[0084] 2. Perspective Transformation

[0085] Next, the preprocessed data will be transformed into a bird's eye view (BEV) to provide more intuitive and complete scrap distribution information. This is essential for the classification, positioning and automatic grasping path planning of scrap. Through perspective conversion, data from different sensors can be fused in the same coordinate system, thereby improving the accuracy and reliability of the perception system. The following details the conversion process of point cloud and image data to BEV.

[0086] (1) Conversion from point cloud to BEV

[0087] Project the 3D point cloud data onto a 2D plane to form a BEV perspective. Assume that the point cloud data is P = {(x i ,y i ,z i )}, the conversion formula is:

[0088] P BEV ={(x i ,y i )|(x i ,y i ,z i )∈P}(6)

[0089] In the actual scrap steel pile scenario, this means ignoring the height information (z-axis) of each point and retaining only the plane coordinates (x and y axes) to obtain the distribution of the scrap steel on the ground.

[0090] (2) Image to BEV conversion

[0091] The 2D image data is mapped to the BEV perspective through perspective conversion. The image points are converted to the world coordinate system using the camera's intrinsic parameter matrix K and extrinsic parameter matrix:

[0092] Convert image coordinates to camera coordinates

[0093] Assume that the image intrinsic parameter matrix K is:

[0094]

[0095] Among them, f x ,f y are the focal lengths of the camera in the horizontal and vertical directions, in pixels; c x ,c y It is the coordinates of the principal point of the image on the image, usually the coordinates of the center point of the image.

[0096] The image point (u, v) is transformed to the camera coordinate system (X c ,Y c ,Z c ):

[0097]

[0098] Convert camera coordinates to world coordinates

[0099] Assume the camera extrinsic matrix is ​​[R|t], where R is the rotation matrix and t is the translation:

[0100]

[0101] Among them, r 11 ,r 12 ,r 13 Describes the projection of the x-axis of the camera coordinate system in the world coordinate system. Similarly, r 21 ,r 22 ,r 23 and r 31 ,r 32 ,r 33 Describes the projection of the y-axis and z-axis of the camera coordinate system in the world coordinate system; t x ,t y ,t z Indicates the translation distance of the origin of the camera coordinate system relative to the origin of the world coordinate system on the x, y, and z axes.

[0102] Camera coordinate system (X c ,Y c ,Z c ) to the world coordinate system (X w ,Y w ,Z w ):

[0103]

[0104] 3. Multi-sensor data feature extraction and alignment

[0105] After converting the point cloud data and image data to the BEV view, it is necessary to extract features from them separately and align and fuse their features under the BEV view.

[0106] (1) Point cloud data feature extraction

[0107] Extracting Point Cloud Data Features This solution selects the classic PointNet network architecture and uses the multi-layer perceptron (MLP) layer to extract the features of each point in the point cloud data. Its special feature is that it can directly receive an unordered set of points as input, and process it through a series of MLP layers to finally output the feature representation of each point, that is, the feature set F lidar .

[0108] (2) Image data feature extraction

[0109] For image data, this solution selects the improved ResNet for image feature extraction, and improves the hierarchical structure of ResNet according to the image data of scrap steel pile to meet actual needs. ResNet solves the gradient vanishing and gradient exploding problems in deep networks by introducing residual blocks. Its core idea is to make the network easier to optimize through identity mapping. The specific process after improvement is as follows:

[0110] Input Layer

[0111] The size of the input image data I is H×W×3, where H and W are the height and width of the image, representing pixels, and 3 represents the three color channels (red, green, and blue) of the image.

[0112] Initial convolutional layer

[0113] Since the image of the scrap steel pile has high resolution and complex details, the larger convolution kernel of the initial convolution layer may cause detail loss. Therefore, according to the actual scene requirements, the convolution kernel size of the initial convolution layer is adjusted from 7×7 to 3×3 to better capture details. At the same time, the number of initial convolution layers is increased to better extract low-level features.

[0114] F0=Conv 3×3,64,s＝1 (I)(11)

[0115] F0 represents the feature map obtained after the first layer of convolution operation using a 3×3 convolution kernel, 64 output channels, and a step size of 1.

[0116] Batch Normalization and Activation Functions

[0117] Perform batch normalization and ReLU activation on the convolution output. The formula is as follows:

[0118] F1=ReLU(BN(F0))(12)

[0119] Among them, BN stands for batch normalization, ReLU is the activation function, and the formulas are:

[0120] ReLU(x)=max(0,x)(13)

[0121]

[0122] Among them, μ B is the mean of F0; is the variance of F0; ò is a small constant to prevent division by zero; γ, β are trainable parameters.

[0123] Max Pooling Layer

[0124] Here we use a 3×3 maximum pooling layer with a stride of 2 to reduce the feature map size, which can be expressed as:

[0125]

[0126] Among them, F2(i,j,k) represents the value of the output feature map F2 at the (i,j)th position and the kth channel, F1(2i+m,2j+n,k) represents the value of the input feature map F1 at the corresponding position and channel; m and n are the offsets of the local window, where m,n∈{0,1,2}.

[0127] Residual Block

[0128] In ResNet, the residual block is an important component. The input feature map is F2, and the output feature map after the residual block is F img , the residual block consists of two convolutional layers and a skip connection. The specific process is:

[0129] The first convolution layer:

[0130] F conv1 =ReLU(BN(Conv 3×3,C,s＝1 (F2)))(16)

[0131] Where C represents the number of channels of the input data, and s represents the step size:

[0132] The second convolution layer:

[0133] F conv2 =BN(Conv3×3,C,s＝1 (F conv1 ))(17)

[0134] Skip connections:

[0135] F img =F2+F conv2 (18)

[0136] In this way, the image feature F is obtained. img

[0137] 4. Feature alignment and fusion

[0138] The feature F extracted from the point cloud lidar and the image feature F img Finally, this solution uses the Deformable Attention mechanism to perform feature alignment. The following are the detailed steps:

[0139] (1) Setting up BEV Queries

[0140] Setting up unified BEV QueriesQ BEV , used to align features in the BEV space. Each BEV Query corresponds to a fixed position in the BEV space. Generate initial BEV Queries Q BEV :

[0141]

[0142] Among them, q i are query points in the BEV space, which are used for query operations in the attention mechanism. Usually, these query points cover the entire BEV space; N is the total number of query points.

[0143] (2) Calculate the offset Δp

[0144] Through the Deformable Attention mechanism, an offset is learned for each query point:

[0145] Δp k =MLP(F img ,F lidar ,Q BEV )(20)

[0146] MLP stands for Multi-Layer Perceptron. The calculated offset is used to adjust the position of the query point to achieve more accurate alignment of features.

[0147] (3) Feature alignment and fusion

[0148] Feature fusion is performed through a multi-head attention mechanism, which is based on the query point Q BEVWith feature F lidar and F img Match and generate fused BEV features:

[0149]

[0150] Where K represents the number of heads in the multi-head attention mechanism, and each head can focus on different feature parts; w k represents the weight coefficient of the kth attention head; p represents the fixed query position in the BEV space, Δp represents the offset; Attention is the attention mechanism function, which obtains the attention distribution by calculating the correlation between the query (Query), key (Key) and value (Value), thereby weighted summing the features.

[0151] Through the above steps, the point cloud data features and image data features are effectively aligned and fused under the attention mechanism of Transformer to form a unified feature representation F BEV This feature representation can better capture the complex information in the scrap steel pile scene and improve the accuracy of the perception task.

[0152] 1. Feature extraction method replacement: Currently, the improved ResNet and attention mechanism are used for feature extraction, which can be replaced by other deep learning models or feature extraction methods, such as Inception network, DenseNet, etc. These methods have their own advantages in feature extraction and can be replaced according to actual needs;

[0153] 2. Fusion strategy replacement: In the data fusion part, the multi-head attention mechanism is currently used, which can be replaced by other fusion strategies, such as convolutional fusion, graph convolutional network (GCN), etc. These methods have different advantages and applicable scenarios when processing multi-source data.

[0154] The present invention converts the data collected by the laser radar and the camera into the same BEV (Bird's Eye View) coordinate system to provide a unified reference frame. Through this conversion, the problem of misalignment of data from different sensors is solved, and the accuracy and consistency of data fusion are improved. In the feature extraction stage, an improved ResNet model is used to enhance the ability to extract delicate features from multi-source data.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-sensor fusion scrap steel pile feature extraction method based on BEV, characterized by: The method comprises the following steps: Data preprocessing, including: Acquire the three-dimensional point cloud data of the scrap steel pile from the LiDAR sensor, and perform statistical filtering and voxel filtering to remove noise and retain parameter information of the scrap steel pile; Obtain two-dimensional image data of the scrap steel pile from the camera, and perform distortion correction and color enhancement to improve image quality and information content; Perspective transformation, including: Project the pre-processed point cloud data onto a two-dimensional plane to form a BEV perspective and obtain the distribution of scrap steel piles on the ground; The image data is converted from the image coordinate system to the world coordinate system using the camera intrinsic matrix and extrinsic matrix, and mapped to the BEV perspective; Multi-sensor data feature extraction, including: Use the PointNet network to extract point cloud data features and obtain the feature representation of each point; The improved ResNet network is used to extract image data features, and the details and features of the scrap steel pile image are better captured by adjusting the convolution kernel size and increasing the number of convolution layers. Feature alignment and fusion, including: Set up unified BEV Queries for feature alignment in BEV space; Through the Deformable Attention mechanism, an offset is learned for each query point to adjust the query point position; A multi-head attention mechanism is used for feature fusion to generate fused BEV features, which can better capture the complex information in the scrap steel pile scene, thereby realizing the identification, classification and positioning of the scrap steel pile.

2. The method for extracting scrap steel pile features based on BEV by multi-sensor fusion according to claim 1, characterized in that: The improved ResNet network includes: Input layer: The size of input image data I is H×W×3, where H and W are the height and width of the image, representing pixels, and 3 represents the three color channels of the image; Initial convolutional layer: Adjust the kernel size of the initial convolutional layer from 7×7 to 3×3 to capture details; increase the number of initial convolutional layers to extract low-level features: F0=Conv 3×3,64,s＝1 (I) F0 represents the feature map obtained after the first layer of convolution operation using a 3×3 convolution kernel, 64 output channels, and a step size of 1. Batch normalization and activation function: Batch normalization and ReLU activation are performed on the convolution output; the formula is as follows: F1 = ReLU(BN(F0)) Among them, BN stands for batch normalization, ReLU is the activation function, and the formulas are: ReLU(x)=max(0,x) Among them, μ B is the mean of F0; is the variance of F0; ò is a small constant to prevent division by zero; γ, β are trainable parameters; Max pooling layer: Use a 3×3 max pooling layer with a stride of 2 to reduce the feature map size, expressed as: Among them, F2(i,j,k) represents the value of the output feature map F2 at the (i,j)th position and the kth channel, and F1(2i+m,2j+n,k) represents the value of the input feature map F1 at the corresponding position and channel; m and n are the offsets of the local window, m,n∈{0,1,2}; Residual block: In ResNet, the residual block is an important component; the input feature map is F2, and the output feature map after the residual block is F img , the residual block consists of two convolutional layers and a skip connection; the specific process is: The first convolution layer: <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> conv1 <h2 style=";text-align:left;direction:ltr"> =ReLU(BN(Conv<h2 style=";text-align:left;direction:ltr"> 3×3,C,s＝1 <h2 style=";text-align:left;direction:ltr"> (F2))) Where C represents the number of channels of the input data, and s represents the step size: The second convolution layer: F conv2 =BN(Conv 3×3,C,s＝1 (F conv1 )) Skip connections: F img =F2+F conv2 That is, we get the image feature F img .

3. The method for extracting scrap steel pile features based on BEV by multi-sensor fusion according to claim 1, characterized in that: The Deformable Attention mechanism includes the following steps: Set up unified BEV Queries for feature alignment in BEV space; Through the Deformable Attention mechanism, an offset is learned for each query point to adjust the query point position; Use the multi-head attention mechanism to perform feature fusion and generate fused BEV features.

4. The method for extracting scrap steel pile features based on BEV by multi-sensor fusion according to claim 1, characterized in that: The multi-sensor data feature extraction is replaced by a feature extraction method of an Inception network or a DenseNet model.

5. The method for extracting scrap steel pile features based on BEV by multi-sensor fusion according to claim 1, characterized in that: The use of a multi-head attention mechanism for feature fusion is replaced by the use of convolutional fusion or graph convolutional network GCN for feature fusion.

Images