Multi-level obstacle classification and recognition method for metal processing environment-oriented unmanned carrier

Abstract

This invention discloses a multi-level obstacle classification and recognition method for unmanned transport vehicles (UTVs) in metal processing environments. The method includes: acquiring RGB images and depth point cloud data from a binocular camera, mapping them to the same coordinate system to generate an RGB-D feature tensor; inputting the feature tensor into a dual-stream attention network to extract RGB texture features and depth geometric features, respectively, and performing pixel-level fusion through a cross-attention module; inputting the fused features into an adaptive weighted network resistant to specular interference, identifying specular regions by calculating gradient covariance, and using a reflection suppression layer to reduce the weight of specular regions, generating candidate obstacle bounding boxes; inputting the candidate region features into a recognition network for classification, and outputting obstacle category labels and confidence scores. This invention achieves accurate recognition of multi-level obstacles in metal processing environments, improving the perception reliability and operational safety of UTVs.

Description

Technical Field

[0001] This invention relates to the technical field of obstacle recognition, and more particularly to a multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments. Background Technology

[0002] With the rapid development of intelligent manufacturing technology, automated guided vehicles (AGVs) are increasingly widely used in metal processing workshops. The metal processing environment is characterized by strong lighting variations, complex metal surface reflections, and diverse obstacle types, which places extremely high demands on the AGV's environmental perception and obstacle recognition capabilities. Accurately identifying and classifying obstacles is one of the key technologies to ensure the safe and efficient operation of AGVs, directly affecting production safety and operational efficiency.

[0003] Currently, AGV obstacle recognition technology mainly relies on LiDAR, vision sensors, or a fusion of both. While traditional LiDAR-based methods can obtain accurate distance information, they face numerous challenges in metal processing environments. The high gloss reflection from metal workpiece surfaces causes specular reflection of the laser beam, resulting in distorted or lost ranging data, severely impacting accurate obstacle detection. Furthermore, LiDAR only provides geometric information and struggles to distinguish between obstacles of different materials and types, failing to meet the demands for refined classification.

[0004] Vision-based obstacle recognition methods can acquire rich texture and color information, but they also have significant limitations under strong lighting conditions in metal processing environments. Highlight areas on metal surfaces can form overexposed spots in RGB images, leading to feature extraction failures; while dark areas may lose important details. Existing RGB-D fusion methods, although combining color images and depth information, often employ simple stitching or weighting strategies in the feature fusion stage, failing to fully consider the correlation between RGB and depth features, resulting in poor fusion performance.

[0005] Furthermore, obstacles in metal processing environments exhibit multi-level and multi-scale characteristics, including static obstacles such as metal sheets and workpieces on the ground, dynamic obstacles such as forklifts and other AGVs, and aerial obstacles such as suspended hoists and pipes. Existing methods mostly employ single-scale feature extraction, which struggles to effectively capture the features of obstacles at different scales, resulting in poor performance in small target detection and long-distance recognition. Therefore, there is an urgent need to develop an obstacle recognition method tailored to the characteristics of metal processing environments, capable of effectively suppressing high-light interference, and achieving accurate multi-level classification. Summary of the Invention

[0006] In view of this, the present invention provides a multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments. The purpose is to achieve accurate identification and classification of multi-level obstacles in metal processing environments by constructing a highlight region adaptive suppression mechanism, a multi-scale point cloud feature extraction network, and an RGB-depth cross-attention fusion module, thereby improving the perception reliability and operational safety of unmanned transport vehicles in complex industrial environments.

[0007] To achieve the above objectives, the present invention provides a multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments, comprising the following steps: S1: Acquire the raw RGB image data and corresponding depth point cloud data collected by the front-view binocular camera of the unmanned transport vehicle, map the RGB image data to the depth point cloud coordinate system, and generate RGB-D four-dimensional feature tensor; S2: The RGB-D four-dimensional feature tensor is input into a feature extraction network based on a two-stream attention mechanism. The feature extraction network includes an RGB feature extraction branch and a deep geometric feature extraction branch. The RGB feature extraction branch uses convolutional layers to extract texture and color features to obtain RGB features, and the deep geometric feature extraction branch uses an improved point cloud feature extraction network to extract local geometric structure features to obtain deep geometric features. The RGB features and deep geometric features are fused at the pixel level through a cross-attention module to output a fused feature map containing multimodal information. S3: The fused feature map is input into an adaptive weighted region generation network that resists specular interference. The adaptive weighted region generation network first calculates the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map to identify the specular regions. Then, a reflection suppression weight layer is introduced to generate a spatial attention mask based on the identified specular regions, which reduces the weight of the specular region features in the fused feature map and enhances the non-spectral region features. Finally, a series of candidate region boxes containing potential obstacles are generated based on the weighted feature map. S4: Input the local features corresponding to the candidate region box into the recognition network for obstacle classification; the recognition network outputs the confidence scores of four types of obstacles for each candidate region, selects the category with the highest confidence score as the final recognition result, and outputs the specific category label and confidence score of all obstacles.

[0008] As a further improvement of the present invention: Optionally, in step S1, the step of mapping RGB image data to a depth point cloud coordinate system to generate an RGB-D four-dimensional feature tensor includes: S101. Obtain the intrinsic parameter matrix of the stereo camera. and extrinsic parameter matrix The intrinsic parameter matrix The extrinsic parameter matrix includes focal length and optical center coordinate parameters. Includes rotation matrix between left and right cameras Translation vector ; S102. For each 3D point in the depth point cloud ,in For point X-axis coordinates in the camera coordinate system For point The Y-axis coordinate value in the camera coordinate system. For point The Z-axis coordinates in the camera coordinate system are calculated using a projection transformation. The corresponding pixel coordinates on the RGB image plane The calculation formula is: ; in, For the first The horizontal pixel coordinates of a point cloud point in an RGB image For the first The vertical pixel coordinates of a point cloud point in an RGB image. This is the intrinsic parameter matrix of the stereo camera; S103. Based on the pixel coordinates Extracting RGB color values ​​at corresponding locations from an RGB image ,in pixel coordinates The red channel value at the corresponding position, pixel coordinates The green channel value at the corresponding position, pixel coordinates The blue channel value at the corresponding position; S104. Merge the three-dimensional spatial coordinates of each point cloud point with its corresponding RGB color value to generate a six-dimensional data point. And organized into an RGB-D four-dimensional feature tensor. ,in For characteristic tensors The height dimension, For characteristic tensors Width dimension For characteristic tensors The number of channels and , Represents the set of real numbers. The first three channels store the X, Y, and Z coordinates respectively, and the last three channels store... , , Color information.

[0009] Optionally, in step S2, the step of extracting local geometric structure features using an improved point cloud feature extraction network in the deep geometric feature extraction branch includes: S201. From the RGB-D four-dimensional feature tensor Extract the first three channels to form the depth coordinate tensor ,in Indicates pixel position The corresponding three-dimensional spatial coordinates; S202. Convert the depth coordinate tensor Reassembled into point cloud data format ,in , The total number of points in the point cloud. For the first Three-dimensional coordinate vectors of points; S203. The improved point cloud feature extraction network adopts a hierarchical sampling and feature aggregation architecture, which includes two scale levels. Each scale level includes a sampling layer, a grouping layer and a feature extraction layer. S204. At the first scale level, use the farthest point sampling algorithm from the point cloud data. Medium sampling There are 1 central point, among which This forms the first layer of center point set. For each center point Surrounding radius Search for neighboring points within a range of meters to form the first... local point set For each local point set A three-layer multilayer perceptron is used to extract local features, and the local features are aggregated through max pooling to obtain the first layer features. ; S205. At the second scale level, from the set of center points of the first level. Medium sampling These central points form the second layer of the central point set. For each second-layer center point, within the radius Search for neighborhood points within a range of meters, apply a three-layer multilayer perceptron to each local point set to extract local features, and obtain the second layer features through max pooling. ; S206. The features extracted from the two scale levels are upsampled and fused through a feature propagation layer, and a distance-based interpolation method is used to merge the second-layer features. Upsampled to the center point of the first layer, and compared with the features of the first layer. After stitching, the features are fused using two layers of multilayer perceptrons; finally, the fused features are upsampled to the original point cloud resolution, and the local geometric structure features of each point are obtained through two layers of multilayer perceptrons. S207. Reconstruct the local geometric structure features from the point cloud format back into the image format to obtain the depth geometric feature tensor. , where 128 is the number of channels for the depth geometric features.

[0010] Furthermore, in step S2, the step of pixel-level fusion of RGB features and deep geometric features using a cross-attention module includes: S211. Optimization of the RGB-D four-dimensional feature tensor through the RGB feature extraction branch. The last three channels are subjected to convolution operations. The RGB feature extraction branch contains four convolutional blocks, each containing a convolutional layer, a batch normalization (BN) layer, and a ReLU activation function, ultimately outputting an RGB feature tensor. ,in , 256 represents the number of channels for the RGB feature; S212. The depth geometric feature tensor is downsampled using a downsampling operation. Adjust to the same spatial resolution as the RGB feature, and use bilinear interpolation. From resolution downsampling to Then, the number of channels is adjusted to 256 through a convolutional layer to obtain the adjusted depth geometric features. ; S213. Convert the RGB feature tensor Generate query vectors through linear transformation layer Through convolutional layers Implement; the adjusted depth geometry features Key vectors are generated through two independent linear transformation layers. Sum value vector All are achieved through convolutional layers Implementation; query vector Key vector Sum value vector Reshape them into two-dimensional matrix forms respectively; S214. Calculate the cross-attention weight matrix The calculation formula is: ; in, This is the cross-attention weight matrix. Key vector matrix The transpose of , where 256 is the dimension of the key vector, i.e., the number of channels. for function; S215. Based on the cross-attention weight matrix value vector We perform weighted summation to generate attention output features, calculated using the following formula: ; in Output the feature matrix for attention; S216. Output attention features Reconstructing back to image format By residual connection with the original RGB feature tensor The feature maps are then fused to generate the final fused feature map. .

[0011] Optionally, in step S3, the step of calculating the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map and identifying the highlight region includes: S301. From the RGB-D four-dimensional feature tensor Extract RGB color information and calculate the position of each pixel. brightness value ; S302. Regarding the brightness value Calculating the brightness gradient using the Sobel operator ; S303. From the RGB-D four-dimensional feature tensor The third channel from which depth coordinate information is extracted yields the depth value tensor. ,in Indicates pixel position The depth value of the corresponding point in the camera coordinate system is the Z-axis coordinate value; for the depth tensor Calculating depth gradient using the Sobel operator ; S304. For each pixel position Defined by pixel position A square neighborhood window centered on In the neighborhood window Internal calculation of brightness gradient With depth gradient covariance ; S305. Based on the covariance value and brightness gradient To determine the highlight area, the criteria are: like and Then the pixel position Marked as highlight area; in The covariance threshold, The brightness gradient threshold; Generate specular region mask When pixel position When the criteria for judging highlight areas are met ,otherwise .

[0012] Furthermore, in step S3, the steps of generating a spatial attention mask based on the identified highlight regions, reducing the weight of highlight region features in the fused feature map, and enhancing non-highlight region features include: S311. Mask the highlight area. Downsampling to the fused feature map using max pooling operation The resolution is used to obtain the downsampled highlight mask. ; S312. Based on downsampled specular mask Generate reflection suppression weight tensor The calculation formula is: ; in, pixel position The reflection suppression weight value at the location, The enhancement weighting coefficient for non-highlight regions. This represents the weighting factor for the highlight region. The pixel position after downsampling The mask value for the highlight area at that location; S313. The reflection suppression weight tensor Extended to fusion feature maps Same dimensions; expand the weight tensor With the fused feature map Perform element-wise multiplication to generate a weighted feature map. ; S314. Based on the weighted feature map A Region Candidate Network (RPN) is constructed to generate candidate region boxes, and the classification scores and bounding box regression parameters of the candidate region boxes are obtained. S315. In the feature map Each position definition Anchor frames of different sizes and aspect ratios, with dimensions of... pixels, aspect ratio For each anchor box, determine whether it contains obstacles based on the classification score. If the foreground score is greater than the threshold of 0.7, the anchor box is retained. Apply bounding box regression parameters to adjust the position and size of the anchor box to generate accurate candidate region boxes. S316. Apply Non-maximum Suppression (NMS) to all retained candidate region boxes; after NMS, retain the top 2000 candidate region boxes with the highest scores. S317. Each candidate region box is a quadruple. It means that among them This represents the horizontal coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. This represents the vertical coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. The width of the candidate region bounding box. This represents the height of the candidate region bounding box.

[0013] Optionally, in step S4, the step of inputting the local features corresponding to the candidate region box into the fine-grained recognition network for obstacle classification includes: S401. Regarding the first Candidate region boxes From the weighted feature map Extract local features from the corresponding regions; use bilinear interpolation to uniformly adjust candidate regions of different sizes to a fixed-size 7×7×256 local feature tensor. ; S402. Convert the 7×7×256 local feature tensor Flattened into a one-dimensional feature vector ; Flatten the feature vector The input is fed into a fully connected classification network; the fully connected classification network consists of three fully connected layers and outputs... Unnormalized scores of each category of obstacles ; S404. (Regarding) The unnormalized scores of each category of obstacle are applied using the Softmax function to generate... Confidence vector of obstacle class ,in For the first Confidence level of obstacle class This represents the total number of obstacle categories. S405. Select the category with the highest confidence level as the first... The final identification result of the candidate region; if the confidence level of the final identification result is less than 0.5, then the candidate region will be... Each candidate region is marked as background or unknown obstacle and discarded; S406. For all candidate regions, output a list of obstacle recognition results. Each result includes: coordinates of the candidate region bounding box. Category labels, confidence levels, and complete 5D confidence vector .

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: This invention accurately identifies specular reflection regions on metal surfaces by calculating the covariance of brightness and depth gradients, and adaptively suppresses the RGB features of these regions while enhancing the contribution of depth features. This mechanism significantly reduces the negative impact of highlight regions on feature extraction while preserving effective texture information. Compared to traditional histogram equalization or simple thresholding methods, the adaptive suppression strategy of this invention can dynamically adjust according to local lighting conditions, avoiding information loss due to over-suppression and ensuring the discriminativeness and robustness of features.

[0015] This invention employs a hierarchical structure with progressive downsampling, enabling the network to perceive both the overall shape of large obstacles and the detailed features of small targets. Each scale level ensures the spatial uniformity of sampling points through a farthest-point sampling strategy, while simultaneously using a ball query method to aggregate the geometric information of neighboring points.

[0016] The RGB-D deep cross-attention fusion module designed in this invention achieves deep collaboration of heterogeneous features. Unlike traditional feature concatenation or simple weighted fusion methods, this invention dynamically learns the correlation and complementarity of the two modalities by calculating the attention weight matrix between RGB features and deep features. The cross-attention mechanism enables RGB features to focus on important spatial regions based on depth information, while deep features can also enhance semantic discriminative capabilities with the help of RGB information. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the confidence distribution curves for different modalities. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0019] Example 1: A multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments, such as... Figure 1 As shown, it includes the following steps: S1: Acquire the raw RGB image data and corresponding depth point cloud data captured by the forward-looking binocular camera of the unmanned transport vehicle, map the RGB image data to the depth point cloud coordinate system, and generate an RGB-D four-dimensional feature tensor, including: In this embodiment, step S1, which maps RGB image data to a depth point cloud coordinate system to generate an RGB-D four-dimensional feature tensor, includes: S101. Obtain the intrinsic parameter matrix of the stereo camera. and extrinsic parameter matrix The intrinsic parameter matrix The extrinsic parameter matrix includes focal length and optical center coordinate parameters. Includes rotation matrix between left and right cameras Translation vector ; S102. For each 3D point in the depth point cloud ,in For point X-axis coordinates in the camera coordinate system For point The Y-axis coordinate value in the camera coordinate system. For point The Z-axis coordinates in the camera coordinate system are calculated using a projection transformation. The corresponding pixel coordinates on the RGB image plane The calculation formula is: ; in, For the first The horizontal pixel coordinates of a point cloud point in an RGB image For the first The vertical pixel coordinates of a point cloud point in an RGB image. This is the intrinsic parameter matrix of the stereo camera; When there are points in the depth point cloud that have missing or abnormal depth values, i.e. When the projection transformation of a point is not possible, a depth repair method based on neighborhood interpolation is adopted: for a point with abnormal depth, a valid depth value is searched within its 3×3 neighborhood window, and the median depth of the valid neighborhood points is calculated as the repair depth value; if all depth values ​​within the neighborhood window are abnormal, the point is directly discarded. S103. Based on the pixel coordinates Extracting RGB color values ​​at corresponding locations from an RGB image ,in pixel coordinates The red channel value at the corresponding position, pixel coordinates The green channel value at the corresponding position, pixel coordinates The blue channel value at the corresponding position; S104. Merge the three-dimensional spatial coordinates of each point cloud point with its corresponding RGB color value to generate a six-dimensional data point. And organized into an RGB-D four-dimensional feature tensor. ,in For characteristic tensors The height dimension, For characteristic tensors Width dimension For characteristic tensors The number of channels and , Represents the set of real numbers. The first three channels store the X, Y, and Z coordinates respectively, and the last three channels store... , , Color information.

[0020] S2: The RGB-D four-dimensional feature tensor is input into a feature extraction network based on a two-stream attention mechanism. This feature extraction network includes an RGB feature extraction branch and a deep geometric feature extraction branch. The RGB feature extraction branch uses convolutional layers to extract texture and color features to obtain RGB features, while the deep geometric feature extraction branch uses an improved point cloud feature extraction network to extract local geometric structure features to obtain deep geometric features. The RGB features and deep geometric features are then fused at the pixel level using a cross-attention module, outputting a fused feature map containing multimodal information, including: In this embodiment, step S2, where the deep geometric feature extraction branch uses an improved point cloud feature extraction network to extract local geometric structure features, includes: S201. From the RGB-D four-dimensional feature tensor Extract the first three channels to form the depth coordinate tensor ,in Indicates pixel position The corresponding three-dimensional spatial coordinates; S202. Convert the depth coordinate tensor Reassembled into point cloud data format ,in , The total number of points in the point cloud. For the first Three-dimensional coordinate vectors of points; S203. The improved point cloud feature extraction network adopts a hierarchical sampling and feature aggregation architecture, which includes two scale levels. Each scale level includes a sampling layer, a grouping layer and a feature extraction layer. S204. At the first scale level, use the farthest point sampling algorithm from the point cloud data. Medium sampling There are 1 central point, among which This forms the first layer of center point set. For each center point Surrounding radius Search for neighboring points within a range of meters to form the first... local point set For each local point set Application of three-layer multilayer perceptron Local features are extracted, where the three layers of the MLP have 64, 64, and 128 neurons respectively. The local features are aggregated through max pooling to obtain the first layer of features. ; When the point cloud density is uneven or there are a large number of noisy points, an adaptive sampling algorithm based on normal vector consistency can be used to replace the farthest point sampling algorithm. The specific steps of the adaptive sampling algorithm based on normal vector consistency are as follows: for each candidate point, calculate the covariance matrix of its k nearest neighbor set, and obtain the eigenvector corresponding to the smallest eigenvalue through eigenvalue decomposition as the normal vector; calculate the average value of the angle between the normal vectors of adjacent points as the geometric complexity index, and prioritize sampling points with high geometric complexity. S205. At the second scale level, from the set of center points of the first level. Medium sampling These central points form the second layer of the central point set. For each second-layer center point, within the radius Search for neighborhood points within a range of meters, and apply the following to each local point set: Extract local features and obtain the second layer of features through max pooling. ; S206. The features extracted from the two scale levels are upsampled and fused through a feature propagation layer, and a distance-based interpolation method is used to merge the second-layer features. Upsampled to the center point of the first layer, and compared with the features of the first layer. After splicing, pass The features are then fused; finally, the fused features are upsampled to the original point cloud resolution. Obtain the local geometric structure features of each point; S207. Reconstruct the local geometric structure features from the point cloud format back into the image format to obtain the depth geometric feature tensor. , where 128 is the number of channels for the depth geometric features.

[0021] In this embodiment, step S2, which involves pixel-level fusion of RGB features and depth geometric features using a cross-attention module, includes: S211. Optimization of the RGB-D four-dimensional feature tensor through the RGB feature extraction branch. The last three channels are subjected to convolution operations. The RGB feature extraction branch contains four convolutional blocks, and the first convolutional block contains a convolutional layer. The first convolutional block contains batch normalized (BN) layers and ReLU activation functions, where the four parameters of Conv represent 3 input channels, 64 output channels, a 3×3 kernel size, and a stride of 1. BN and ReLU are used, with a stride of 2 for downsampling; the third convolutional block contains BN and ReLU; the fourth convolutional block contains Batch Normalization (BN) and ReLU; the final output is an RGB feature tensor. ,in , 256 represents the number of channels for the RGB feature; S212. The depth geometric feature tensor is downsampled using a downsampling operation. Adjust to the same spatial resolution as the RGB feature, and use bilinear interpolation. From resolution downsampling to Then through convolutional layers Adjusting the number of channels to 256 yields the adjusted depth geometry. ; S213. Convert the RGB feature tensor Generate query vectors through linear transformation layer Through convolutional layers Implement; the adjusted depth geometry features Key vectors are generated through two independent linear transformation layers. Sum value vector All are achieved through convolutional layers Implementation; query vector Key vector Sum value vector Reshape them into two-dimensional matrix forms respectively; S214. Calculate the cross-attention weight matrix The calculation formula is: ; in, This is the cross-attention weight matrix. Key vector matrix The transpose of , where 256 is the dimension of the key vector, i.e., the number of channels. for function; S215. Based on the cross-attention weight matrix value vector We perform weighted summation to generate attention output features, calculated using the following formula: ; in Output the feature matrix for attention; S216. Output attention features Reconstructing back to image format By residual connection with the original RGB feature tensor The feature maps are then fused to generate the final fused feature map. The calculation formula is: ; in To fuse feature maps.

[0022] S3: The fused feature map is input into an adaptive weighted region generation network that resists specular interference. The adaptive weighted region generation network first calculates the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map to identify specular regions. Then, a reflection suppression weight layer is introduced to generate a spatial attention mask based on the identified specular regions, reducing the weight of the specular region features in the fused feature map and enhancing the non-spectral region features. Finally, a series of candidate region boxes containing potential obstacles are generated based on the weighted feature map, including: In this embodiment, step S3, which involves calculating the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map to identify the highlight region, includes: S301. From the RGB-D four-dimensional feature tensor Extract RGB color information and calculate the position of each pixel. brightness value The calculation formula is: in pixel position brightness value, , , Pixel positions The red, green, and blue channel values; S302. Regarding the brightness value Calculating the brightness gradient using the Sobel operator The Sobel operator includes horizontal direction operators. and vertical direction operator ,in: ; Calculate the horizontal gradient component and vertical gradient components , Demonstrates convolution operation; brightness gradient magnitude The calculation formula is: ; in pixel position The brightness gradient magnitude; pixel position The magnitude of the horizontal gradient component; pixel position The magnitude of the vertical gradient component; S303. From the RGB-D four-dimensional feature tensor The third channel from which depth coordinate information is extracted yields the depth value tensor. ,in Indicates pixel position The depth value of the corresponding point in the camera coordinate system is the Z-axis coordinate value; for the depth tensor Calculating depth gradient using the Sobel operator ; S304. For each pixel position Defined by pixel position A square neighborhood window centered on The window side length is 5 pixels; in the neighboring window Internal calculation of brightness gradient With depth gradient covariance ; S305. Based on the covariance value and brightness gradient To determine the highlight area, the criteria are: like and Then the pixel position Marked as highlight area; in The covariance threshold, The brightness gradient threshold; Generate specular region mask When pixel position When the criteria for judging highlight areas are met ,otherwise .

[0023] In this embodiment, step S3, which involves generating a spatial attention mask based on the identified highlight regions, performing weight reduction processing on the highlight region features in the fused feature map, and enhancing the non-highlight region features, includes: S311. Mask the highlight area. Downsampling to the fused feature map using max pooling operation The resolution is used to obtain the downsampled highlight mask. ; S312. Based on downsampled specular mask Generate reflection suppression weight tensor The calculation formula is: ; in, pixel position The reflection suppression weight value at the location, The enhancement weighting coefficient for non-highlight regions. This represents the weighting factor for the highlight region. The pixel position after downsampling The mask value for the highlight area at that location; S313. The reflection suppression weight tensor Extended to fusion feature maps Same dimensions; expand the weight tensor With the fused feature map Perform element-wise multiplication to generate a weighted feature map. ; S314. Based on the weighted feature map A Region Candidate Network (RPN) is constructed to generate candidate region boxes; the RPN contains a 3×3 convolutional layer. Features are extracted, and then the classification score and bounding box regression parameters are output through two 1×1 convolutional layers respectively; the classification score is obtained by... Output, where The number of anchor boxes at each location, where 2 indicates foreground / background binary classification; bounding box regression parameters are obtained through... The output, 4, represents the four coordinate offsets of the bounding box. ; S315. In the feature map Each position definition Anchor frames of different sizes and aspect ratios, with dimensions of... pixels, aspect ratio For each anchor box, determine whether it contains obstacles based on the classification score. If the foreground score is greater than the threshold of 0.7, the anchor box is retained. Apply bounding box regression parameters to adjust the position and size of the anchor box to generate accurate candidate region boxes. S316. Apply non-maximum suppression (NMS) to all retained candidate region boxes, with a suppression threshold of 0.5. That is, if the intersection-union ratio (IoU) of two candidate boxes is greater than 0.5, the candidate box with the higher classification score is retained, and the candidate box with the lower score is removed. After NMS, the top 2000 candidate region boxes with the highest scores are retained. S317. Each candidate region box is a quadruple. It means that among them This represents the horizontal coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. This represents the vertical coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. The width of the candidate region bounding box. This represents the height of the candidate region bounding box.

[0024] S4: Input the local features corresponding to the candidate region bounding box into the recognition network for obstacle classification; the recognition network outputs the confidence scores of four obstacle categories for each candidate region, selects the category with the highest confidence score as the final recognition result, and outputs the specific category labels and confidence scores of all obstacles, including: In this embodiment, step S4, which involves inputting the local features corresponding to the candidate region box into the fine-grained recognition network for obstacle classification, includes: S401. Regarding the first Candidate region boxes From the weighted feature map Extract local features from the corresponding regions; use bilinear interpolation to uniformly adjust candidate regions of different sizes to a fixed-size 7×7×256 local feature tensor. ; S402. Convert the 7×7×256 local feature tensor Flattened into a one-dimensional feature vector ; Flatten the feature vector The input is fed into a fully connected classification network; the fully connected classification network consists of three fully connected layers, the first layer... The feature dimension is mapped from 12544 to 512, followed by batch normalization (BN) and ReLU activation functions; the second layer... The feature dimension is mapped from 512 to 128, followed by batch normalization (BN) and ReLU activation functions; the third layer... Mapping the feature dimension from 128 to 4, the output is the unnormalized score for the four categories of obstacles. ; S404. Apply the Softmax function to the unnormalized scores of the four obstacle categories to generate confidence vectors for the four obstacle categories. ,in For the first Confidence level of obstacle class This represents the total number of obstacle categories in this embodiment. , These correspond to four types of obstacles: fixed equipment, temporary storage, workers, and other vehicles. S405. Select the category with the highest confidence level as the first... The final identification result of the candidate region; if the confidence level of the final identification result is less than 0.5, then the candidate region will be... Each candidate region is marked as background or unknown obstacle and discarded; S406. For all candidate regions, output a list of obstacle recognition results. Each result includes: coordinates of the candidate region bounding box. Category labels, confidence scores, and a complete 4-dimensional confidence vector. ;like Figure 2 As shown, Figure 2 The confidence distribution of different schemes for the difficult-to-identify category of temporary stockpiles is shown. The average confidence of the scheme of this invention is 0.87, which is significantly higher than the 0.63-0.71 of the single-modal scheme, proving the effectiveness of the cross-attention mechanism for multimodal feature fusion.

[0025] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0026] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0027] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments, characterized in that, Includes the following steps: S1: Acquire the raw RGB image data and corresponding depth point cloud data collected by the front-view binocular camera of the unmanned transport vehicle, map the RGB image data to the depth point cloud coordinate system, and generate RGB-D four-dimensional feature tensor; S2: Input the RGB-D four-dimensional feature tensor into a feature extraction network based on a two-stream attention mechanism. The feature extraction network includes an RGB feature extraction branch and a deep geometric feature extraction branch. The RGB feature extraction branch uses convolutional layers to extract texture and color features to obtain RGB features, while the deep geometric feature extraction branch uses an improved point cloud feature extraction network to extract local geometric structure features to obtain deep geometric features. The RGB features and deep geometric features are fused at the pixel level through a cross-attention module to output a fused feature map containing multimodal information. S3: Input the fused feature map into an adaptive weighted region generation network that resists specular interference; the adaptive weighted region generation network first calculates the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map to identify the specular region; Subsequently, a reflection suppression weight layer is introduced to generate a spatial attention mask based on the identified highlight regions. The highlight region features in the fused feature map are weighted down, while the non-highlight region features are enhanced. Finally, a series of candidate region boxes containing potential obstacles are generated based on the weighted feature map. S4: Input the local features corresponding to the candidate region box into the recognition network for obstacle classification; the recognition network outputs the confidence scores of four types of obstacles for each candidate region, selects the category with the highest confidence score as the final recognition result, and outputs the specific category label and confidence score of all obstacles.

2. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 1, characterized in that, In step S1, the step of mapping RGB image data to a depth point cloud coordinate system to generate an RGB-D four-dimensional feature tensor includes: S101. Obtain the intrinsic parameter matrix of the stereo camera. and extrinsic parameter matrix The intrinsic parameter matrix The extrinsic parameter matrix includes focal length and optical center coordinate parameters. Includes rotation matrix between left and right cameras Translation vector ; S102. For each 3D point in the depth point cloud ,in For point X-axis coordinates in the camera coordinate system For point The Y-axis coordinate value in the camera coordinate system. For point The Z-axis coordinates in the camera coordinate system are calculated using a projection transformation. The corresponding pixel coordinates on the RGB image plane The calculation formula is: ； in, For the first The horizontal pixel coordinates of a point cloud point in an RGB image For the first The vertical pixel coordinates of a point cloud point in an RGB image. This is the intrinsic parameter matrix of the stereo camera; S103. Based on the pixel coordinates Extracting RGB color values ​​at corresponding locations from an RGB image ,in pixel coordinates The red channel value at the corresponding position, pixel coordinates The green channel value at the corresponding position, pixel coordinates The blue channel value at the corresponding position; S104. Merge the three-dimensional spatial coordinates of each point cloud point with its corresponding RGB color value to generate a six-dimensional data point. And organized into an RGB-D four-dimensional feature tensor. ,in For characteristic tensors The height dimension, For characteristic tensors Width dimension For characteristic tensors The number of channels and , Represents the set of real numbers. The first three channels store the X, Y, and Z coordinates respectively, and the last three channels store... , , Color information.

3. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 2, characterized in that, In step S2, the step of extracting local geometric structure features using an improved point cloud feature extraction network in the deep geometric feature extraction branch includes: S201. From the RGB-D four-dimensional feature tensor Extract the first three channels to form the depth coordinate tensor ,in Indicates pixel position The corresponding three-dimensional spatial coordinates; S202. Convert the depth coordinate tensor Reassembled into point cloud data format ,in , The total number of points in the point cloud. For the first Three-dimensional coordinate vectors of points; S203. The improved point cloud feature extraction network adopts a hierarchical sampling and feature aggregation architecture, which includes two scale levels. Each scale level includes a sampling layer, a grouping layer and a feature extraction layer. S204. At the first scale level, use the farthest point sampling algorithm from the point cloud data. Medium sampling There are 1 central point, among which This forms the first layer of center point set. For each center point Surrounding radius Search for neighboring points within a range of meters to form the first... local point set For each local point set A three-layer multilayer perceptron is used to extract local features, and the local features are aggregated through max pooling to obtain the first layer features. ; S205. At the second scale level, from the set of center points of the first level. Medium sampling These central points form the second layer of the central point set. For each second-layer center point, within the radius Search for neighborhood points within a range of meters, apply a three-layer multilayer perceptron to each local point set to extract local features, and obtain the second layer features through max pooling. ; S206. The features extracted from the two scale levels are upsampled and fused through a feature propagation layer, and a distance-based interpolation method is used to merge the second-layer features. Upsampled to the center point of the first layer, and compared with the features of the first layer. After stitching, the features are fused using two layers of multilayer perceptrons; finally, the fused features are upsampled to the original point cloud resolution, and the local geometric structure features of each point are obtained through two layers of multilayer perceptrons. S207. Reconstruct the local geometric structure features from the point cloud format back into the image format to obtain the depth geometric feature tensor. , where 128 is the number of channels for the depth geometric features.

4. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 3, characterized in that, In step S2, the step of performing pixel-level fusion of RGB features and deep geometric features through the cross-attention module includes: S211. Optimization of the RGB-D four-dimensional feature tensor through the RGB feature extraction branch. The last three channels are subjected to convolution operations. The RGB feature extraction branch contains four convolutional blocks, each containing a convolutional layer, a batch normalization (BN) layer, and a ReLU activation function, ultimately outputting an RGB feature tensor. ,in , 256 represents the number of channels for the RGB feature; S212. The depth geometric feature tensor is downsampled using a downsampling operation. Adjust to the same spatial resolution as the RGB feature, and use bilinear interpolation. From resolution downsampling to Then, the number of channels is adjusted to 256 through a convolutional layer to obtain the adjusted depth geometric features. ; S213. Convert the RGB feature tensor Generate query vectors through linear transformation layer Through convolutional layers Implement; the adjusted depth geometry features Key vectors are generated through two independent linear transformation layers. Sum value vector All are achieved through convolutional layers Implementation; query vector Key vector Sum value vector Reshape them into two-dimensional matrix forms respectively; S214. Calculate the cross-attention weight matrix The calculation formula is: ； in, This is the cross-attention weight matrix. Key vector matrix The transpose of , where 256 is the dimension of the key vector, i.e., the number of channels. for function; S215. Based on the cross-attention weight matrix value vector We perform weighted summation to generate attention output features, calculated using the following formula: ； in Output the feature matrix for attention; S216. Output attention features Reconstructing back to image format By residual connection with the original RGB feature tensor The feature maps are then fused to generate the final fused feature map. .

5. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 4, characterized in that, In step S3, the step of calculating the covariance matrix of the brightness gradient and depth gradient of each pixel in the fused feature map and identifying the highlight region includes: S301. From the RGB-D four-dimensional feature tensor Extract RGB color information and calculate the position of each pixel. brightness value ; S302. Regarding the brightness value Calculating the brightness gradient using the Sobel operator ; S303. From the RGB-D four-dimensional feature tensor The third channel from which depth coordinate information is extracted yields the depth value tensor. ,in Indicates pixel position The depth value of the corresponding point in the camera coordinate system is the Z-axis coordinate value; for the depth tensor Calculating depth gradient using the Sobel operator ; S304. For each pixel position Defined by pixel position A square neighborhood window centered on In the neighborhood window Internal calculation of brightness gradient With depth gradient covariance ; S305. Based on the covariance value and brightness gradient To determine the highlight area, the criteria are: like and Then the pixel position Marked as highlight area; in The covariance threshold, The brightness gradient threshold; Generate specular region mask When pixel position When the criteria for judging highlight areas are met ,otherwise .

6. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 5, characterized in that, In step S3, the steps of generating a spatial attention mask based on the identified highlight regions, reducing the weight of highlight region features in the fused feature map, and enhancing non-highlight region features include: S311. Mask the highlight area. Downsampling to the fused feature map using max pooling operation The resolution is used to obtain the downsampled highlight mask. ; S312. Based on downsampled specular mask Generate reflection suppression weight tensor The calculation formula is: ； in, pixel position The reflection suppression weight value at the location, The enhancement weighting coefficient for non-highlight regions. This represents the weighting factor for the highlight region. The pixel position after downsampling The mask value for the highlight area at that location; S313. The reflection suppression weight tensor Extended to fusion feature maps Same dimensions; expand the weight tensor With the fused feature map Perform element-wise multiplication to generate a weighted feature map. ; S314. Based on the weighted feature map A Region Candidate Network (RPN) is constructed to generate candidate region boxes, and the classification scores and bounding box regression parameters of the candidate region boxes are obtained. S315. In the feature map Each position definition Anchor frames of different sizes and aspect ratios, with dimensions of... pixels, aspect ratio For each anchor box, determine whether it contains obstacles based on the classification score. If the foreground score is greater than the threshold of 0.7, the anchor box is retained. Apply bounding box regression parameters to adjust the position and size of the anchor box to generate accurate candidate region boxes. S316. Apply Non-maximum Suppression (NMS) to all retained candidate region boxes; after NMS, retain the top 2000 candidate region boxes with the highest scores. S317. Each candidate region box is a quadruple. It means that, among them This represents the horizontal coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. This represents the vertical coordinates of the top-left corner of the candidate region bounding box in the feature map coordinate system. The width of the candidate region bounding box. This represents the height of the candidate region bounding box.

7. The multi-level obstacle classification and recognition method for unmanned transport vehicles in metal processing environments according to claim 6, characterized in that, In step S4, the step of inputting the local features corresponding to the candidate region box into the fine-grained recognition network for obstacle classification includes: S401. Regarding the first Candidate region boxes From the weighted feature map Extract local features from the corresponding regions; use bilinear interpolation to uniformly adjust candidate regions of different sizes to a fixed-size 7×7×256 local feature tensor. ; S402. Convert the 7×7×256 local feature tensor Flattened into a one-dimensional feature vector ; Flatten the feature vector The input is fed into a fully connected classification network; the fully connected classification network consists of three fully connected layers, and the output... Unnormalized scores of each category of obstacles ; S404. (Regarding) The unnormalized scores of each category of obstacle are applied using the Softmax function to generate... Confidence vector of obstacle class ,in For the first Confidence level of obstacle class This represents the total number of obstacle categories. S405. Select the category with the highest confidence level as the first... The final identification result of the candidate region; if the confidence level of the final identification result is less than 0.5, then the candidate region will be... Each candidate region is marked as background or unknown obstacle and discarded; S406. For all candidate regions, output a list of obstacle recognition results. Each result includes: coordinates of the candidate region bounding box. Category labels, confidence levels, and complete 5D confidence vector .

Images