A three-dimensional target detection method for indoor mobile robots based on highly distilled guidance and positioning error

Abstract

The application discloses a kind of indoor mobile robot three-dimensional target detection methods based on height distillation guide and positioning error, it belongs to computer vision and intelligent perception field.The present application solves the problem that existing method does not consider explicit spatial structure constraint, high computational complexity, poor stability in complex scene and poor generalization ability in cross-domain scene.Specifically: obtain the RGB image of the region to be detected by the monocular camera deployed on the indoor autonomous mobile robot, obtain the point cloud data by laser radar or depth camera;Based on the height distribution distillation module, the height distribution prediction of the pixel in the RGB image is obtained;The two-dimensional feature map of the RGB image and the height distribution prediction are spliced, and the BEV feature map of the RGB image is obtained based on the splicing result;Grid division is carried out on BEV space, and the point cloud BEV feature map is obtained;Based on the image and point cloud BEV feature map, the fusion BEV feature is obtained, and the target detection head outputs the target detection result based on the fusion BEV feature.The application can be applied to three-dimensional target detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent sensing technology, specifically relating to a method for three-dimensional target detection of indoor mobile robots based on height distillation guidance and positioning error. Background Technology

[0002] Currently, multimodal fusion-based 3D object detection technology has been widely applied in fields such as autonomous driving, robot navigation, intelligent inspection, warehousing and logistics, and augmented reality. For indoor mobile robot scenarios, this technology not only fulfills the basic functions of environmental perception and obstacle recognition, but is also a core technological prerequisite supporting robots in completing autonomous navigation, trajectory planning, dynamic obstacle avoidance, and interactive operation tasks. Among these, existing multimodal fusion-based 3D object detection methods can be summarized into the following main paradigms in terms of technical architecture:

[0003] (1) Voting-based methods, such as VoteNet, ImVoteNet, H3DNet, BRNet, etc.;

[0004] (2) Transformer-based methods, such as 3DETR, Group-Free, Token Fusion, BrT, OneFormer3D, etc.;

[0005] (3) Methods based on sparse convolution, such as GSDN, FCAF3D, TR3D, DSPDet3D, etc.

[0006] However, existing multimodal fusion-based 3D target detection technologies still have the following problems:

[0007] (1) Voting noise based on voting methods is unstable:

[0008] Voting-based methods rely on point-level offset regression and clustering strategies to generate target centers. Since the quality of center prediction is highly dependent on the effectiveness of local feature extraction, and the subsequent clustering process is sensitive to offset errors, center confusion easily occurs in scenes with multiple nearby objects, occlusion, or dense clustering. Furthermore, voting, as a local operation mechanism, lacks global consistency constraints, thus limiting the stability and localization robustness of this method in complex scenes.

[0009] (2) Transformer-based methods lack explicit spatial structure inductive bias:

[0010] Pure Transformer-based 3D object detection methods primarily rely on self-attention mechanisms to directly model inter-point relationships, without explicitly introducing regularized spatial structure priors. Their geometric relationship modeling depends on the network's implicit learning of the data, lacking explicit spatial inductive bias, resulting in model training performance being highly sensitive to data scale and quality. In complex environments, this method is prone to problems such as blurred target boundaries, spatial center drift, and increased size estimation errors, making it difficult to meet the detection requirements of indoor mobile robots for medium- to high-precision localization.

[0011] (3) The computational complexity of sparse convolution-based methods is high:

[0012] While sparse 3D convolution-based methods improve efficiency by avoiding computation in empty voxel regions, their 3D convolution operations and memory overhead remain significant. As network depth increases, the layer-by-layer diffusion effect of active voxels further exacerbates the computational burden. In multimodal fusion scenarios, the precise alignment process in 3D space is inherently complex, making this method unsuitable for dynamic environments with high temperature, high humidity, vibration, and multiple noise interferences. In complex environments, this method is prone to issues such as point cloud sparsity, spatial distortion, and feature mismatch, thus failing to meet the requirements of real-time robot perception and stable operation in complex scenarios.

[0013] (4) Weak cross-domain generalization ability:

[0014] The difficulty of 3D annotation in real-world indoor and outdoor scenes leads to a scarcity of training data and slow model convergence. Existing methods are typically limited to training and validation within a single dataset, resulting in a significant drop in object detection accuracy when switching between datasets or between real and synthetic scenes. In particular, when migrating models from training environments to real-world application scenarios, or when facing changes in robot platforms, sensor configurations, and environments, feature distribution shifts are easily triggered, leading to severe degradation in detection performance and revealing a significant deficiency in cross-domain generalization capabilities.

[0015] (5) Lack of positioning error modeling mechanism for robot performance testing:

[0016] In the process of performance testing and standard setting for indoor mobile robots, target localization accuracy is one of the important indicators for measuring the robot's environmental perception and dynamic response capabilities. The spatial localization results of the robot in relation to obstacles, target objects, and environmental structures directly affect core functions such as path planning, motion control, obstacle avoidance decisions, and robotic arm grasping. Therefore, the spatial localization error in the 3D target detection results can, to a certain extent, reflect the robot's overall performance and its adaptability to complex environments. However, current performance testing methods for robot 3D target detection lack systematic modeling and analysis, making it difficult to meet the needs of overall robot performance testing and multi-level evaluation.

[0017] (6) The positioning error assessment method is singular:

[0018] Existing 3D target detection methods generally rely on a single or fixed IoU threshold (e.g., using only mAP@0.25 to evaluate model performance) when assessing localization errors. This evaluation design struggles to comprehensively characterize the model's localization accuracy under varying degrees of stringency, and is ineffective in adapting to real-world application scenarios that range greatly from basic environmental perception to high-precision manipulation. Such evaluation methods only reflect the model's approximate detection capability, failing to further describe key localization indicators such as target boundary fitting accuracy, spatial center offset error, and scale estimation error. However, in robot overall performance evaluation, the requirements for localization accuracy exhibit significant task dependence and scale differences across tasks. Basic navigation tasks have a high tolerance for spatial errors, while operations such as fine grasping, human-robot collaboration, and obstacle avoidance in confined spaces impose stringent and multi-dimensional constraints on localization accuracy. Therefore, an evaluation system using a single fixed IoU threshold not only fails to comprehensively characterize the robot's spatial perception capabilities in diverse application scenarios but also cannot support the construction of a unified, fine-grained, and highly scalable robot perception performance testing standard.

[0019] Therefore, there is an urgent need to construct a 3D target detection method that possesses explicit / semi-explicit spatial structure modeling capabilities, low computational complexity, high robustness in complex scenarios, and strong generalization ability in cross-domain scenarios. Furthermore, it is necessary to simultaneously achieve deep coupling between the localization error evaluation mechanism and the multi-level IoU threshold system, thereby significantly improving the target localization accuracy and detection stability of mobile robots in unstructured environments. Ultimately, this will provide interpretable, traceable, and scalable key technical support for robot overall performance testing, intelligence level evaluation, and the formulation of related testing standards. Summary of the Invention

[0020] This invention aims to address the problems of existing 3D target detection methods, such as lack of explicit spatial structure constraints, excessive computational complexity, insufficient stability in complex scenes, and poor generalization ability across different scenarios. Therefore, it proposes a 3D target detection method for indoor mobile robots based on height distillation guidance and positioning error.

[0021] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a three-dimensional target detection method for indoor mobile robots based on height distillation guidance and positioning error, the method specifically including the following steps:

[0022] Step 1: Acquire RGB images of the area to be detected using a monocular camera deployed on the indoor autonomous mobile robot, and acquire point cloud data using a lidar or depth camera deployed on the indoor autonomous mobile robot.

[0023] Step 2: Extract two-dimensional feature maps from the RGB image of the region to be detected using the backbone network. Then the two-dimensional feature map After passing through the height prediction head, the height distribution prediction result of pixels in the RGB image of the area to be detected is obtained in the vertical direction.

[0024] Step 3: The two-dimensional feature map of the RGB image of the region to be detected and the height distribution prediction result are stitched together. Then, the stitched result is projected from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map of the RGB image of the region to be detected.

[0025] Step 4: After dividing the BEV space into grids, use the established mapping function to map points with depths greater than or equal to the threshold to the BEV space. Based on the mapping results, filter the points in each grid obtained, and then obtain the point cloud BEV feature map based on the filtered point cloud.

[0026] Step 5: Fuse the BEV feature map corresponding to the RGB image and the BEV feature map of the point cloud to obtain the fused BEV feature, and generate a proposal domain and random candidate boxes based on the RGB image of the region to be detected.

[0027] The fused BEV features are then used as input to the object detection head, which outputs the object detection results based on the generated proposal domain and random candidate boxes.

[0028] Furthermore, in step one, the lidar directly acquires point cloud data, and the depth map acquired by the depth camera is converted into point cloud data in the following way:

[0029] Based on the depth of each pixel in the depth map Define the depth mask image:

[0030]

[0031] in:

[0032] Representing a depth map pixels in The depth value;

[0033] Represents the pixels in the depth mask map The value;

[0034] Obtain the valid point set based on the pixel values ​​in the depth mask image. :

[0035]

[0036] For a valid pixel in the set of valid points , will effective pixels The depth value is denoted as Then define the intermediate vector. for:

[0037]

[0038] in:

[0039] express transpose;

[0040] Using the inverse of the camera intrinsic matrix, pixel coordinates are converted into 3D points in the camera coordinate system:

[0041]

[0042] in:

[0043] Indicates valid pixels The corresponding 3D point in the camera coordinate system Representing the intrinsic parameter matrix The reverse.

[0044] Furthermore, the specific process of step two is as follows:

[0045]

[0046] in:

[0047] Represents pixels in an RGB image Features;

[0048] Indicates a high-prediction head;

[0049] Indicates the Softmax activation function layer;

[0050] Represents pixels in an RGB image The predicted height distribution in the vertical direction.

[0051] Furthermore, the specific process of step three is as follows:

[0052] Step 31: Define the BEV space range: ;

[0053] in:

[0054] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0055] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0056] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0057] Step 3.2: Obtain the depth of the corresponding point in the point cloud data for each pixel in the RGB image, and remove points in the point cloud data whose depth is less than the threshold. The pixels corresponding to the points in the RGB image and the pixels whose projection range exceeds the BEV space range;

[0058] The features of the remaining pixels in the RGB image are concatenated with the height distribution prediction results of the remaining pixels in the RGB image to obtain the concatenated result.

[0059] Step 3: Project the stitching result from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map corresponding to the RGB image. .

[0060] Furthermore, the specific process of step four is as follows:

[0061] Step 41: Set the grid size for each direction of the BEV space as follows. , and The BEV space is divided into grids according to the set grid size, where, shaft and The axial direction is uniformly divided. The axial direction is divided non-uniformly;

[0062] Step 42: Establish a mapping function and use it to map points to the BEV space grid.

[0063] in:

[0064] Representing a three-dimensional point Mapping function values ​​to the BEV space grid;

[0065] Indicates an indicator function; Indicates taking the absolute value; Indicates rounding down;

[0066] Step 43: Based on the mapping results in Step 42, filter the points within the grid to obtain the points retained in each grid.

[0067] Step 4: Aggregate the points retained within each grid:

[0068]

[0069] in:

[0070] Represents a grid The actual number of points retained internally. Represents a grid The first one that was preserved inside The coordinates of the points Represents a grid The aggregation result;

[0071] Steps four and five: Use an intermediate encoder to convert the aggregation results of each grid into a dense BEV feature map. ;

[0072] Step 46: Use the SECOND network to process the dense BEV feature map Processing:

[0073]

[0074] in:

[0075] This represents the output of the SECOND network. Indicates a SECOND network;

[0076] Step 47: Using the SECONDFPN network to... Perform multi-scale feature fusion:

[0077]

[0078] in:

[0079] This represents the result of multi-scale feature fusion. Indicates SECONDFPN network;

[0080] Step 48, Transform the point cloud into the BEV embedding space, which is the same as the image features, to obtain the BEV feature map. :

[0081]

[0082] in:

[0083] This indicates a convolutional layer.

[0084] Furthermore, the specific process of step four is as follows:

[0085] Each grid The axial dimension is constant. Each grid The axial dimension is constant. For a uniformly divided dimension, the grid boundaries are respectively ;

[0086] in:

[0087] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers in the axial direction;

[0088] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers in the axial direction;

[0089] For dimensions with non-uniform partitioning, the number of mesh layers along the depth axis is denoted as... The LID non-uniform grid boundary sequence is Let the lower boundary of the 0th layer mesh in the depth axis direction be... Let the depth axis direction be the first The upper boundary of the layer mesh is Then the depth axis direction of the first The lower boundary of the layered mesh is generated by a quadratic function:

[0090]

[0091] in:

[0092] , Indicates the depth axis direction. The lower boundary of the layer mesh;

[0093] Depth axis direction Width of layer grid for:

[0094] .

[0095] Furthermore, the fusion of the BEV feature map corresponding to the RGB image and the point cloud BEV feature map specifically involves:

[0096] BEV feature map of image Point cloud BEV feature map The concatenation is performed along the channel dimension, and then the concatenation result is passed through a convolutional layer to obtain the fused BEV features.

[0097] Furthermore, the backbone network, height prediction head, intermediate encoder, SECOND network, and SECONDFPN network need to be trained together with a height distribution distillation module, which constructs height distribution labels for RGB images based on point cloud data.

[0098] Furthermore, the operation process of the highly distributed distillation module is as follows:

[0099] Vertical range of point cloud data Evenly divided into A height range for pixels in an RGB image. :

[0100] like For effective pixels, then based on the camera intrinsic parameter matrix Get pixels In point cloud data The corresponding point in the image, based on pixels In point cloud data The height of the corresponding point in the middle Construct highly distributed labels:

[0101]

[0102] in:

[0103] Represents pixels In the The height distribution values ​​of each height interval Indicates the first The height value at the center of each height interval Indicates the first The height value at the center of each height interval The bandwidth parameter of the Gaussian kernel;

[0104] like For invalid pixels, a uniform distribution is constructed as the height distribution label:

[0105]

[0106] Will As pixels High-distribution labels.

[0107] Furthermore, when training the backbone network, height prediction head, intermediate encoder, SECOND network, and SECONDFPN network in conjunction with the height distribution distillation module, the loss function used is... for:

[0108]

[0109] in:

[0110] Represents classification loss. The weights representing the classification loss;

[0111] Indicates location loss. The weights representing the localization loss;

[0112] Indicates dimensional loss. The weight representing the size loss;

[0113] Indicates attitude loss, The weights representing the pose loss;

[0114] Indicates feature alignment loss, The weights represent the feature alignment loss.

[0115] This indicates the distillation loss due to the high distribution of KL divergence. The weight representing the distillation loss due to the KL divergence height distribution;

[0116]

[0117] in:

[0118] express The first in One element, Represents an arbitrarily small constant;

[0119]

[0120] in:

[0121] To balance the weights for each category, To focus on weight, As an intermediate variable;

[0122]

[0123] in:

[0124] This indicates that the current area is the real target. This indicates the probability that the current area is the target.

[0125]

[0126] in:

[0127] This represents the three-dimensional coordinates of the predicted target center location. The actual three-dimensional coordinates of the target center position. For uncertain parameters, The base of the natural logarithm. Represents the L1 norm;

[0128]

[0129] in:

[0130] Indicates the predicted target size. For the actual target size;

[0131]

[0132] in:

[0133] This represents the relative rotation angle between the predicted direction and the true direction.

[0134]

[0135] in:

[0136] and All are intermediate variables;

[0137]

[0138] in:

[0139] For the effective mask of the point cloud, This represents the point cloud BEV feature map obtained after the last network parameter update. This indicates element-wise multiplication;

[0140]

[0141] in:

[0142] This represents the image feature map obtained after the last network parameter update. Indicates the valid image mask;

[0143] Using the calculated loss function Backpropagation is performed to adjust the parameters of the backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module until the target localization error on the test set meets the requirements. Training stops when the target localization error on the test set meets the requirements, and the trained backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module are obtained.

[0144] Furthermore, based on the detection requirements of different scenarios, different levels of IoU detection thresholds are set for the target localization error of the test set;

[0145] The IoU detection thresholds are set to 0.15, 0.25, and 0.50. Then, based on the set IoU detection thresholds, the average accuracy of all detection boxes under different confidence levels is calculated until the average accuracy is greater than the accuracy threshold to meet the target localization error requirements.

[0146] The beneficial effects of this invention are:

[0147] The method of this invention balances accuracy, efficiency and generalization ability in the 3D target detection task of indoor autonomous mobile robots, and is particularly suitable for high-requirement application scenarios such as robot navigation, obstacle avoidance and environmental understanding in complex indoor environments.

[0148] (1) Significantly improves the geometric consistency of modal fusion

[0149] This invention introduces a cross-modal height distillation module, using the precise height distribution provided by the point cloud modality as a teacher signal to guide the image modality to learn the height distribution characteristics of each pixel location, thereby establishing explicit spatial structure constraints. This effectively solves the spatial ambiguity problem caused by the lack of vertical information when the image modality is projected into the BEV space, thus improving the overall consistency of multimodal fusion.

[0150] (2) Enhance the robustness of the model in complex indoor scenes.

[0151] This invention employs a non-uniform BEV mesh generation strategy, adaptively adjusting the mesh resolution in the depth direction based on the target distance. This preserves the spatial details of small targets in the foreground while enhancing the perception of large targets in the distance. Compared to traditional uniform mesh generation methods, this significantly improves the model's detection performance in indoor scenes with dense, occluded, and multi-scale targets without increasing computational burden.

[0152] (3) Effectively alleviates information loss caused by modality missing

[0153] Through a bidirectional guided alignment module, this invention achieves complementary enhancement of image and point cloud features in the BEV space. In the effective region of the point cloud, image features are aligned to point cloud features, improving geometric accuracy; in the missing point cloud region, point cloud features are aligned to image features, introducing semantic information to fill spatial gaps. This mechanism significantly improves the model's robustness and detection completeness in scenarios with sensor failure, occlusion, or sparse point clouds.

[0154] (4) Significantly reduce computational overhead and improve training and inference efficiency.

[0155] This invention proposes a projection point filtering mechanism that removes invalid projection points by setting a depth confidence threshold, significantly reducing computational redundancy during the projection of image features into the BEV space. Experiments show that when the threshold is set to 0.001, approximately 83% of invalid projection points can be filtered out, greatly reducing memory usage and computation time, and enabling real-time processing of high-resolution BEV features.

[0156] (5) Enhance generalization ability across scenarios and datasets.

[0157] This invention enhances the model's adaptability to unknown scenarios through a dual mechanism of BEV alignment fusion guided by highly distributed distillation. In cross-dataset tests (such as migration from synthetic to real-world scenarios), the detection accuracy decrease is significantly lower than that of existing methods, demonstrating excellent generalization performance and suitability for diverse real-world application environments.

[0158] (6) A unified BEV representation facilitates subsequent task integration.

[0159] The fused BEV features output by this invention are characterized by regularization, structure, and a combination of semantics and geometry, making it easy to seamlessly integrate with downstream tasks (such as trajectory prediction, path planning, target tracking, etc.) and possessing good scalability and practicality. Attached Figure Description

[0160] Figure 1 This is a flowchart of a three-dimensional target detection method for an indoor mobile robot based on height distillation guidance and positioning error, according to the present invention.

[0161] Figure 2 A flowchart illustrating the workflow of the cross-modal height-distributed distillation module;

[0162] Figure 3 RGB images provided for the public dataset SUNRGBD;

[0163] Figure 4 Depth images provided for the public dataset SUNRGBD;

[0164] Figure 5 To adopt Figure 4 The point cloud extracted from the depth information;

[0165] Figure 6 The method of the present invention in Figure 3 and Figure 5 The results of the test are shown in the image above. Detailed Implementation

[0166] The BEV representation of this invention projects 3D point clouds and image features onto a unified 2D planar mesh structure, establishing a stable spatial coordinate mapping relationship under regular spatial partitioning. This allows image semantic information and point cloud geometric information to be aligned and fused within the same reference frame. Simultaneously, this representation primarily compresses the vertical direction while accurately preserving spatial distribution information in the horizontal and depth directions, achieving a structured representation of the spatial structure. This unified spatial modeling approach not only reduces the complexity of cross-modal alignment but also improves overall modeling stability. Furthermore, the depth information provided by the point cloud weakens the occlusion relationship in the image modality. This representation helps maintain the integrity of the overall target structure, enabling the network to more stably identify the spatial occupancy range of the target and reducing the risk of feature loss or misjudgment caused by occlusion. The method of this invention will be described in detail below with reference to the accompanying drawings:

[0167] Specific implementation method one: Combining Figure 1 This embodiment describes a method for 3D target detection in an indoor mobile robot based on height distillation guidance and positioning error. The method specifically includes a training phase and a detection phase.

[0168] During the training phase:

[0169] Step S1: Obtain several RGB images and the point cloud data corresponding to each RGB image;

[0170] Specifically: An RGB image can be represented as:

[0171]

[0172] in:

[0173] Represents an RGB image. Represents the pixels in an RGB image The value;

[0174] Represents pixels exist The value of the channel, Represents pixels exist The value of the channel, Represents pixels exist The value of the channel, Represent real numbers;

[0175] Point cloud data It can be point cloud data collected directly using LiDAR, or it can be point cloud data obtained based on depth maps;

[0176] (1) Point cloud data acquired by lidar , This indicates the first point in the point cloud data acquired by the lidar. One point, This indicates the first point in the point cloud data acquired by the lidar. The coordinates of a point in the point cloud coordinate system This indicates the first point in the point cloud data acquired by the lidar. The reflection intensity at each point;

[0177] (2) For depth maps Based on the depth of each pixel in the depth map Define the effective depth mask:

[0178]

[0179] in:

[0180] Representing a depth map pixels in The depth value, Represents the pixels in the depth mask map The value;

[0181] Obtain the valid point set based on the pixel values ​​in the depth mask image. :

[0182]

[0183] For a valid pixel in the set of valid points , will effective pixels The depth value is denoted as Then define the intermediate vector. for:

[0184]

[0185] in:

[0186] express transpose;

[0187] Using the inverse of the camera intrinsic matrix, pixel coordinates are converted into 3D points in the camera coordinate system:

[0188]

[0189] in:

[0190] Indicates valid pixels The corresponding 3D point in the camera coordinate system Representing the intrinsic parameter matrix The reverse;

[0191] Camera intrinsic parameter matrix for:

[0192]

[0193] in:

[0194] , , and Represents the camera intrinsic parameter matrix The parameters in Horizontal focal length Vertical focal length Main point coordinate, Main point coordinate;

[0195] Each valid pixel is processed separately to obtain point cloud data. :

[0196]

[0197] Step S2: Pass the RGB image through the backbone network (ConvNext) to extract the two-dimensional feature map. , Then the two-dimensional feature map The image is then processed by a lightweight height prediction head (the Height branch of the Center Head network used in this invention) to obtain the height distribution prediction results of pixels in the RGB image in the vertical direction (student prediction).

[0198]

[0199] in:

[0200] Represents pixels in an RGB image Features Indicates a highly predictive head. Indicates the Softmax activation function layer;

[0201] Represents pixels in an RGB image The height distribution prediction results in the vertical direction can be regarded as the image modality's understanding of the vertical spatial structure, that is, the height distribution information inferred from 2D image features.

[0202] The key to indoor 3D object detection is fusing geometric and semantic features from different modalities and performing accurate modeling. Point cloud modalities naturally possess precise coordinate information, accurately describing the vertical distribution of objects; while image modalities can only implicitly infer height information from 2D pixels, resulting in significant uncertainty. Therefore, if the image modalities can learn the height distribution characteristics of the point cloud modalities at each pixel location before projecting image features into the BEV space, the image branch can generate more geometrically consistent BEV features in subsequent processing. Therefore:

[0203] Construct height distribution labels for RGB images based on point cloud data (teacher soft labels):

[0204] Vertical range of point cloud data Evenly divided into Each height range (bins) is used to define non-zero pixels in the RGB image as valid pixels and other pixels as invalid pixels; for pixels in the RGB image... :

[0205] like For effective pixels, then based on the camera intrinsic parameter matrix Get pixels In point cloud data The corresponding point in the image, based on pixels In point cloud data The height of the corresponding point in the middle Construct highly distributed labels:

[0206]

[0207] in:

[0208] Represents pixels In the The height distribution values ​​of each height interval Indicates the first The height value at the center of each height interval Indicates the first The height value at the center of each height interval;

[0209] The bandwidth parameter of the Gaussian kernel is used to control the smoothness of the distribution. A larger value indicates a smoother distribution, and a smoother distribution has a higher tolerance for high uncertainty. A smaller value indicates that the distribution is more concentrated near the actual height. In practical applications, The specific value can be set according to the sensor noise level or task requirements (e.g., Indicates the truth value of height in (The likelihood of fluctuations within the interval is relatively high). Therefore, this soft label constructed using a Gaussian distribution retains the uncertainty information of the height value by assigning non-zero probabilities to the intervals adjacent to the true height. Compared with one-hot labels that only have one correct category, it can provide richer supervision signals for model training.

[0210] like For invalid pixels, a uniform distribution is constructed as the height distribution label:

[0211]

[0212] Will As pixels High-distribution labels;

[0213] like Figure 2 As shown, in order to guide the image height distribution using point clouds, the KL divergence height distribution distillation loss is calculated based on the height distribution results from the height prediction head and the height distribution labels obtained from the point cloud data.

[0214]

[0215] in:

[0216] This indicates the distillation loss due to the high distribution of KL divergence. express The first in One element, To represent an arbitrarily small constant, to prevent division by zero;

[0217] After height distribution distillation, the height distribution information guides the image projection points to fall in the correct vertical range, avoiding projection misalignment caused by height ambiguity. Each pixel position in the image branch has accurate height distribution perception capability.

[0218] Given image features and depth features ,pass and The multiplication operation can obtain the projected features. Number of projection points Will follow The computational burden of this feature projection step is enormous, leading to a sharp increase in dimensionality and limiting the BEV feature resolution, which in turn hinders the unified realization of indoor and outdoor 3D object detection.

[0219] The vast majority of projected points are of no practical use due to their extremely small values. This is essentially due to the feature map obtained from depth prediction. The smaller values ​​at the corresponding positions indicate that the model predicts the absence of targets in these specific BEV grids. Therefore, the computational cost of projecting features onto these irrelevant grids can be completely saved. Based on the above findings, this method removes invalid projection points according to a preset threshold τ: specifically through filtering... The corresponding projection points with a confidence level below the threshold τ at medium depth are implemented.

[0220] Step S3: Define the BEV space range: ;

[0221] in:

[0222] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0223] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0224] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0225] Obtain the depth of the corresponding point in the point cloud data for each pixel in the RGB image, and remove points in the point cloud data whose depth is less than a threshold. The pixels corresponding to the points in the RGB image and the pixels whose projection range exceeds the BEV space range;

[0226] The features of the remaining pixels in the RGB image are concatenated with the predicted height distribution of the remaining pixels in the RGB image to obtain the concatenated result. ;

[0227]

[0228] Then, using the camera intrinsic parameter matrix, the stitched result is... Projecting from the camera coordinate system to the BEV coordinate system yields the BEV feature map corresponding to the RGB image. ;

[0229] And based on BEV feature maps Calculate the effective mask for the image region:

[0230]

[0231] in:

[0232] Indicates the valid image mask. BEV feature map The number of channels, Represents pixels in an image The mask value, BEV feature map medium pixel In the The value of each channel, Indicates an indicator function;

[0233] Step S4: In order to obtain the BEV feature map of the point cloud, perform the following steps S41 to S44.

[0234] Step S41: Set the grid size for each direction of the BEV space as follows: , and This invention The value is 0.1. The value is 0.1. The value is 0.2; the BEV space is divided into grids according to the set grid size, where, shaft and The axial direction is uniformly divided. Non-uniform division along the axial direction Corresponding to the width direction, Corresponding to the height direction, Corresponding depth direction;

[0235] Specifically:

[0236] Each grid The axial dimension is constant. Each grid The axial dimension is constant. For a uniformly divided dimension, the grid boundaries are respectively ;

[0237] in:

[0238] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers in the axial direction;

[0239] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers in the axial direction;

[0240] Indoor scenes are far more complex than outdoor scenes. Therefore, traditional uniform mesh partitioning schemes for BEV (Balanced Element Detection) cannot meet the multi-scale requirements of indoor detection. The network needs to maintain high resolution for near targets while ensuring detection efficiency for larger, distant targets. This invention achieves this by using a smaller mesh for near camera areas and a larger mesh for distant areas. This allows the network to effectively perceive larger, distant targets indoors while maintaining a small mesh for near objects, without increasing the total mesh count and thus avoiding additional computational burden. Specifically:

[0241] For dimensions with non-uniform partitioning, the number of mesh layers along the depth axis is denoted as... ( (equal to the number of grid cells obtained according to the non-uniform partitioning strategy), the LID non-uniform grid boundary sequence is: Let the lower boundary of the 0th layer mesh in the depth axis direction be... Let the depth axis direction be the first The upper boundary of the layer mesh is Then the depth axis direction of the first The lower boundary of the layered mesh is generated by a quadratic function:

[0242]

[0243] in:

[0244] , Indicates the depth axis direction. The lower boundary of the layer mesh;

[0245] The width of the grid along the axis varies with The value increases linearly with the increase of the depth axis direction. Width of layer grid for:

[0246]

[0247] Moreover, in this invention, Axial direction and The number of grid cells in the axial direction satisfies:

[0248]

[0249] in:

[0250] Axial direction and Discard any remaining areas in the axial direction that are less than one grid width;

[0251] Step S42: Establish a mapping function and use it to map points to the BEV space grid.

[0252]

[0253] in:

[0254] Representing a three-dimensional point Mapping function values ​​to the BEV space grid; This indicates an indicator function. The value of the indicator function is 1 when the condition within the square brackets is met; otherwise, the value of the exponential function is 0. This indicates taking the absolute value. As an intermediate variable, ;

[0255] Step S43: Based on the mapping results in step S42, filter the points within the grid to obtain the points retained in each grid. It should be noted that a maximum of points can be retained in each grid. When the number of points in a grid is insufficient When this happens, all points within the grid are retained;

[0256] Aggregate the points retained within each grid:

[0257]

[0258] in:

[0259] Represents a grid The actual number of points retained internally. Represents a grid The first one that was preserved inside The coordinates of the points Represents a grid The aggregation result;

[0260] Mesh via mapping function The aggregation results are mapped to the BEV space, and then the intermediate encoder (SparseEncoder) is used to convert the mapping results of each grid into dense BEV feature maps. , That is, the height is compressed to the channel dimension through an intermediate encoder;

[0261] in:

[0262] Indicates the number of channels. and Indicates the BEV grid size after downsampling;

[0263]

[0264] in, Indicates the downsampling factor;

[0265] And based on dense BEV feature maps Effective region mask for generating point clouds:

[0266]

[0267] in:

[0268] Indicates the valid mask of the point cloud. Representing dense BEV feature maps The number of channels, Points in the valid mask of the point cloud The mask value, Representing dense BEV feature maps Points in In the The value of each channel, Indicates an indicator function;

[0269] Step S44: Use the SECOND network to process the dense BEV feature map. Processing is performed to further extract features:

[0270]

[0271] in:

[0272] This represents the output of the SECOND network. Indicates a SECOND network;

[0273] Using SECONDFPN network Perform multi-scale feature fusion:

[0274]

[0275] in:

[0276] This represents the result of multi-scale feature fusion. Indicates SECONDFPN network;

[0277] Will Transform the point cloud into the BEV embedding space, which is the same as the image features, to obtain the BEV feature map. :

[0278]

[0279] in:

[0280] This indicates a convolutional layer.

[0281] Step S5: Obtain the BEV feature map corresponding to the RGB image. Point cloud BEV feature map To merge, soon and Perform concatenation along the channel dimension, and then pass the concatenation result through a convolutional layer to obtain the fused BEV features;

[0282] Step S6: Generate a proposal domain and random candidate boxes based on the RGB image (i.e., generate 3D center point proposals using the CenterHead of the RGB image). Then, use the fused BEV features and the generated proposal domain and random candidate boxes as input to the DETR3D Head. Output the target detection results through the target detection head, i.e., output the target center position, target size, target orientation (including the target's yaw angle, pitch angle, and roll angle), and class probability.

[0283] It should be noted that the loss function used throughout the entire network training process of this invention... as follows:

[0284]

[0285] (1) Classification loss for:

[0286]

[0287] in:

[0288] To balance the weights for each category, To focus on weight, As an intermediate variable;

[0289]

[0290] in:

[0291] This indicates that the current area is the real target. This indicates the probability that the current area is the target.

[0292] (2) Positioning loss for:

[0293]

[0294] in:

[0295] This represents the three-dimensional coordinates of the predicted target center location. The actual three-dimensional coordinates of the target center position. For uncertain parameters, The base of the natural logarithm. Represents the L1 norm;

[0296] Size loss for:

[0297]

[0298] in:

[0299] Indicates the predicted target size. For the actual target size;

[0300] Attitude loss for:

[0301]

[0302] in:

[0303] This represents the rotation matrix of the predicted target direction relative to the Cartesian coordinate system. This represents the rotation matrix of the true target orientation relative to the Cartesian coordinate system. The superscript T indicates the transpose of the matrix. This represents the computation of the trace of a matrix. This represents the relative rotation angle between the predicted direction and the true direction.

[0304] Feature alignment loss for:

[0305]

[0306] By guiding the image through the point cloud (within the effective area of ​​the point cloud), image features are aligned to precise point cloud features:

[0307]

[0308] in:

[0309] For the effective mask of the point cloud, This represents the point cloud BEV feature map obtained after the last network parameter update. This indicates element-wise multiplication;

[0310] By guiding the point cloud through the image (in regions where the image is valid but the point cloud is invalid), the point cloud features learn the semantic information of the image:

[0311]

[0312] in:

[0313] This represents the image feature map obtained after the last network parameter update;

[0314] Using the calculated loss function Backpropagation is performed to adjust the parameters of the backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module until the target localization error on the test set meets the requirements. Training stops when the target localization error on the test set meets the requirements, and the trained backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module are obtained.

[0315] This module achieves feature alignment and complementary enhancement in the BEV space through bidirectional information repair, providing higher quality fused features for subsequent detection heads.

[0316] The network parameters are adjusted in reverse based on the calculated loss function value until the total loss function converges or the maximum number of training iterations is reached. Training is then stopped to obtain the trained network parameters before starting the actual object detection process.

[0317] In the actual target detection process:

[0318] Step 1: Acquire RGB images of the area to be detected using a monocular camera deployed on the indoor autonomous mobile robot, and acquire point cloud data using a lidar or depth camera deployed on the indoor autonomous mobile robot.

[0319] Step 2: Extract two-dimensional feature maps from the RGB image of the region to be detected using the backbone network. Then, the two-dimensional feature map is passed through the height prediction head to obtain the height distribution prediction result of the pixels in the RGB image of the region to be detected in the vertical direction.

[0320] Specifically:

[0321]

[0322] in:

[0323] Represents pixels in an RGB image Features Indicates a highly predictive head. This represents the Softmax activation function layer. Represents pixels in an RGB image Predicted height distribution in the vertical direction;

[0324] Step 3: The two-dimensional feature map of the RGB image of the region to be detected and the height distribution prediction result are stitched together. Then, the stitched result is projected from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map of the RGB image of the region to be detected.

[0325] Specifically:

[0326] Step 31: Define the BEV space range: ;

[0327] in:

[0328] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0329] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0330] Indicates BEV space Minimum coordinate value in the axial direction, Indicates BEV space Maximum coordinate value in the axial direction;

[0331] Step 3.2: Obtain the depth of the corresponding point in the point cloud data for each pixel in the RGB image, and remove points in the point cloud data whose depth is less than the threshold. The pixels corresponding to the points in the RGB image and the pixels whose projection range exceeds the BEV space range;

[0332] The features of the remaining pixels in the RGB image are concatenated with the height distribution prediction results of the remaining pixels in the RGB image to obtain the concatenated result.

[0333] Step 3: Project the stitching result from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map corresponding to the RGB image. ;

[0334] Step 4: After dividing the BEV space into grids, use the established mapping function to map points with depths greater than or equal to the threshold to the BEV space. Based on the mapping results, filter the points in each grid obtained, and then obtain the point cloud BEV feature map based on the filtered point cloud.

[0335] Specifically:

[0336] Step 41: Set the grid size for each direction of the BEV space as follows. , and The BEV space is divided into grids according to the set grid size, where, shaft and The axial direction is uniformly divided. The axial direction is divided non-uniformly;

[0337] The specific process of step 41 is as follows:

[0338] Each grid The axial dimension is constant. Each grid The axial dimension is constant. For a uniformly divided dimension, the grid boundaries are respectively ;

[0339] in:

[0340] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers along the axial direction;

[0341] , express Axial direction The lower boundary of the layer mesh, express The number of grid layers in the axial direction;

[0342] For dimensions with non-uniform partitioning, the number of mesh layers along the depth axis is denoted as... The LID non-uniform grid boundary sequence is Let the lower boundary of the 0th layer mesh in the depth axis direction be... Let the depth axis direction be the first The upper boundary of the layer mesh is Then the depth axis direction of the first The lower boundary of the layered mesh is generated by a quadratic function:

[0343]

[0344] in:

[0345] , Indicates the depth axis direction. The lower boundary of the layer mesh;

[0346] Depth axis direction Width of layer grid for:

[0347]

[0348] Step 42: Establish a mapping function and use it to map points to the BEV space grid.

[0349] in:

[0350] Representing a three-dimensional point Mapping function values ​​to the BEV space grid; Indicates an indicator function; Indicates taking the absolute value; Indicates rounding down;

[0351] Step 43: Based on the mapping results in Step 42, filter the points within the grid to obtain the points retained in each grid.

[0352] Step 4: Aggregate the points retained within each grid:

[0353]

[0354] in:

[0355] Represents a grid The actual number of points retained internally. Represents a grid The first one that was preserved inside The coordinates of the points Represents a grid The aggregation result;

[0356] Steps four and five: Use an intermediate encoder to convert the aggregation results of each grid into a dense BEV feature map. ;

[0357] Step 46: Use the SECOND network to process the dense BEV feature map Processing:

[0358]

[0359] in:

[0360] This represents the output of the SECOND network. Indicates a SECOND network;

[0361] Step 47: Using the SECONDFPN network to... Perform multi-scale feature fusion:

[0362]

[0363] in:

[0364] This represents the result of multi-scale feature fusion. Indicates SECONDFPN network;

[0365] Step 48, Transform the point cloud into the BEV embedding space, which is the same as the image features, to obtain the BEV feature map. :

[0366]

[0367] in:

[0368] Indicates a convolutional layer;

[0369] Step 5: Fuse the BEV feature map corresponding to the RGB image and the BEV feature map of the point cloud to obtain the fused BEV feature, and generate a proposal domain and random candidate boxes based on the RGB image of the region to be detected.

[0370] The specific fusion method is as follows: The BEV feature map of the image is... Point cloud BEV feature map Perform concatenation along the channel dimension, and then pass the concatenation result through a convolutional layer to obtain the fused BEV features;

[0371] The fused BEV features are then used as input to the object detection head, which outputs the object detection results based on the generated proposal domain and random candidate boxes.

[0372] The method of this invention can adapt to the detection accuracy requirements in different scenarios, for example:

[0373] When the IoU detection threshold between the detection box and the ground truth box is set to 0.15, the average accuracy mAP@0.15 of the method of the present invention at different confidence levels on the test set is statistically analyzed.

[0374] When the IoU detection threshold between the detection box and the ground truth box is set to 0.25, the average accuracy mAP@0.25 of the method of the present invention at different confidence levels on the test set is statistically analyzed.

[0375] When the IoU detection threshold between the detection box and the ground truth box is set to 0.50, the average accuracy mAP@0.50 of the method of the present invention at different confidence levels on the test set is statistically analyzed.

[0376] These three levels of indicators correspond to the basic perception level, task perception level, and precision operation level in 3D target detection performance, respectively. The performance of the method of this invention is verified by the results on the test set.

[0377] In summary, this invention proposes a highly distilled guided 3D target detection method for indoor mobile robots. It is applicable to indoor 3D target detection scenarios such as navigation, obstacle avoidance, and environmental understanding for autonomous mobile robots. Using RGB images and point clouds as input, high-precision 3D target detection is achieved through multimodal BEV feature fusion. The specific implementation steps are as follows:

[0378] (1) Input data preprocessing: Normalize and augment the input RGB image; convert the depth map into a point cloud in the camera coordinate system by combining the camera intrinsic parameters; remove noise points and invalid points directly from the laser point cloud to obtain valid point cloud data.

[0379] (2) Non-uniform BEV mesh generation: To meet the multi-scale detection requirements of indoor scenes, a uniform mesh is used in the horizontal and vertical directions of the BEV space, and a non-uniform mesh is generated in the depth direction according to a quadratic function to achieve a mesh distribution with high resolution at near distances and low resolution at far distances, while ensuring that the number of meshes is the same as that of the uniform mesh generation scheme so as not to increase the computational burden.

[0380] (3) Image modal BEV feature construction: First, extract two-dimensional feature maps through the image backbone network and predict the depth distribution at the same time; introduce a cross-modal height distribution distillation module, construct pixel-level height distribution soft labels with point cloud as teacher signal, guide the image branch to learn height distribution features through a lightweight height prediction head, and stitch the features with the original image features to obtain enhanced features; set a threshold based on the depth distribution to filter invalid projection points, project the enhanced features and generate image BEV feature maps.

[0381] (4) Point cloud modal BEV feature construction: The point cloud is gridded under a non-uniform grid, and the point cloud features in each grid are aggregated; the discrete point features are converted into dense BEV features through sparse coding and cylinder scattering, and then the point cloud features are extracted through the backbone network and multi-scale feature fusion network. Finally, feature mapping is performed to obtain the point cloud BEV feature map.

[0382] (5) Bidirectional guided feature fusion: Construct a valid point cloud mask and a valid image mask but invalid point cloud mask. A bidirectional guided feature fusion module is used to achieve cross-modal BEV feature complementarity enhancement. In the valid point cloud region, the image features are aligned with the point cloud features to improve geometric accuracy. In the invalid point cloud region, the point cloud features are aligned with the image features to supplement semantic information and obtain the fused unified BEV features.

[0383] (6) 3D target detection and loss optimization: Several candidate boxes are generated by querying based on the fused BEV features. The center position, size, orientation and other parameters of the 3D bounding box are regressed and the target category probability is output. A total loss function including classification, localization, size, pose, feature alignment and height distillation is constructed. The model parameters are optimized through end-to-end training. Redundant detection boxes are eliminated by non-maximum suppression during the inference stage, and the final 3D target detection result is output.

[0384] Obtain from the public dataset SUNRGBD, such as Figure 3 The RGB image shown and as Figure 4 The depth image shown is obtained by using the method of this invention from... Figure 4 Extract from depth image Figure 5 The point cloud shown is then processed using the method of this invention. Figure 3 and Figure 5 Processing yields the following results: Figure 6 The target detection results shown demonstrate the effectiveness of the method of the present invention.

[0385] The core technical solution of this invention can be flexibly adjusted according to the hardware conditions, detection accuracy and efficiency requirements of the actual application scenario. For example, the type of image backbone network can be replaced, the division range of BEV grid can be adjusted, and the interval division method of the height distillation module can be modified. All of these can achieve the BEV multimodal fusion detection effect guided by cross-modal height distillation and adapt to different indoor three-dimensional target detection scenarios.

Claims

1. A method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error, characterized in that, The method specifically includes the following steps: Step 1: Acquire RGB images of the area to be detected using a monocular camera deployed on the indoor autonomous mobile robot, and acquire point cloud data using a lidar or depth camera deployed on the indoor autonomous mobile robot. Step 2: Extract two-dimensional feature maps from the RGB image of the region to be detected using the backbone network. Then the two-dimensional feature map After passing through the height prediction head, the vertical height distribution prediction result of the pixels in the RGB image of the region to be detected is obtained; Step 3: The two-dimensional feature map of the RGB image of the region to be detected and the height distribution prediction result are stitched together. Then, the stitched result is projected from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map of the RGB image of the region to be detected. Step 4: After dividing the BEV space into grids, use the established mapping function to map points with depths greater than or equal to the threshold to the BEV space. Based on the mapping results, filter the points in each grid obtained, and then obtain the point cloud BEV feature map based on the filtered point cloud. Step 5: Fuse the BEV feature map corresponding to the RGB image and the BEV feature map of the point cloud to obtain the fused BEV feature, and generate a proposal domain and random candidate boxes based on the RGB image of the region to be detected. The fused BEV features are then used as input to the object detection head, which outputs the object detection results based on the generated proposal domain and random candidate boxes.

2. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 1, characterized in that, In step one, the lidar directly collects point cloud data, and the depth map collected by the depth camera is converted into point cloud data in the following way: Based on the depth of each pixel in the depth map Define the depth mask image: in: Representing a depth map pixels in The depth value; Represents the pixels in the depth mask map The value; Obtain the valid point set based on the pixel values ​​in the depth mask image. : For a valid pixel in the set of valid points , will effective pixels The depth value is denoted as Then define the intermediate vector. for: in, express Transpose of; Using the inverse of the camera intrinsic matrix, pixel coordinates are converted into 3D points in the camera coordinate system: in: Indicates valid pixels The corresponding 3D point in the camera coordinate system; Representing the intrinsic parameter matrix The reverse.

3. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 1, characterized in that, The specific process of step two is as follows: in: Represents pixels in an RGB image Features; Indicates a high-prediction head; Indicates the Softmax activation function layer; Represents pixels in an RGB image The predicted height distribution in the vertical direction.

4. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 3, characterized in that, The specific process of step three is as follows: Step 31: Define the BEV space range: ; in: Indicates BEV space in Minimum coordinate value in the axial direction, Indicates BEV space in Maximum coordinate value in the axial direction; Indicates BEV space in Minimum coordinate value in the axial direction, Indicates BEV space in Maximum coordinate value in the axial direction; Indicates BEV space in Minimum coordinate value in the axial direction, Indicates BEV space in Maximum coordinate value in the axial direction; Step 3.2: Obtain the depth of the corresponding point in the point cloud data for each pixel in the RGB image, and remove points in the point cloud data whose depth is less than the threshold. The pixels corresponding to the points in the RGB image and the pixels whose projection range exceeds the BEV space range; The features of the remaining pixels in the RGB image are concatenated with the height distribution prediction results of the remaining pixels in the RGB image to obtain the concatenated result. Step 3: Project the stitching result from the camera coordinate system to the BEV coordinate system using the camera intrinsic parameter matrix to obtain the BEV feature map corresponding to the RGB image. .

5. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 4, characterized in that, The specific process of step four is as follows: Step 41: Set the grid size for each direction of the BEV space as follows. , and The BEV space is divided into grids according to the set grid size, where, shaft and The axial direction is uniformly divided. The axial direction is divided non-uniformly; Step 42: Establish a mapping function and use it to map points to the BEV space grid. in: Representing a three-dimensional point Mapping function values ​​to the BEV space grid; Indicates an indicator function; Indicates taking the absolute value; Indicates rounding down; Step 43: Based on the mapping results in Step 42, filter the points within the grid to obtain the points retained in each grid. Step 4: Aggregate the points retained within each grid: in: Represents a grid The actual number of points retained internally. Represents a grid The first one that was preserved inside The coordinates of the points; Represents a grid The aggregation result; Steps four and five: Use an intermediate encoder to convert the aggregation results of each grid into a dense BEV feature map. ; Step 46: Use the SECOND network to process the dense BEV feature map Processing: in: This represents the output of the SECOND network. Indicates a SECOND network; Step 47: Using the SECONDFPN network to... Perform multi-scale feature fusion: in: This represents the result of multi-scale feature fusion. Indicates SECONDFPN network; Step 48, Transform the point cloud into the BEV embedding space, which is the same as the image features, to obtain the BEV feature map. : in: This indicates a convolutional layer.

6. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 5, characterized in that, The specific process of step 41 is as follows: Each grid The axial dimension is constant. Each grid The axial dimension is constant. For a uniformly divided dimension, the grid boundaries are respectively ; in: , express Axial direction The lower boundary of the layer mesh, express Number of grid layers in the axial direction; , express Axial direction The lower boundary of the layer mesh, express Number of grid layers in the axial direction; For dimensions with non-uniform partitioning, the number of mesh layers along the depth axis is denoted as... The LID non-uniform grid boundary sequence is Let the lower boundary of the 0th layer mesh in the depth axis direction be... Let the depth axis direction be the first The upper boundary of the layer mesh is Then the depth axis direction of the first The lower boundary of the layered mesh is generated by a quadratic function: in: , Indicates the depth axis direction. The lower boundary of the layer mesh; Depth axis direction Width of layer grid for: 。 7. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 6, characterized in that, The backbone network, height prediction head, intermediate encoder, SECOND network, and SECONDFPN network need to be trained together by a height distribution distillation module, which constructs height distribution labels for RGB images based on point cloud data.

8. The method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error as described in claim 7, characterized in that, The working process of the highly distributed distillation module is as follows: Vertical range of point cloud data Evenly divided into A height range for pixels in an RGB image. : like For effective pixels, then based on the camera intrinsic parameter matrix Get pixels In point cloud data The corresponding point in the image, based on pixels In point cloud data The height of the corresponding point in the middle Construct highly distributed labels: in: Represents pixels In the Height distribution values ​​for each height interval; Indicates the first The height value at the center of each height interval; Indicates the first The height value at the center of each height interval; The bandwidth parameter of the Gaussian kernel; like For invalid pixels, a uniform distribution is constructed as the height distribution label: Will As pixels High-distribution labels.

9. A method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error, as described in claim 8, is characterized in that... When training the backbone network, height prediction head, intermediate encoder, SECOND network, and SECONDFPN network in conjunction with the height distribution distillation module, the loss function used is: in: Represents classification loss. The weights representing the classification loss; Indicates location loss. The weights representing the localization loss; Indicates dimensional loss. The weight representing the size loss; Indicates attitude loss, The weights representing the pose loss; Indicates feature alignment loss, The weights represent the feature alignment loss. This indicates the distillation loss due to the high distribution of KL divergence. The weight representing the distillation loss due to the KL divergence height distribution; in: express The first in One element, Represents an arbitrarily small constant; in: To balance the weights for each category, To focus on weight, As an intermediate variable; in: This indicates that the current area is the real target. This indicates the probability that the current area is the target. in: This represents the three-dimensional coordinates of the predicted target center location. The actual three-dimensional coordinates of the target center position. For uncertain parameters, The base of the natural logarithm. Represents the L1 norm; in: Indicates the predicted target size. For the actual target size; in: This represents the relative rotation angle between the predicted direction and the true direction. in: and All are intermediate variables; in: For the effective mask of the point cloud, This represents the point cloud BEV feature map obtained after the last network parameter update. This indicates element-wise multiplication; in: This represents the image feature map obtained after the last network parameter update. Indicates the valid image mask; Using the calculated loss function Backpropagation is performed to adjust the parameters of the backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module until the target localization error on the test set meets the requirements. Training stops when the target localization error on the test set meets the requirements, and the trained backbone network, altitude prediction head, intermediate encoder, SECOND network, SECONDFPN network, and altitude distribution distillation module are obtained.

10. A method for three-dimensional target detection of an indoor mobile robot based on height distillation guidance and positioning error, as described in claim 9, is characterized in that... Based on the detection requirements of different scenarios, different levels of IoU detection thresholds are set for the target localization error of the test set; The IoU detection threshold is set to 0.15, 0.25 or 0.

50. Then, based on the set IoU detection threshold, the average accuracy of all detection boxes under different confidence levels is calculated until the average accuracy is greater than the accuracy threshold to meet the target positioning error requirements.

Images