A three-dimensional target detection method for real-time perception in a mobile environment

Abstract

The application belongs to the technical field of target detection, and discloses a three-dimensional target detection method for real-time sensing in a mobile environment, which comprises the following steps: extracting image two-dimensional features by inputting a multi-view image of a mobile device into an image backbone network; inputting the image two-dimensional features into a depth prediction network and a semantic prediction network to obtain a depth prediction value and a semantic prediction value, performing multi-level depth supervision on the depth prediction value, and performing semantic supervision on the semantic prediction value; projecting the image two-dimensional features into a three-dimensional space to obtain BEV features; performing bilinear interpolation sampling on the BEV features by using a mixed resolution grid to form new BEV features; inputting the new BEV features into a BEV feature encoding network, and then obtaining a three-dimensional target detection result by using a 3D detection head; and performing detection supervision on the three-dimensional target detection result, and updating network parameters in combination with all the supervision. The application fully utilizes depth information to improve the detection effect, and adapts to actual detection requirements by using a mixed resolution grid.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a three-dimensional target detection method for real-time perception in a mobile environment. BACKGROUND

[0002] With the rapid development of intelligent technology, mobile devices such as vehicles, automated guided vehicles (AGVs), drones, and mobile robots have been widely applied in the fields of industry, transportation, logistics, and services. These devices require high real-time perception and target detection of the surrounding environment when performing tasks such as autonomous driving, intelligent transportation, environmental monitoring, and human-computer interaction. In particular, in complex and dynamic mobile environments, accurate three-dimensional target detection is crucial for collision avoidance, path planning, and decision-making execution. For vehicle, AGV, drone, and mobile robot application scenarios, a three-dimensional target detection method for real-time perception in a mobile environment can significantly improve the environmental perception ability of the device and provide important technical support for autonomous driving, intelligent logistics, and multi-field applications.

[0003] Traditional three-dimensional target detection methods have deficiencies in spatial representation intuitiveness, time sequence information utilization, and multi-sensor data fusion. For example, two-dimensional image detection is easily affected by lighting conditions, occlusions, and changes in viewing angle, and the sparsity of data from a single sensor and the limited field of view often make it difficult to fully perceive the three-dimensional information of the target.

[0004] In recent years, three-dimensional target detection methods based on multi-sensor data fusion and efficient computing have gradually attracted attention. These methods integrate data from multiple sensors such as cameras and lidars, combine Bird's Eye View (BEV) technology and deep learning models, and can provide more comprehensive and robust target perception capabilities in complex and dynamic environments.

[0005] Compared with traditional methods, the BEV method can provide more intuitive and unified environment representation by uniformly converting multi-sensor data into a bird's eye view perspective, thereby showing significant advantages in three-dimensional target detection, but still has some deficiencies. On the one hand, the existing BEV three-dimensional target detection method usually only focuses on the absolute depth information of each point of the target, and the geometric structure features of the target itself and the spatial distribution information between targets are not utilized, resulting in insufficient supervision of depth estimation, thereby affecting the detection accuracy. On the other hand, the existing BEV three-dimensional target detection method usually sets a uniform size grid in the BEV space, and considering the actual demand, the detection result needs to be more accurate in the vicinity of the distance itself. The setting of the uniform size grid makes the entire detection range at the same detection demand level, which cannot well adapt to the actual detection demand. Considering the above factors, the present application proposes a three-dimensional target detection method for mobile environment real-time perception. SUMMARY

[0006] The present application aims to provide a three-dimensional target detection method for mobile environment real-time perception, which fully utilizes depth information to improve detection effect, and adapts to actual detection demand through mixed resolution grid.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0008] A three-dimensional target detection method for mobile environment real-time perception, comprising:

[0009] Inputting the multi-view images of the mobile device in the training set into the image backbone network to extract image two-dimensional features;

[0010] Inputting the image two-dimensional features into the depth prediction network and the semantic prediction network respectively to obtain depth prediction values and semantic prediction values, performing multi-level depth supervision on the depth prediction values, and performing semantic supervision on the semantic prediction values;

[0011] Projecting the image two-dimensional features into a three-dimensional space to obtain BEV features;

[0012] Using a mixed resolution grid to bilinearly interpolate and sample the BEV features to form new BEV features;

[0013] Inputting the new BEV features into the BEV feature encoding network, and then obtaining the three-dimensional target detection result of the mobile device environment through the 3D detection head;

[0014] Detecting the three-dimensional target detection result, and updating the image backbone network, the depth prediction network, the semantic prediction network, the BEV feature encoding network and the 3D detection head in combination with the multi-level depth supervision and the semantic supervision;

[0015] Real-time acquisition of new mobile device multi-view images, input into the trained image backbone network, depth prediction network, semantic prediction network, BEV feature encoding network and 3D detection head, obtain the corresponding three-dimensional target detection result, and obtain the bird's eye view detection result based on the three-dimensional target detection result.

[0016] The following also provides several optional modes, but not as an additional limitation to the above overall scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined with the above overall scheme, and can also be combined between multiple optional modes.

[0017] As preferred, the multi-level depth supervision of the depth prediction value comprises:

[0018] (1) Semantically weighted point-level absolute depth supervision: one-hot encoding and down-sampling the depth ground truth and semantic ground truth of the mobile device multi-view image to the size of the image two-dimensional feature to obtain depth label value D lable and semantic label value S lable , then based on the depth prediction value D pred and the depth label value D lable , the Binary Cross Entropy loss is calculated as follows:

[0019]

[0020] In the formula, L1 represents the semantically weighted point-level absolute depth loss of the mobile device multi-view image, Binary Cross Entropy loss with input depth prediction value D pred and depth label value D lable , ω s is the foreground-background weight factor, ω1 is the foreground weight factor, and ω2 is the background weight factor. S lable ≠0 indicates that the pixel point belongs to the foreground, and S lable =0 indicates that the pixel point belongs to the background.

[0021] (2) Target internal structure level relative depth supervision: taking the pixel point with the smallest absolute value of the internal depth prediction error of each target in the mobile device multi-view image as the internal structure reference point θ of the target, the depth prediction value and the depth label value of the internal structure reference point θ are respectively denoted as and The absolute value of the depth prediction error is the absolute value of the difference between the depth prediction value and the depth label value. The depth prediction difference and the depth label difference of each internal pixel point relative to the internal structure reference point θ are calculated, and the MSE loss of the two is calculated, the formula is as follows:

[0022]

[0023] L2 represents the target internal structure level relative depth loss of the mobile multi-view image, the MSE loss of the input being the depth prediction difference and the depth label difference ;

[0024] (3) Target spatial distribution level relative depth supervision: take the point with the minimum target internal depth label value in each mobile multi-view image as the representative point a of the target, and the depth prediction value and the depth label value of the representative point a are denoted as and respectively. Calculate the difference between the depth prediction value and the depth label value , and take the representative point with the minimum absolute difference value in each mobile multi-view image as the target spatial distribution reference point β of the mobile multi-view image where the representative point is located. The depth prediction value and the depth label value of the spatial distribution reference point β are denoted as and respectively. Then calculate the depth prediction difference and the depth label difference of other representative points a in each mobile multi-view image relative to the spatial distribution reference point β of the mobile multi-view image where the representative point is located, and calculate the MSE loss of the two, the formula is as follows:

[0025]

[0026] L3 represents the target spatial distribution level relative depth loss of the mobile multi-view image, the MSE loss of the input being the depth prediction difference and the depth label difference ;

[0027] (4) Calculate the total depth supervision loss as follows:

[0028] L Depth = ε1*L1+ ε2*L2+ ε3*L3

[0029] L Depth represents the total depth supervision loss of the mobile multi-view image, ε1 represents the first level loss coefficient, ε2 represents the second level loss coefficient, and ε3 represents the third level loss coefficient.

[0030] As a preferred, the semantic supervision on the semantic prediction value comprises:

[0031] The semantic true value of the mobile multi-view image is one-hot encoded and down-sampled to the size of the image two-dimensional feature to obtain the semantic label value S lable , based on the semantic prediction value S predand semantic label value S lable , the Binary Cross Entropy loss is calculated as follows:

[0032]

[0033] In the formula, L Seg represents semantic loss, and ε4 represents a semantic loss coefficient, represents the input as a semantic prediction value S pred and semantic label value S lable Binary Cross Entropy loss.

[0034] As preferred, the projecting the image two-dimensional feature into a three-dimensional space to obtain the BEV feature comprises:

[0035] The image two-dimensional feature is outer multiplied with the depth prediction value to obtain the stretched image two-dimensional feature.

[0036] The stretched image two-dimensional feature is filtered to remove features with a depth prediction value less than a depth threshold value or a semantic prediction value less than a semantic threshold value.

[0037] The remaining features in the stretched image two-dimensional feature after filtering are projected into a three-dimensional space to obtain the BEV feature.

[0038] As preferred, the bilinear interpolation sampling of the BEV feature using the mixed resolution grid to form a new BEV feature comprises:

[0039] A plane rectangular coordinate system is established with the center of the grid plane of the BEV feature space as the origin.

[0040] The division for the x-axis is as follows: in the range of |x|≤x1, division is made according to a grid length of d1, in the range of x1<|x|≤x2, division is made according to a grid length of d2, and in the range of x2<|x|≤x3, division is made according to a grid length of d3, where x1<x2<x3, d1<d2<d3.

[0041] The division for the y-axis is as follows: in the range of |y|≤y1, division is made according to a grid length of d1, in the range of y1<|y|≤y2, division is made according to a grid length of d2, and in the range of y2<|y|≤y3, division is made according to a grid length of d3, where x1=y1, x2=y2, and x3=y3.

[0042] The BEV feature is bilinearly interpolated and sampled based on the new grid obtained after division to obtain a new BEV feature.

[0043] As preferred, the detection supervision of the three-dimensional target detection result comprises:

[0044] Obtaining a class label and a three-dimensional detection box label marked by a mobile device multi-view image corresponding to the three-dimensional target detection result;

[0045] Calculating a Gaussian focus loss according to the class label and a class prediction value in the three-dimensional target detection result as a class prediction loss;

[0046] Calculating an L1 loss according to the three-dimensional detection box label and a three-dimensional detection box prediction value in the three-dimensional target detection result as a three-dimensional detection box prediction loss.

[0047] As preferred, the real-time acquisition of the new mobile device multi-view image is input into the trained image backbone network, the depth prediction network, the semantic prediction network, the BEV feature encoding network and the 3D detection head to obtain the corresponding three-dimensional target detection result, including:

[0048] Acquiring the new mobile device multi-view image and inputting the image two-dimensional feature to be perceived into the trained image backbone network to extract;

[0049] Inputting the image two-dimensional feature to be perceived into the trained depth prediction network and the semantic prediction network respectively to obtain a depth prediction value and a semantic prediction value;

[0050] Projecting the image two-dimensional feature to be perceived into a three-dimensional space to obtain a BEV feature to be perceived;

[0051] Using a mixed resolution grid to bilinearly interpolate and sample the BEV feature to be perceived to form a new BEV feature to be perceived;

[0052] Inputting the new BEV feature to be perceived into the trained BEV feature encoding network, and then obtaining the three-dimensional target detection result of the mobile device environment through the trained 3D detection head.

[0053] The three-dimensional target detection method for mobile environment real-time perception provided by the application fully utilizes the depth information through multi-level depth supervision of semantic weighted point-level absolute depth supervision, target internal structure level relative depth supervision and target space distribution level relative depth supervision, and sets a mixed resolution BEV grid to adapt to different detection requirements by setting grids of different sizes according to the distance from the center without changing the detection range and the size of the BEV feature map. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart for the three-dimensional target detection method for mobile environment real-time perception of the application applied in the training stage;

[0055] Figure 2 An embodiment schematic diagram of the mobile device multi-view image applied by the application;

[0056] Figure 3 An embodiment schematic diagram of the mixed resolution grid division of the present application;

[0057] Figure 4 An embodiment schematic diagram of the three-dimensional target detection method for mobile environment real-time perception of the present application applied in the inference stage;

[0058] Figure 5 An embodiment schematic diagram of the three-dimensional target detection result and the bird's eye view detection result of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application.

[0061] Embodiment 1: A three-dimensional target detection method for mobile environment real-time perception, applied in the training stage, as shown in Figure 1 The method comprises the following steps:

[0062] Step 1: inputting the multi-view images of the mobile device in the training set into the image backbone network to extract the image two-dimensional features (referred to as image features), denoted as F img ∈R N×C×H×W wherein N represents the number of multi-view images of the mobile device in the training set, C represents the feature channel number of the image two-dimensional features, and H and W represent the feature map size of the image two-dimensional features. In this embodiment, the backbone network adopts a ResNet50 neural network.

[0063] The multi-view images of the mobile device should be understood as obtaining one or more view images around the mobile device. If there are multiple views, multiple images corresponding to the multiple views at the same time are taken as the original input of the image backbone network. Taking a common vehicle as an example, Figure 2As shown, the acquired mobile device multi-view image contains images captured by image acquisition devices in front, left front, left rear, rear, right rear and right front of the vehicle. In other embodiments, the view angles can be flexibly combined according to application scenarios and hardware conditions, for example, only one view angle in front, or three view angles in front + left front + right front, or two view angles in front + rear, or eight view angles if the mobile device is large or the camera view angle range is small, etc. The present embodiment is not limited.

[0064] Step 2, input the image two-dimensional features into the depth prediction network and the semantic prediction network respectively to obtain the depth prediction value and the semantic prediction value, perform multi-level depth supervision on the depth prediction value, and perform semantic supervision on the semantic prediction value.

[0065] Step 21, input the image two-dimensional features F img into the depth prediction network and the semantic prediction network respectively to obtain the depth prediction value D pred ∈R N×118×H×W and the semantic prediction value S pred ∈R N×2×H×W , wherein 118 represents 118 different discrete depths, 2 represents the length of the corresponding dimension, and contains foreground probability and background probability for distinguishing foreground and background, for example, (0.4, 0.6) represents that the foreground probability of the pixel point is 0.4 and the background probability is 0.6.

[0066] The discrete depth range in the present embodiment is 1-60m with an interval of 0.5m, and it should be noted that in other embodiments, the number, range and interval of the discrete depth can be adjusted as needed. And the depth prediction network and the semantic prediction network in the present embodiment are convolutional neural networks composed of basic convolutional modules, which are not specifically limited in the present embodiment.

[0067] Step 22, perform multi-level depth supervision on the depth prediction value D pred , which is semantic weighted point-level absolute depth supervision, target internal structure level relative depth supervision and target spatial distribution level relative depth supervision, specifically as follows:

[0068] (1) Semantic weighted point-level absolute depth supervision: one-hot encoding and down-sampling the depth ground truth and semantic ground truth of the mobile device multi-view image to the image feature map size to obtain the depth label value Dl able ∈R N×118×H×W and the semantic label value S lable ∈R N×2×H×W , then based on the depth prediction value D pred and the depth label value D lable , calculate the Binary CrossEntropy loss as follows:

[0069]

[0070] In the formula, L1 represents the semantic weighted point-level absolute depth loss of the multi-view image of the mobile device, , which is the Binary CrossEntropy loss of the input being the depth prediction value D pred and the depth label value D lable , ω s is the foreground-background weight factor, ω1 is the foreground weight factor, ω2 is the background weight factor, S lable ≠0 indicates that the pixel point belongs to the foreground, and S lable =0 indicates that the pixel point belongs to the background. Different weight factors ω s are assigned to the foreground and the background when the loss is calculated in the embodiment, so as to balance a large number of background pixels and a small number of foreground pixels. The semantic weighted point-level absolute depth supervision only supervises the absolute depth error of each point, the points are independent of each other, and the supervision of the foreground pixels is emphasized.

[0071] (2) Target internal structure level relative depth supervision: taking the pixel point with the minimum absolute value of the internal depth prediction error of each target in the multi-view image of the mobile device as the internal structure reference point θ of the target, the depth prediction value and the depth label value of the internal structure reference point θ are respectively denoted as and The absolute value of the depth prediction error is the absolute value of the difference between the depth prediction value and the depth label value. The depth prediction difference and the depth label difference of each internal pixel point of the target relative to the internal structure reference point θ are calculated, and the MSE loss of the two is calculated, and the formula is as follows:

[0072]

[0073] In the formula, L2 represents the target internal structure level relative depth loss of the multi-view image of the mobile device, , which is the MSE loss of the input being the depth prediction difference and the depth label difference . The target internal structure level relative depth supervision supervises the relative depth between the internal points of the target. All points of each target within the same image are correlated, and the geometric structure information of the target itself is fully utilized.

[0074] (3) Target spatial distribution level relative depth supervision: taking the point with the minimum depth label value of each target in each mobile device multi-view image as the representative point α of the target, the depth prediction value and the depth label value of the representative point α are respectively denoted as and The depth prediction value and the depth label value the difference between the depth prediction value and the depth label value of the spatial distribution reference point β of the mobile multi-view image in which the representative point α is located, and the MSE loss of the two is calculated, and the formula is as follows: and The depth prediction difference and the depth label difference of the other representative points α in each mobile multi-view image relative to the spatial distribution reference point β of the mobile multi-view image in which the representative point α is located are calculated again, and the MSE loss of the two is calculated, and the formula is as follows:

[0075]

[0076] In the formula, L3 represents the target spatial distribution level relative depth loss of the mobile multi-view image, The MSE loss of the input depth prediction difference and the depth label difference The target spatial distribution level relative depth supervision supervises the relative depth between the target representative points, and the targets are related to each other, and the spatial distribution information between the targets is fully utilized.

[0077] (4) Calculate the total depth supervision loss, and give different loss coefficients to each level:

[0078] L Depth = ε1*L1+ ε2*L2+ ε3*L3

[0079] In the formula, L Depth represents the total depth supervision loss of the mobile multi-view image, ε1 represents the first level loss coefficient, ε2 represents the second level loss coefficient, and ε3 represents the third level loss coefficient.

[0080] Step 23, based on the semantic prediction value S pred and the semantic label value S lable , calculate the Binary Cross Entropy loss to supervise the semantic prediction value:

[0081]

[0082] In the formula, L Seg represents the semantic loss, ε4 represents the semantic loss coefficient, The Binary Cross Entropy loss of the input semantic prediction value S pred and the semantic label value S lable .

[0083] Step 24, calculate the total loss L ZAs follows:

[0084] L Z = L Depth + L Seg

[0085] Step 3, project the image two-dimensional features into the three-dimensional space to obtain the BEV features.

[0086] Based on the Lift-Splat-Shoot (LSS for short) algorithm, the specific process of obtaining the BEV features by the embodiment is as follows: the image two-dimensional features F img are cross-multiplied with the depth prediction value to obtain the stretched image two-dimensional features (which is an operation inside the Lift-Splat-Shoot algorithm); in order to improve the projected features, the features whose depth prediction value is less than a depth threshold (which is set to 0.0085 in the embodiment and can be adjusted) or whose semantic prediction value is less than a semantic threshold (which is set to 0.25 in the embodiment and can be adjusted) are filtered from the stretched image two-dimensional features; then the remaining features after filtering the stretched image two-dimensional features are projected into the three-dimensional space to obtain the BEV features, denoted as F , where C BEV represents the number of feature channels, and 128 represents the BEV feature map size, which can be selected according to actual conditions.

[0087] Step 4, bilinear interpolation sampling is performed on the BEV features by using the mixed resolution grid to form new BEV features.

[0088] The grid of the original BEV feature space is composed of 128*128 squares with a side length of 0.8m, in order to adapt to the actual detection requirements, without changing the number of grids and the detection range, bilinear interpolation sampling is performed on the BEV features by using the mixed resolution grid to form new BEV features, denoted as F′ BEV . The specific operation is as follows: first, the grid of the original BEV feature space is re-divided to obtain a new grid, i.e., the mixed resolution grid, and then the original BEV features are bilinearly interpolated and sampled based on the mixed resolution grid to form new BEV features.

[0089] The specific division process of the mixed resolution grid is as follows: a plane rectangular coordinate system is established with the center of the grid plane of the original BEV feature space as the origin; the division of the x-axis is as follows: in the range of |x|≤x1, division is performed with a grid length of d1, in the range of x1<|x|≤x2, division is performed with a grid length of d2, and in the range of x2<|x|≤x3, division is performed with a grid length of d3, wherein x1<x2<x3, d1<d2<d3; the division of the y-axis is as follows: in the range of |y|≤y1, division is performed with a grid length of d1, in the range of y1<|y|≤y2, division is performed with a grid length of d2, and in the range of y2<|y|≤y3, division is performed with a grid length of d3, wherein x1=y1, x2=y2, and x3=y3. x1 is the first division boundary value of the x-axis, x2 is the second division boundary value of the x-axis, x3 is the third division boundary value of the x-axis, y1 is the first division boundary value of the y-axis, y2 is the second division boundary value of the y-axis, y3 is the third division boundary value of the y-axis, d1 is the first length value, d2 is the second length value, and d3 is the third length value.

[0090] As shown in Figure 3 For the original BEV feature grid composed of 128*128 squares with a side length of 0.8 m, the embodiment provides an excellent division example as follows: a plane rectangular coordinate system is established with the center of the grid plane as the origin, in the range of |x|≤9.6, division is performed with a grid length of 0.6 m, in the range of 9.6<|x|≤35.2, division is performed with a grid length of 0.8 m, and in the range of 35.2<|x|≤51.2, division is performed with a grid length of 1 m, and the y direction is divided in the same way, thereby forming 3*3=9 different size grids. The mixed resolution grid does not increase the total number of grids, and therefore does not affect the speed of detection, and the near-small and far-large local grid design adapts to the actual detection demand.

[0091] Step 5: input the new BEV feature into the BEV feature encoding network, and obtain the three-dimensional target detection result of the mobile device environment through the 3D detection head.

[0092] Step 5: input the new BEV feature F B ′ EV into the BEV feature encoding network, and obtain the three-dimensional target detection result of the mobile device environment through the 3D detection head. The three-dimensional target detection result includes a class prediction value and a three-dimensional detection box prediction value in a three-dimensional space centered on the target, wherein the three-dimensional detection box prediction value includes three-dimensional coordinates of the detection box center, length, width, and height of the detection box, and a yaw angle of the detection box.

[0093] Step 6: perform detection supervision on the three-dimensional target detection result, and update the image backbone network, the depth prediction network, the semantic prediction network, the BEV feature encoding network, and the 3D detection head in combination with multi-level depth supervision and semantic supervision.

[0094] The detection supervision includes category prediction supervision and three-dimensional detection box prediction supervision, specifically: obtaining a category label and a three-dimensional detection box label marked by a mobile device multi-view image corresponding to a three-dimensional target detection result; calculating a Gaussian focal loss (Gaussian Focal Loss) as a category prediction loss according to the category label and a category prediction value in the three-dimensional target detection result; and calculating an L1 loss (L1 Loss) as a three-dimensional detection box prediction loss according to the three-dimensional detection box label and a three-dimensional detection box prediction value in the three-dimensional target detection result.

[0095] The total loss L of the image two-dimensional feature is added to the Gaussian focal loss and the L1 loss as a final detection loss, and the parameters of the image backbone network, the depth prediction network, the semantic prediction network, the BEV feature encoding network and the 3D detection head are updated by using the gradient descent method according to the final detection loss until the training is completed. Z The total loss L of the image two-dimensional feature is added to the Gaussian focal loss and the L1 loss as a final detection loss, and the parameters of the image backbone network, the depth prediction network, the semantic prediction network, the BEV feature encoding network and the 3D detection head are updated by using the gradient descent method according to the final detection loss until the training is completed.

[0096] Embodiment 2: A three-dimensional target detection method for real-time perception in a mobile environment, applied in an inference stage, as shown in FIG. 2, comprising the following steps: Figure 4 As shown in FIG. 2, comprising the following steps:

[0097] Step 1, real-time collection of new mobile device multi-view images, input into the trained image backbone network to extract image two-dimensional features to be perceived, denoted as F img ∈R N×C×H×W Wherein N represents the number of mobile device multi-view images in the training set, C represents the feature channel number of the image two-dimensional feature, and H and W represent the feature map size of the image two-dimensional feature. In this embodiment, the backbone network uses a ResNet50 neural network.

[0098] Step 2, input the image two-dimensional features to be perceived into the trained depth prediction network and semantic prediction network respectively to obtain depth prediction values and semantic prediction values.

[0099] Step 3, project the image two-dimensional features to be perceived into a three-dimensional space to obtain BEV features to be perceived.

[0100] Step 4, bilinear interpolation sampling of the BEV features to be perceived is performed using a hybrid resolution grid to form new BEV features to be perceived.

[0101] Step 5, input the new BEV features to be perceived into the trained BEV feature encoding network, and then obtain a three-dimensional target detection result of the mobile device environment through the trained 3D detection head, and further obtain an aerial perspective detection result based on the three-dimensional target detection result.

[0102] The new BEV features F B ′EV The input BEV feature encoding network is further input into a 3D detection head, and the 3D detection head obtains a three-dimensional target detection result of the target in the three-dimensional space, the three-dimensional target detection result including a class prediction value and a three-dimensional bounding box prediction value in a three-dimensional space centered on the target, wherein the three-dimensional bounding box prediction value includes three-dimensional coordinates of a center of the bounding box, length, width and height of the bounding box, and a yaw angle of the bounding box. That is, eight corner points of the three-dimensional bounding box are obtained, and space coordinate conversion is performed in combination with camera extrinsic and intrinsic parameters to project the eight corner points in the three-dimensional space to a 2D plane of an image, so as to obtain the three-dimensional target detection result on the image. In addition, the three-dimensional space is flattened by removing the height dimension, and a bird's eye view plane display is obtained.

[0103] The embodiment provides a detection result of a vehicle as a mobile device, as shown in Figure 5 , as shown in Figure 5 , the left side in Figure 5 is a three-dimensional target detection result, including three-dimensional bounding boxes of various targets, and different colors represent targets of different categories.

[0104] For specific limitations of the three-dimensional target detection method for real-time perception in a mobile environment in the embodiment, refer to the limitations of the three-dimensional target detection method for real-time perception in a mobile environment in Embodiment 1, and the embodiment will not be described in detail.

[0105] It is easy to understand that, in order to facilitate understanding, the application is described as a three-dimensional target detection method for real-time perception in a mobile environment, which is divided into a training stage and an inference stage. However, in actual application, the training stage and the inference stage can exist separately, or can exist simultaneously in sequence, that is, Embodiment 1 and Embodiment 2 can be implemented separately, or Embodiment 1 and Embodiment 2 can be combined to implement a new whole embodiment.

[0106] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the description.

[0107] The above-described embodiments only express several embodiments of the application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that, for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are within the scope of protection of the application. Therefore, the protection scope of the application should be subject to the appended claims.

Claims

1. A three-dimensional target detection method for real-time perception in mobile environments, characterized in that: The three-dimensional target detection method for real-time perception in a mobile environment includes: The multi-view images of mobile devices in the training set are input into the image backbone network to extract the two-dimensional features of the images; The two-dimensional features of the image are input into the depth prediction network and the semantic prediction network respectively to obtain the depth prediction value and the semantic prediction value, and multi-level depth supervision is performed on the depth prediction value and semantic supervision is performed on the semantic prediction value; Project the two-dimensional features of the image into three-dimensional space to obtain BEV features; The mixed-resolution grid is used to perform bilinear interpolation sampling on the BEV features to form new BEV features; The new BEV features are input into the BEV feature encoding network, and then the 3D detection head is used to obtain the 3D object detection results of the mobile device environment; Perform detection supervision on the 3D object detection results, and combine multi-level depth supervision and semantic supervision to update the image backbone network, depth prediction network, semantic prediction network, BEV feature encoding network and 3D detection head; Collect new multi-view images from mobile devices in real time and input them into the trained image backbone network, depth prediction network, semantic prediction network, BEV feature encoding network and 3D detection head to obtain the corresponding 3D object detection results. Based on the 3D object detection results, the bird's-eye view detection results are obtained. The multi-level depth supervision of the depth prediction value includes: (1) Semantic-weighted point-level absolute depth supervision: The depth truth value and semantic truth value of the multi-view image of the mobile device are one-hot encoded and downsampled to the size of the image's two-dimensional features to obtain the depth label value and semantic label values , then based on the depth prediction value and depth tag value , calculate the Binary Cross Entropy loss as follows: ； ； Where, Representing semantically weighted point-level absolute depth loss for multi-view images on mobile devices, Indicates that the input is a depth prediction value and depth tag value Binary CrossEntropy loss, is the foreground background weight factor, is the foreground weight factor, is the background weight factor, Indicates that the pixel belongs to the foreground, Indicates that the pixel belongs to the background; (2) Relative depth supervision at the target internal structure level: The pixel with the smallest absolute value of the depth prediction error of each target in the multi-view image of the mobile device is taken as the internal structure reference point of the target. , internal structure reference point The depth prediction value and depth label value are recorded as and The absolute value of the depth prediction error is the absolute value of the difference between the depth prediction value and the depth label value. Calculate the relative value of the internal structure reference point of each target's other pixels. The depth prediction difference and depth label difference are calculated, and the MSE loss of the two is calculated as follows: ； Where, Represents the relative depth loss of the target internal structure level of multi-view images of mobile devices, Indicates that the input is a depth prediction difference and the depth label difference MSE loss; (3) Relative depth supervision at the target spatial distribution level: In each multi-view image of a mobile device, the point with the smallest depth label value inside each target is taken as the representative point of the target. , representing the point The depth prediction value and depth label value are recorded as and , calculate the depth prediction value and depth tag value The difference between the two, in each mobile device multi-view image, the representative point with the smallest absolute difference is taken as the target space distribution reference point of the mobile device multi-view image where the representative point is located , spatial distribution reference points The depth prediction value and depth label value are recorded as and , and then calculate other representative points in each mobile device multi-view image Spatial distribution reference points relative to the multi-view images of the mobile device The depth prediction difference and depth label difference are calculated, and the MSE loss of the two is calculated as follows: ； Where, Represents the relative depth loss of target spatial distribution level of multi-view images of mobile devices, Indicates that the input is a depth prediction difference and the depth label difference MSE loss; (4) Calculate the total loss of deep supervision as follows: ； Where, represents the total loss of deep supervision for multi-view images on mobile devices, represents the first-level loss coefficient, represents the second-level loss coefficient, Represents the third-level loss coefficient.

2. The three-dimensional target detection method for real-time perception in mobile environments according to claim 1 is characterized in that: The semantic supervision of the semantic prediction value includes: The semantic truth value of the multi-view image of the mobile device is one-hot encoded and downsampled to the size of the two-dimensional feature of the image to obtain the semantic label value , based on semantic prediction value and semantic label values , calculate the Binary CrossEntropy loss as follows: ； Where, represents semantic loss, represents the semantic loss coefficient, Indicates that the input is a semantic prediction value and semantic label values Binary Cross Entropy loss.

3. The three-dimensional target detection method for real-time perception in mobile environments according to claim 1 is characterized in that: The projecting of the two-dimensional features of the image into the three-dimensional space to obtain the BEV features includes: Perform outer product between the two-dimensional features of the image and the depth prediction value to obtain the stretched two-dimensional features of the image; Filter out the features whose depth prediction value is less than the depth threshold or whose semantic prediction value is less than the semantic threshold in the two-dimensional features of the stretched image; The remaining features after filtering in the stretched two-dimensional features of the image are projected into the three-dimensional space to obtain the BEV features.

4. The three-dimensional target detection method for real-time perception in mobile environments according to claim 1, characterized in that: The bilinear interpolation sampling of the BEV features using the mixed resolution grid to form new BEV features includes: Establish a plane rectangular coordinate system with the center of the grid plane of the BEV feature space as the origin; against The axis is divided as follows: Within the range, the side length of each grid is Divide, in Within the range, the side length of each grid is Divide, in The side length of each grid is Divide, among which , ; against The axis is divided as follows: Within the range, the side length of each grid is Divide, in Within the range, the side length of each grid is Divide, in The side length of each grid is Divide, among which , , ; Based on the new grid obtained after division, bilinear interpolation sampling is performed on the BEV features to obtain new BEV features.

5. The three-dimensional target detection method for real-time perception in mobile environments according to claim 1, characterized in that: The detection supervision of the three-dimensional target detection results includes: Obtain the category labels and 3D detection frame labels marked on the multi-view images of the mobile device corresponding to the 3D object detection results; Calculate the Gaussian focus loss as the category prediction loss based on the category label and the category prediction value in the 3D object detection result; The L1 loss is calculated based on the 3D detection box label and the 3D detection box prediction value in the 3D object detection result as the 3D detection box prediction loss.

6. The three-dimensional target detection method for real-time perception in mobile environments according to claim 1, characterized in that: The method collects new multi-view images of mobile devices in real time, inputs the trained image backbone network, depth prediction network, semantic prediction network, BEV feature encoding network and 3D detection head, and obtains the corresponding three-dimensional object detection results, including: Collect new multi-view images from mobile devices and input them into the trained image backbone network to extract the two-dimensional features of the image to be perceived; The two-dimensional features of the image to be perceived are input into the trained depth prediction network and semantic prediction network respectively to obtain the depth prediction value and the semantic prediction value; Project the two-dimensional features of the image to be perceived into three-dimensional space to obtain the BEV features to be perceived; A mixed-resolution grid is used to perform bilinear interpolation sampling on the BEV features to be sensed to form new BEV features to be sensed; The new BEV features to be perceived are input into the trained BEV feature encoding network, and then the three-dimensional target detection results of the mobile device environment are obtained through the trained 3D detection head.

Images