A feature enhancement based three-dimensional perception method

Abstract

The application discloses a three-dimensional perception method based on feature enhancement, comprising the following steps: inputting an acquired RGB image into a 2D detector for target detection to obtain bounding box information as region proposals, each bounding box information being for a target; obtaining foreground points of an original point cloud on the RGB image; obtaining a truncated cone formed from the RGB image to a 3D space; obtaining a completed point cloud after feature completion; performing dimension splicing on point cloud information, region proposal information and centroid information of the completed point cloud to obtain spliced point cloud; sending the spliced point cloud into a segmentation network for instance segmentation to output 3D bounding box information of an instance target. The application uses a ball query strategy to complete key information, aiming at completing information of point cloud near an instance object after downsampling based on an original point cloud; meanwhile, an extra semantic information is provided for each point after feature extraction based on a multi-level connection feature expansion strategy, so that the bounding box regression effect of a regression network is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of 3D target detection, and in particular to a three-dimensional perception method based on feature enhancement. BACKGROUND

[0002] At present, 3D target detection methods mainly include Voxel-based method and Point-based method. The Voxel-based method divides large-scale point cloud data into voxel blocks for feature extraction, which inevitably causes information loss, mainly because the number of points in each voxel is finally a fixed value, and if the number of points in a voxel exceeds the limit, the excess points will be directly filtered out. The Point-based method directly uses original point cloud for feature extraction, which can retain more accurate point information, but due to the unordered nature and large number of points in the original point cloud, the computational cost is very high.

[0003] We believe that the information integrity of features plays a key role in three-dimensional perception. For example: (1) In real scene point cloud, the number of background points is generally much larger than that of foreground points. Referring to the idea in the 2D target detection network RetinaNet, using focal loss in the training stage can slightly reduce the influence of this phenomenon. However, some models based on original point cloud use random downsampling strategy when implementing downsampling, which may cause serious loss of object-related information. (2) To overcome the defects of voxel-based models, some researchers use methods such as balancing voxel size, interpolation, and combining point cloud and voxel to reduce the information loss caused by voxel division.

[0004] In summary, the strategy of "perfecting features" is beneficial to improving the effect of three-dimensional perception in both types of models, but existing models pay more attention to optimizing the model architecture to improve the effect of three-dimensional perception, and "perfecting features" is only a small auxiliary means to improve the index, and there is no significant in-depth research in this direction. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provide a three-dimensional perception method based on feature enhancement.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] In the first aspect of the present application, a three-dimensional perception method based on feature enhancement is provided, which is used for processing original point cloud collected by a radar sensor and RGB image collected by a camera, and characterized in that it comprises the following steps:

[0008] The obtained RGB image is input into a 2D detector for target detection to obtain bounding box information as a region proposal, each bounding box information being for one target;

[0009] The foreground points are obtained from the original point cloud, and the foreground points of the original point cloud are projected and transformed through the conversion relationship between the radar sensor coordinates and the camera coordinates to obtain the foreground points of the original point cloud on the RGB image;

[0010] The foreground points in the region proposal are screened out in combination with the region proposal and the foreground points, and the screened foreground points are re-projected and transformed to the 3D space to obtain a truncated cone formed from the RGB image to the 3D space;

[0011] The original point cloud information in the radius range of each centroid position is supplemented into the truncated cone by using a sphere query operation after the centroid position of the object is found, to obtain a completed point cloud after feature completion;

[0012] The point cloud information of the completed point cloud, the region proposal information and the centroid information are dimensionally spliced to obtain a spliced point cloud;

[0013] The spliced point cloud is sent into a segmentation network for instance segmentation, and 3D bounding box information of an instance target is output.

[0014] Further, the projection and transformation of the foreground points of the original point cloud is as follows:

[0015]

[0016] In the formula, X is the original point cloud coordinates, i.e. (x, y, z, 1) T Y is the pixel coordinate system coordinate obtained after the projection and transformation, i.e. (u, v, 1) T ; is the projection matrix of the i-th camera, represents the correction matrix of the 0th camera, represents the rotation and translation matrix from the radar sensor coordinate system to the camera coordinate system.

[0017] Further, the centroid position of the object is found, and a voting mechanism is used to obtain the approximate centroid position of the object, including:

[0018] The point cloud features of the truncated cone are extracted by using an encoder to generate a series of seed points, i.e. a process of gradually obtaining deep features from shallow features through down-sampling;

[0019] The seed points are input to a voting layer, which first generates a 3+256-dimensional feature vector for the 256-dimensional feature of each seed point through an mlp+bn layer. The network learns the spatial offset and feature offset of the seed point to the object center point to obtain the center point by voting each point, i.e. the difference between the real value distance from the object center and the network output voting Δx . i ;

[0020] The clustered center obtained by the voting layer is clustered to obtain the object centroid position.

[0021] Further, the encoder uses sequentially connected set abstraction (SA) layers and feature propagation (FP) layers to extract features from the truncated cone points.

[0022] Further, in the process of gradually obtaining deep features from shallow features through downsampling, a centroid-aware downsampling strategy is introduced, i.e. points closer to the centroid are assigned higher weight masks Mask i The formula is as follows:

[0023]

[0024] Where f, b, l, r, u, and d represent the distances of a point to the 6 surfaces of the real boundary box, i.e. the front, back, left, right, top, and bottom surfaces.

[0025] Further, in the process of obtaining the approximate centroid position of the object using the voting mechanism, a first detection head is set to supervise the difference between the object center point and the predicted centroid position.

[0026] Further, the adaptive ball query operation includes the following steps:

[0027] Each obtained centroid position forms the center point of a sphere, and after setting the radius manually, the points inside each sphere are used as a local region for feature extraction;

[0028] Where the manually set radius is adaptively set according to the depth of the centroid position obtained by the voting layer; since the radar sensor has the characteristics of near large and far small, the radius of the object farther from the radar sensor is set smaller; the baseline standard of the radius is obtained by actual measurement, and the radius at any depth is calculated according to the principle of similar triangles.

[0029] Further, the point cloud information, region proposal information, and centroid information of the completed point cloud are dimensionally spliced to obtain a spliced point cloud, wherein:

[0030] The spliced point cloud includes eight dimensions, four dimensions of point cloud information of the completed point cloud, three dimensions of region proposal information, and one dimension of centroid information;

[0031] The four dimensions of the point cloud information of the completed point cloud are respectively an x-axis coordinate, a y-axis coordinate, a z-axis coordinate, and a reflection intensity r of a corresponding point;

[0032] The three dimensions of the region proposal information are respectively mask information, a category label, and an index label of the corresponding point; the mask information indicates that the point belongs to a foreground point or a background point, and is determined by combining an original point cloud using a projection transformation and boundary judgment; the category label indicates a category of a 2D bounding box to which the point belongs; and the index label indicates a number of 2D bounding boxes to which each point belongs;

[0033] The one dimension of the centroid information indicates a distance between a centroid obtained through a voting layer and a current point, and is used to provide additional information for a segmentation network, i.e., a point closer to the centroid is more likely to be a point constituting a target object.

[0034] Further, the spliced point cloud is sent into the segmentation network to perform instance segmentation, and 3D bounding box information of an instance target is output, including:

[0035] A PointNet segmentation network is used to perform instance segmentation on the spliced point cloud in each truncated cone, and the instance segmentation is binary classification, i.e., each point belongs to an object point or a background point;

[0036] After the object-related points are separated, a second detection head is input to perform 3D bounding box regression, wherein parameters required to be regressed by the second detection head include an orientation angle, a 3D bounding box size, and an angle point loss obtained through an angle point position and a center position.

[0037] Further, a loss function of the 3D bounding box regression is implemented in a weighted accumulation manner, including a centroid-aware down-sampling loss L seg_aware , an object center-aware loss L vote-reg , and a bounding box regression loss; wherein:

[0038] The centroid-aware down-sampling loss L seg_aware is fused in a loss function of a segmentation task by using a weight mask Mask i to achieve an effect of being implicitly combined into network training, and a formula is as follows:

[0039]

[0040] In the formula, s i represents a category of a predicted point, and p represents a probability that the predicted point belongs to the category.

[0041] The object center-aware loss Lvote-reg That is, the distance from the center of the object to the true value And the vote output Δx from the network i The difference is monitored using the following formula:

[0042]

[0043] In the formula, N represents the number of voting points on the object's surface, and a 3D spatial bias Δx is learned for each voting point. i To ensure the moved point is closer to the object's centroid, L1 loss is used to monitor the distance between the moved point and the object's centroid. l[point i [on object] indicates whether the voting point is on an object;

[0044] Bounding box regression loss includes orientation angle loss L head 3D bounding box size loss L size Corner loss L corner Three terms, orientation angle loss L head and 3D bounding box size loss L size The formulas are all as follows:

[0045]

[0046] Where y i For the true value, Here, δ is the predicted value, and δ is the hyperparameter.

[0047] Corner loss L corner The formula is as follows:

[0048]

[0049] in and These represent the corner points of the predicted and actual bounding boxes, respectively. This represents the corner point obtained by flipping the actual bounding box of the ground.

[0050] The beneficial effects of this invention are:

[0051] An exemplary embodiment of the present invention employs a strategy that combines 2D images to assist 3D target perception. It combines information from both dimensions in the form of a truncated cone, where only one target within each cone is fed into the network for perception. This model significantly reduces computational load while improving perception accuracy. The 2D images not only provide auxiliary region suggestion information but also additional feature information to the 3D perception network.

[0052] In the prior art, due to the large number of scene point clouds, a downsampling operation is required before being sent into a feature extraction network, which may cause serious loss of object-related information. In the present example embodiment, the key information is completed by using a ball query strategy, and the ball query operation is intended to complete the information of the point cloud near the instance object after downsampling based on the original point cloud.

[0053] Meanwhile, the method in the present example embodiment provides additional semantic information for each point after feature extraction based on the multi-level connection feature expansion strategy, thereby improving the bounding box regression effect of the regression network. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of a three-dimensional perception method based on feature enhancement provided in an example embodiment of the present application;

[0055] Figure 2 A schematic diagram of a truncated cone extraction process provided in an example embodiment of the present application

[0056] Figure 3 An adaptive ball query schematic diagram provided in an example embodiment of the present application;

[0057] Figure 4 A feature expansion schematic diagram provided in an example embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0059] In the description of the present application, it should be noted that the directions or positional relationships indicated by "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are described based on the directions or positional relationships described in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, "first" and "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0060] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms of "mounting", "connection", "connecting" should be understood in a broad sense and for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be direct connection, can also be indirect connection through an intermediate medium, and can be internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0061] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict.

[0062] Referring to Figure 1 , Figure 1 A flow chart of a feature enhancement-based three-dimensional perception method provided in an exemplary embodiment of the present application is shown, which is used for processing raw point clouds collected by a radar sensor and RGB images collected by a camera, and includes the following steps:

[0063] The acquired RGB image is input to a 2D detector for target detection to obtain bounding box information as a region proposal, each bounding box information being for one target;

[0064] The foreground points are obtained using the raw point clouds, and the foreground points of the raw point clouds are projected and transformed using the conversion relationship between the radar sensor coordinates and the camera coordinates to obtain the foreground points of the raw point clouds on the RGB image;

[0065] In combination with the region proposal and the foreground points, the foreground points within the region proposal are screened out, and the screened foreground points are re-projected and transformed to the 3D space to obtain a truncated cone formed from the RGB image to the 3D space;

[0066] After finding the object centroid positions, a sphere query operation is used to supplement the raw point cloud information within the radius range sphere of each centroid position into the truncated cone to obtain a completed point cloud after feature completion;

[0067] The point cloud information of the completed point cloud, the region proposal information and the centroid information are dimensionally spliced to obtain a spliced point cloud;

[0068] The spliced point cloud is sent to a segmentation network for instance segmentation, and 3D bounding box information of the instance target is output.

[0069] Specifically, in the present exemplary embodiment, first, an RGB image is input and detected using a 2D detector. Second, the obtained bounding box information is used as a region proposal, and a truncated cone is extracted using the original point cloud, and the point cloud in the truncated cone is the point cloud in the bounding box after the original point cloud is projected to the RGB image. However, due to the deviation of the 2D detector, the generated truncated cone cannot completely contain the instance object information, so a completion operation is used to complete the object information. Before completion, the approximate position of the object needs to be found, and then a sphere query operation is used to supplement the original point cloud information in the sphere with a radius range around the centroid position into the truncated cone. Then, the feature expansion is used to splice the additional auxiliary information obtained by the region proposal and the centroid estimation with the point cloud in the truncated cone in the dimension. Finally, the point cloud with the expanded information is sent to the segmentation network, and the 3D bounding box information of the instance target is output.

[0070] To solve the problem of large amount of calculation in the prior art Point-based method, an exemplary embodiment of the present application adopts a strategy of combining the use of 2D images to assist 3D target perception, and combines two dimensions of information in the form of a truncated cone, wherein each truncated cone is only sent into the network for perception for one target. This model greatly reduces the amount of calculation while improving the perception accuracy. The 2D image can provide supplementary feature information for the three-dimensional perception network in addition to providing auxiliary region proposal information.

[0071] In addition, in the prior art, due to the large amount of scene point cloud, a downsampling operation is required before the feature extraction network is sent, which may cause serious loss of object-related information. In the present exemplary embodiment, the key information is completed by using a sphere query strategy, and the sphere query operation is intended to complete the information of the point cloud near the instance object after downsampling based on the original point cloud.

[0072] Meanwhile, the method in the present exemplary embodiment provides additional semantic information for each point after feature extraction based on the multi-level connection feature expansion strategy (i.e., the point cloud information of the completed point cloud, the region proposal information, and the centroid information are spliced in the dimension to obtain the spliced point cloud), which improves the bounding box regression effect of the regression network.

[0073] More preferably, in an exemplary embodiment, the projection transformation is performed on the foreground points of the original point cloud, and the formula is as follows:

[0074]

[0075] In the formula, X is the original point cloud coordinate (x, y, z, 1) T , Y is the pixel coordinate system coordinate (u, v, 1) T obtained after the projection transformation; is the projection matrix of the i th camera, K0, which represents the rectification matrix of camera 0 (KITTI dataset has multiple cameras, but all are transformed based on camera 0), Radar sensor coordinate system to camera coordinate system rotation translation matrix (for example, 3 rows and 4 columns, the first 3 rows and 3 columns are a rotation matrix, and the fourth column is a translation matrix).

[0076] Specifically, the truncated cone extraction process needs to use the original point cloud and the RGB picture in combination. First, the RGB image is input into a 2D detector to perform target detection, and the obtained bounding box is called a region proposal. Then, the foreground points are obtained using the original point cloud. Since the coordinates of the radar sensor, the camera coordinate parameters, and the coordinate system conversion relationship are all given in the dataset (KITTI), the projection transformation can be obtained by matrix calculation. The foreground points of the original point cloud on the RGB image are obtained. In combination with the region proposal and the foreground points, the foreground points within the region proposal can be screened out. These points are re-transformed (back projection) to the 3D space, and as shown in Figure 2 , we can form a series of truncated cones from the image to the 3D space (one truncated cone is only for a single target). Finally, the point cloud in the truncated cone is used for classification or segmentation in the subsequent network.

[0077] More preferably, in an exemplary embodiment, the finding of the object centroid position adopts a voting mechanism to obtain the approximate centroid position of the object, including:

[0078] The finding of the object centroid position adopts a voting mechanism to obtain the approximate centroid position of the object, including:

[0079] The point cloud features of the truncated cone are extracted using the encoder to generate a series of seed points, which is a process of gradually obtaining deep features from shallow features through downsampling;

[0080] The seed points are input into the voting layer, which first generates a 3+256-dimensional feature vector for each 256-dimensional feature of the seed points through the mlp+bn layer. The network learns the spatial offset and feature offset of the seed point to the object center point to obtain the center point by voting each point, that is, the difference between the true value and the object center and the voting Δx i output by the network is supervised;

[0081] The clustered center obtained by the voting layer is clustered to obtain the object centroid position.

[0082] Specifically, in the present exemplary embodiment, a voting mechanism is used to obtain the approximate center of mass position of the object, and the present exemplary embodiment does not need to find the precise center of mass of the object, but only needs to find the rough center of mass to use the ball query algorithm, so part of the network precision and depth can be sacrificed in the encoding stage. In a preferred exemplary embodiment, the encoder uses the backbone of the conventional PointNet (Set Abstraction and Feature Propagation) for feature extraction (the encoder uses sequentially connected set abstraction (SA) layers and feature propagation (FP) layers to extract features of the points of the frustum), a voting layer generates a 3+256-dimensional feature vector for the 256-dimensional feature of each point through an mlp+bn layer, and sets the difference between the sum of the true values from the center of the object and the votes output by the network as the loss to constrain the votes for all voting points belonging to the object. Finally, the clustering center obtained by the voting layer is clustered.

[0083] More preferably, in an exemplary embodiment, in the process of gradually obtaining deep features from shallow features through downsampling, a center-aware downsampling strategy is introduced, that is, points closer to the center of mass are assigned higher weights i The formula is as follows:

[0084]

[0085] Wherein f, b, l, r, u, d respectively represent the distance of a point to the 6 surfaces of the real boundary box of the ground, and the 6 surfaces are front, back, left, right, top and bottom.

[0086] More preferably, in an exemplary embodiment, in the process of obtaining the approximate center of mass position of the object using the voting mechanism, a first detection head is provided to supervise the difference between the object center point and the predicted center of mass position.

[0087] More preferably, in an exemplary embodiment, as Figure 3 indicated, the ball query operation is an adaptive ball query operation, including the following steps:

[0088] Each obtained center of mass position forms the center point of a sphere, and after setting the radius artificially, the points in each sphere are used as a local region for feature extraction; this operation is to combine the newly expanded information for further feature extraction, so that the features input into the bounding box prediction network are more abundant. Since the prior feature map has been obtained through SA and FP in the voting layer, an additional feature extraction operation is performed on the point cloud after feature expansion (to ensure the same scale as the prior feature map), and the two are summed through add operation, so that the new features are integrated on the basis of the prior features.

[0089] Wherein, the radius set by artificial setting is adaptively set according to the depth of the center position obtained by the voting layer; since the radar sensor has the characteristics of near large and far small, the object farther away from the radar sensor is set to a smaller radius; the baseline standard of the radius is obtained by actual measurement, and the radius at any depth is calculated according to the principle of similar triangles. Wherein, the depth represents the distance of a point from the camera in the camera coordinate system.

[0090] Specifically, in the present exemplary embodiment, the method aims to improve the ability of local feature extraction, and the main idea is to form a center point of a sphere by setting a radius for each obtained center point, and use the points in each sphere as a local region for feature extraction.

[0091] More preferably, in an exemplary embodiment, the point cloud information of the completed point cloud, the region proposal information and the center information are dimensionally spliced to obtain a spliced point cloud, wherein:

[0092] As shown in Figure 4 The spliced point cloud includes eight dimensions, which are four dimensions of the point cloud information of the completed point cloud, three dimensions of the region proposal information and one dimension of the center information, and such spliced information is not affected by any data enhancement;

[0093] The four dimensions of the point cloud information of the completed point cloud are respectively the x-axis coordinate, the y-axis coordinate, the z-axis coordinate and the reflection intensity r of the corresponding point;

[0094] The three dimensions of the region proposal information are respectively the mask information seg, the class label cls and the index label idx of the corresponding point; the mask information represents that the point belongs to a foreground point or a background point, which is determined by combining the original point cloud using projection transformation and boundary judgment; the class label represents the category of the 2D bounding box to which the point belongs; and the index label represents the number of 2D bounding boxes to which each point belongs;

[0095] The one dimension of the center information ctr represents the distance between the center obtained by the final clustering of the voting layer and the current point, which is used to provide additional information to the segmentation network, i.e., the points closer to the center are more likely to be points constituting the target object.

[0096] More preferably, in an exemplary embodiment, the spliced point cloud is input into a segmentation network for instance segmentation, and 3D bounding box information of the instance target is output, including:

[0097] The spliced point cloud is input into a segmentation network for instance segmentation, and 3D bounding box information of the instance target is output, including:

[0098] The spliced point cloud in each truncated cone is segmented using a PointNet segmentation network, and the instance segmentation is binary classification, i.e., each point belongs to an object point or a background point.

[0099] After separating the relevant points of the object, the data is input into the second detection head for 3D bounding box regression. The parameters that the second detection head needs to regress include the orientation angle, the 3D bounding box size, and the corner loss obtained through the corner point position and the center position.

[0100] It should be noted that the orientation angle and bounding box size are obtained directly from the bounding box prediction network. The orientation angle represents the angle between the predicted bounding box and the upward axis of the camera coordinate system. The corner loss is the sum of the distances between the eight corners of the predicted box and the ground truth box. Since the position of the corner is determined by its center, size, and orientation, the corner loss can normalize the multi-task training of these parameters, as shown in the formula:

[0101]

[0102] in and These represent the corner points of the predicted and actual bounding boxes, respectively. This represents the corner point obtained by flipping the ground true bounding box, which is intended to avoid the huge loss caused by flipping the heading estimation. i, j, and k represent the size, orientation, and corner point index, respectively.

[0103] More preferably, in an exemplary embodiment, the loss function of the 3D bounding box regression is implemented in a weighted accumulation manner, including a centroid-aware downsampling loss L. seg_aware Object center perception loss L vote-reg (Right now Figure 1 Head1), bounding box regression loss (i.e. Figure 1 (Head2 in the text); where:

[0104] Centroid sensing downsampling loss L seg_aware To utilize weighted mask i The fusion is achieved by implicitly integrating it into the loss function of the segmentation task, as shown in the following formula:

[0105]

[0106] In the formula, s i Indicates the category of the predicted point. This indicates the probability that the predicted point belongs to this category;

[0107] Object center perception loss L vote-reg That is, the distance from the center of the object to the true value And the vote output Δx from the network i The difference is monitored using the following formula:

[0108]

[0109] where N denotes the number of object surface voting points, and a bias Δx in 3D space is learned for each voting point i To make the moved point closer to the object centroid, L1 loss is adopted to supervise the distance between the moved point and the object centroid, i.e. l[point i on object] denotes whether the voting point is on the object or not;

[0110] The bounding box regression loss includes three terms, i.e., orientation angle loss L head , 3D bounding box size loss L size , and corner point loss L corner The formulas of the orientation angle loss L head and the 3D bounding box size loss L size are as follows:

[0111]

[0112] where y i is the true value, is the predicted value, and δ is a hyperparameter;

[0113] The formula of the corner point loss L corner is as follows:

[0114]

[0115] where and denote the predicted and ground truth bounding box corner points, respectively, denotes the corner point obtained by flipping the ground truth bounding box.

[0116] Another example embodiment of the present application provides a feature-enhanced three-dimensional perception method, which has the same inventive concept as the above example embodiments and comprises a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the feature-enhanced three-dimensional perception method when executing the computer instructions.

[0117] The electronic device is in the form of a general computing device. The components of the electronic device can include, but are not limited to, the above-mentioned at least one processing unit, the above-mentioned at least one storage unit, and a bus connecting different system components, including the storage unit and the processing unit.

[0118] The storage unit stores program codes executable by the processing unit, so that the processing unit executes the steps described in the above “example method” section of the present specification according to various example embodiments of the present application. For example, the processing unit can execute the steps described in the above “example method” section of the present specification according to various example embodiments of the present application.Figure 1 The method shown in the figure.

[0119] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0120] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0121] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0122] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to this exemplary embodiment.

[0124] With the same inventive concept as the above exemplary embodiments, still another exemplary embodiment of the present application provides a storage medium having stored thereon computer instructions, which, when executed, perform the steps of the feature enhancement based three-dimensional perception method.

[0125] Based on such understanding, the technical solution of the present embodiment or the part of the technical solution that essentially contributes to the prior art or the part of the technical solution can be embodied in the form of a software product (program product) stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application.

[0126] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0127] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or component.

[0128] The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0129] The program code may be executed by one or more programmable processing devices, which can include processors, microprocessor, microcomputer or microcontrollers, as well as other forms of computing or processing devices. The processing devices can be central processing units (CPUs), graphics processing units (GPUs), or both, or any other processing devices capable of processing digital data. The processing devices can take the form of general-purpose processors or special-purpose processors. The processing devices can be used in a single computer system, or can be distributed among many computer systems. The processing devices can be used to execute one or more software programs, such as the program code.

[0130] Obviously, the above-described embodiments are only examples for the purpose of clarity and are not intended to limit the implementation. For those skilled in the art, other different forms of changes or variations can be made on the basis of the above description. All the implementations do not need to be exhausted here. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A feature enhancement based three-dimensional perception method for processing raw point clouds collected by a radar sensor and RGB images collected by a camera, characterized in that: The method comprises the following steps: The acquired RGB image is input into a 2D detector for target detection to obtain bounding box information as a region proposal, each of which is for a target; Foreground points are acquired from the original point cloud, and projection transformation is performed on the foreground points of the original point cloud through the conversion relationship between the radar sensor coordinates and the camera coordinates to obtain the foreground points of the original point cloud on the RGB image; The foreground points in the region proposal are screened out in combination with the region proposal and the foreground points, and the screened foreground points are re-projected into the 3D space to obtain a truncated cone formed from the RGB image to the 3D space; After the object centroid position is found, a ball query operation is used to supplement the original point cloud information in the radius range of each centroid position into the truncated cone to obtain a completed point cloud after feature completion; The point cloud information, region proposal information and centroid information of the completed point cloud are dimensionally spliced to obtain a spliced point cloud; The spliced point cloud is input into a segmentation network for instance segmentation, and 3D bounding box information of an instance target is output.

2. The feature enhancement based three-dimensional perception method of claim 1, wherein: The projection transformation of the foreground points of the original point cloud is as follows: ； In the formula, X is the original point cloud coordinate, that is Y is the pixel coordinate system coordinate obtained after projection transformation, that is ; Pi is the projection matrix of the i-th camera, R0 is the correction matrix of the 0-th camera, Rr2c is the rotation and translation matrix from the radar sensor coordinate system to the camera coordinate system.

3. The feature enhancement based three-dimensional perception method of claim 1, wherein: The object centroid position is found by using a voting mechanism to acquire the approximate centroid position of the object, comprising: The point cloud features of the truncated cone are extracted by using an encoder to generate a series of seed points, which is a process of gradually obtaining deep features from shallow features through down-sampling; The seed points are input to the voting layer, which first generates a 3+256 dimensional feature vector for each seed point's 256 dimensional feature through an mlp+bn layer. The network learns the spatial and feature offsets of the seed points to the object center point to obtain the center point by voting each point, i.e., the difference between the real value distance from the object center and the network output voting and network output voting is supervised. The clustering centers obtained by the voting layer are clustered to obtain the object centroid position.

4. The feature enhancement based three-dimensional perception method of claim 3, wherein: The encoder uses sequentially connected set abstraction (SA) layers and feature propagation (FP) layers to extract features of the points of the truncated cone.

5. The feature enhancement based three-dimensional perception method of claim 3, wherein: In the process of gradually obtaining deep features from shallow features by downsampling, a centroid-aware downsampling strategy is introduced, i.e., points closer to the centroid are assigned higher weights The formula is as follows: ； Wherein, f, b, l, r, u, d respectively represent the distance of a point to the 6 surfaces of the real bounding box, and the 6 surfaces are front, back, left, right, top and bottom.

6. The feature enhancement based three-dimensional perception method of claim 3, wherein: In the process of acquiring the approximate centroid position of the object by using the voting mechanism, a first detection head is set to supervise the position difference between the object center point and the predicted centroid.

7. The feature enhancement based three-dimensional perception method of claim 3, wherein: The adaptive ball query operation includes the following steps: Each obtained centroid position forms a center point of a ball, and after the radius is artificially set, the points in each ball are used as a local region for feature extraction; Wherein, the artificially set radius is adaptively set according to the depth of the centroid position obtained by the voting layer; since the radar sensor has the characteristics of near large and far small, the radius of the object farther away from the radar sensor is set smaller; the baseline standard of the radius is obtained by actual measurement, and the radius at any depth is calculated according to the principle of similar triangles.

8. The feature enhancement based three-dimensional perception method of claim 3, wherein: The point cloud information, region proposal information and centroid information of the completed point cloud are dimensionally spliced to obtain a spliced point cloud, wherein: The spliced point cloud includes eight dimensions, which are four dimensions of the point cloud information of the completed point cloud, three dimensions of the region proposal information and one dimension of the centroid information; The four dimensions of the point cloud information of the completed point cloud are respectively the x-axis coordinate, y-axis coordinate, z-axis coordinate and reflection intensity r of the corresponding point; Three dimensions of the region proposal information are mask information, category label and index label of the corresponding point respectively; the mask information indicates that the point belongs to a foreground point or a background point, which is determined by combining original point cloud, using projection transformation and boundary judgment; the category label indicates the category of the 2D bounding box to which the point belongs; and the index label indicates the number of 2D bounding boxes to which each point belongs; One dimension of the center information indicates the distance between the center obtained by the final clustering of the voting layer and the current point, which is used to provide additional information for the segmentation network, i.e., the closer the point is to the center, the more likely it is to be a point constituting the target object.

9. The feature enhancement based three-dimensional perception method of claim 5, wherein: The spliced point cloud is sent into the segmentation network for instance segmentation, and 3D bounding box information of the instance target is output, including: PointNet segmentation network is used to perform instance segmentation on the spliced point cloud in each truncated cone, and the instance segmentation is binary classification, i.e., each point belongs to an object point or a background point; After the object-related points are separated, a second detection head is input for 3D bounding box regression, wherein the parameters required to be regressed by the second detection head include an orientation angle, a 3D bounding box size, and an angle point loss obtained through an angle point position and a center position.

10. The feature enhancement based three-dimensional perception method of claim 9, wherein: The loss function of the 3D bounding box regression is realized in a weighted cumulative manner, including a centroid-aware down-sampling loss , an object center-aware loss , a bounding box regression loss; wherein: centroid-aware down-sampling loss for utilizing weight masks The fusion achieves the effect of implicitly combining into the network training in the loss function of the segmentation task, and the formula is as follows: ； In the formula, s i denotes the category of the prediction point, denotes the probability that the prediction point belongs to this category; Object center perception loss That is, the supervision is performed by the difference between the real value distance from the object center And the network output vote The formula is as follows: ； In the formula, The number of object surface voting points is represented, and a 3D space bias is learned for each voting point In order to make the points after moving closer to the centroid of the object, L1 loss is used to supervise the distance between the points after moving and the centroid of the object, that is, , ; Bounding box regression loss L Huber Including orientation angle loss L head 3D bounding box size loss L size Corner point loss L corner Three, orientation angle loss L head And 3D bounding box size loss L size The formulas of the three are as follows: ； wherein is the true value, is the predicted value, is the hyperparameter; corner loss L corner The formula is as follows: ； wherein and respectively represent the predicted and ground truth bounding box corners, represents the corners obtained by flipping the ground truth bounding box.

Images