Point cloud and image feature fusion method based on aerial view hybrid coding

Abstract

The invention discloses a point cloud and image feature fusion method and device based on aerial view hybrid coding. The method comprises the following steps: generating a cylindrical object coding score in an aerial view according to corresponding point cloud and image information in each cylindrical object; comparing the columnar body coding scores with a preset threshold value to obtain classifications to which the columnar bodies belong, generating columnar body features by adopting corresponding feature coding strategies for the columnar bodies in the classifications, and obtaining multi-modal aerial view features based on the columnar body features, according to the invention, the cylinder coding score is generated by analyzing the sparsity of the point cloud data in each cylinder, so that each cylinder in the space is helped to learn the corresponding feature, and the appropriate feature coding strategy is dynamically selected for the cylinders with different data sparsity degrees through the generated cylinder coding score. And outputting multi-modal aerial view features with higher discrimination capability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a method and device for fusing point cloud and image features based on bird's-eye view hybrid coding. Background Art

[0002] In recent years, with the rapid development of sensor technology, a large number of scientific researchers have devoted themselves to the research of technology based on the fusion perception of cameras and lidar. The key to this perception technology lies in the effective fusion of image and point cloud features. However, the completely different data representation forms have caused great difficulties in the fusion between these two features. The current mainstream fusion scheme is to project the image into the space where the point cloud is located, and then fuse the corresponding image and point cloud features in the voxel grid of the bird's-eye view, so as to generate comprehensive multi-modal bird's-eye view features. The targets in the bird's-eye view can effectively avoid the frequent target occlusion phenomenon in the perspective view. Therefore, there are advantages that other fusion schemes cannot match in performing advanced target perception on bird's-eye view features. However, when the image features are transformed into the bird's-eye view space, factors such as depth error and sensor calibration error will cause image feature calibration problems in the bird's-eye view space. In addition, the image contains both foreground pixels and background pixels. The traditional method uniformly transforms these pixels into the bird's-eye view space and fuses them with the corresponding point cloud features. This will not only cause feature confusion between the image and the point cloud, but also reduce the inference speed of the model due to redundant feature fusion in a large number of background areas, thereby affecting the detection performance of the model. Thus, it is extremely urgent to develop a fusion technology that can efficiently fuse image and point cloud features on the bird's-eye view. Summary of the Invention

[0003] The present invention aims to at least solve one of the technical problems existing in the prior art. To this end, in the first aspect of the present invention, a method for fusing point cloud and image features based on bird's-eye view hybrid coding is proposed, including: dividing the bird's-eye view space of the point cloud into a plurality of cylinders of the same size along a preset coordinate system; processing the image information, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point cloud in the cylinder diagram based on a preset bird's-eye view coding score generation algorithm to obtain the coding scores of each cylinder; using the bird's-eye view hybrid feature coding algorithm to compare the coding scores of each cylinder with a plurality of set thresholds respectively to obtain the classification to which each cylinder belongs, and adopting corresponding feature coding strategies for the cylinders in each classification to generate the features of each cylinder, and obtaining multi-modal bird's-eye view features based on the features of each cylinder.

[0004] Optionally, the method further includes: using a Swin Transformer image feature extraction network to extract multi-view image features of a pre-input image; extracting the multi-view image features through a feature pyramid network to obtain multi-scale image features.

[0005] Optionally, processing the image information, the distance between the columnar body and the sensor, the number of point clouds in the columnar body, and the local offset of the point clouds in the histogram for each columnar body according to a preset bird's-eye view encoding score generation algorithm to obtain the encoding score for each columnar body, including: processing the image information, the distance between the columnar body and the sensor, the number of point clouds in the columnar body, and the local offset of the point clouds in the histogram for each columnar body according to a preset bird's-eye view encoding score generation algorithm, and correspondingly obtaining the foreground feature of the columnar body, the distance feature of the columnar body, the point cloud number feature of the columnar body, and the point cloud offset feature of the columnar body; concatenating the foreground feature of the columnar body, the distance feature of the columnar body, the point cloud number feature of the columnar body, and the point cloud offset feature of the columnar body, and then generating the encoding score for each columnar body through convolution and activation functions.

[0006] Optionally, the classifications to which each columnar body belongs include background columnar bodies, dense columnar bodies, moderately sparse columnar bodies, and extremely sparse columnar bodies; adopting corresponding feature encoding strategies for the columnar bodies in each classification to generate the features of each columnar body, including: for the background columnar bodies, retaining the sparse features of the background columnar bodies as the features of the background columnar bodies; for the dense columnar bodies, learning the point cloud features of the dense columnar bodies to obtain the features of the dense columnar bodies; for the moderately sparse columnar bodies, fusing the multi-scale image features and point cloud features corresponding to the moderately sparse columnar bodies to obtain the features of the moderately sparse columnar bodies; for the extremely sparse columnar bodies, using the reference points distributed in the extremely sparse columnar bodies to learn the multi-scale image features nearby, generating the image features of the extremely sparse columnar bodies, and jointly obtaining the multi-modal bird's-eye view features by combining the features of the background columnar bodies, the features of the dense columnar bodies, the features of the moderately sparse columnar bodies, and the image features of the extremely sparse columnar bodies.

[0007] Optionally, learning the point cloud features of the dense columnar bodies to obtain the features of the dense columnar bodies includes: using the PointPillar algorithm to perform feature encoding on the dense columnar bodies to generate dense bird's-eye view features; obtaining the features of the dense columnar bodies through the two-dimensional index of the dense columnar bodies in the dense bird's-eye view features.

[0008] Optionally, fusing the multi-scale image features and point cloud features corresponding to the moderately sparse columnar bodies to obtain the features of the moderately sparse columnar bodies includes: using the bilinear interpolation algorithm to extract the point cloud features of the moderately sparse columnar bodies to obtain the point cloud features of the moderately sparse columnar bodies; using the point cloud features of the moderately sparse columnar bodies to query the multi-scale image features of the internal point set at the multi-view image features; adding the queried multi-scale image features and the point cloud features of the moderately sparse columnar bodies to obtain the features of the moderately sparse columnar bodies.

[0009] Optionally, learning multi-scale image features near the reference points distributed in the extremely sparse columnar body to generate extremely sparse columnar body image features includes: uniformly placing a plurality of reference points along the z-axis direction of the preset coordinate system and obtaining the features of each reference point; extracting the point cloud features of the extremely sparse columnar body as point cloud voxel query features; inputting the point cloud voxel query features, multi-scale image features and reference point features into the deformable attention encoder, and outputting the attention weights and bird's-eye view offsets corresponding to each reference point; for each reference point, adjusting the position of the reference point to the target area based on the bird's-eye view offset; weighted summing the multi-scale image features in the target area to obtain the multi-scale aggregation features corresponding to each reference point; weighted summing the multi-scale aggregation features of all reference points based on the attention weights to obtain the extremely sparse columnar body image features.

[0010] Optionally, obtaining the multi-modal bird's-eye view features from the various columnar body features includes: converting the feature channels of the columnar body features of each category into multi-modal columnar body features with the same number of channels through a linear layer; arranging the point cloud bird's-eye view features with the same number of channels according to their positions in the original point cloud bird's-eye view space to obtain the multi-modal bird's-eye view features.

[0011] Optionally, dividing the point cloud bird's-eye view space into a plurality of columnar bodies of the same size along the preset coordinate system includes: taking the position where the lidar is located as the coordinate origin, taking the front of the lidar as the x-axis, and establishing a lidar coordinate system according to the right-hand rule; uniformly dividing the preset point cloud bird's-eye view space in the point cloud space into a plurality of columnar bodies along the x-axis and y-axis at a fixed step size.

[0012] To achieve the above object, the present application also provides a point cloud and image feature fusion system based on bird's-eye view hybrid coding, including: a bird's-eye view division module for dividing the point cloud bird's-eye view space into a plurality of columnar bodies of the same size along the preset coordinate system; a bird's-eye view coding score generation module for processing the image information, the distance between the columnar body and the sensor, the number of point clouds in the columnar body, and the local offset of the point cloud in the columnar diagram in each columnar body based on a preset bird's-eye view coding score generation algorithm to obtain the coding score of each columnar body; a bird's-eye view hybrid feature coding module for comparing the coding scores of each columnar body with a plurality of set thresholds respectively by using a bird's-eye view hybrid feature coding algorithm to obtain the classification to which each columnar body belongs, and adopting corresponding feature coding strategies for the columnar bodies in each classification to generate the features of each columnar body, and obtaining multi-modal bird's-eye view features based on the features of each columnar body.

[0013] An embodiment of the present invention provides a method and apparatus for fusing point cloud and image features based on bird's-eye view hybrid coding. Compared with the prior art, the beneficial effects are as follows: The bird's-eye view space of the point cloud is divided into multiple cylinders of the same size along a preset coordinate system; based on a preset bird's-eye view coding score generation algorithm, the image information in each cylinder, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point cloud in the histogram are processed to obtain the coding score of each cylinder; the bird's-eye view hybrid feature coding algorithm is used to compare the coding scores of each cylinder with multiple set thresholds respectively to obtain the classification to which each cylinder belongs, and a corresponding feature coding strategy is adopted for the cylinders in each classification to generate the features of each cylinder. Based on the features of each cylinder, multi-modal bird's-eye view features are obtained. The present application can use the preset bird's-eye view coding score generation algorithm to analyze the sparsity of the point cloud data in each cylinder to generate the coding score of the cylinder, so as to help each cylinder in the space learn the corresponding features. The bird's-eye view hybrid feature coding algorithm is used to dynamically select a suitable feature coding strategy for the cylinders with different data sparsity degrees based on the generated coding scores of the cylinders, so as to effectively fuse the corresponding image and point cloud information in the bird's-eye view, and then output more discriminative multi-modal bird's-eye view features. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0015] Figure 1 It is a flowchart of a method for fusing point cloud and image features based on bird's-eye view hybrid coding provided by an embodiment of the present invention;

[0016] Figure 2 It is a schematic flow diagram of a bird's-eye view coding score generation algorithm for a method for fusing point cloud and image features based on bird's-eye view hybrid coding provided by an embodiment of the present invention;

[0017] Figure 3 It is a schematic structural diagram of a bird's-eye view hybrid feature coding algorithm for a method for fusing point cloud and image features based on bird's-eye view hybrid coding provided by an embodiment of the present invention;

[0018] Figure 4 It is a schematic block diagram of a system for fusing point cloud and image features based on bird's-eye view hybrid coding provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] This specification provides method operation steps as described in the embodiments or flowcharts, but based on routine or non-creative labor, it may include more or fewer operation steps. When actually executed in a system or server product, it can be executed in the order shown in the embodiments or the drawings or in parallel (for example, in an environment of parallel processors or multi-threaded processing).

[0021] To overcome the deficiencies of the prior art, the present invention provides a method for fusing point cloud and image features with hybrid feature encoding on a bird's-eye view. This method can be executed by a processor of a service or a terminal. The processor first divides the space where the point cloud is located into a number of columnar bodies of equal size, and then jointly generates bird's-eye view columnar body encoding scores based on the distance between the centers of these columnar bodies and the sensor, as well as their corresponding image features, the number of point clouds they contain, and the local position offsets of the internal point clouds. Then, after comparing the columnar body scores with multiple set thresholds, all the columnar bodies in the bird's-eye view are divided into four categories: background columnar bodies, dense columnar bodies, moderately sparse columnar bodies, and extremely sparse columnar bodies. For background columnar bodies, the processor does not need to perform any processing and only needs to maintain the sparsity inside; for dense columnar bodies, since the internal point clouds can provide rich target information, the processor can directly learn the features of the point clouds therein and use them as the features of the columnar body where they are located; for moderately sparse columnar bodies, the sparse internal point clouds are not sufficient to generate effective target features. At this time, the processor needs to combine the image information corresponding to the columnar body to enrich the features of the current columnar body; for extremely sparse columnar bodies, the internal points can no longer provide effective target information. At this time, the processor needs to use the reference points distributed inside the columnar body to learn the image features nearby, so as to extract relevant target information to generate columnar body image features. According to the above classification, the processor respectively adopts targeted feature encoding strategies to learn the point cloud and image features in various columnar bodies, and then combines these columnar bodies and their features together to form a comprehensive bird's-eye view feature. This bird's-eye view feature is then converted into a consistent feature channel through a two-dimensional convolution for the features of each columnar body it contains. The fused bird's-eye view feature can be further combined with other region proposal networks to meet high-level task requirements such as predicting target bounding boxes.

[0022] Reference Figure 1 , the method for fusing point cloud and image features with hybrid encoding based on a bird's-eye view may include:

[0023] Step S10 may include the following execution process:

[0024] S101. Take the position where the lidar is located as the coordinate origin, take the front direction of the lidar as the x-axis, and establish a lidar coordinate system according to the right-hand rule.

[0025] S102. Uniformly divide the preset bird's-eye view space of the point cloud in the point cloud space into a number of cylinders along the x-axis and y-axis at a fixed step size.

[0026] In implementation, the processor takes the position where the lidar is located as the coordinate origin, takes the front direction of the lidar as the x-axis, and then establishes a lidar coordinate system according to the right-hand rule. The processor then uniformly divides the preset area range in the point cloud space into a number of cylinder regions along the x-axis and y-axis at a step size of 0.75 m. To facilitate subsequent fusion of image and point cloud features within the cylinder, the bird's-eye view encoding score generation module first simply encodes the point cloud in the cylinder using the cylinder feature encoding strategy proposed by PointPillar in each cylinder, thereby generating initial point cloud cylinder features.

[0027] In an embodiment of the present application, the method for fusing point cloud and image features based on bird's-eye view hybrid encoding may further include:

[0028] Use the SwinTransformer image feature extraction network to extract multi-view image features of the pre-input image.

[0029] Extract multi-view image features through a feature pyramid network to obtain multi-scale image features.

[0030] In the specific execution process, the processor uses SwinTransformer as the image feature extraction network to generate multi-view image features. where, H I and W I are the width and height of the image respectively, and C I is the feature channel number of the image. These image features are then passed through a feature pyramid network to generate three high-level image features with sizes of 1 / 4, 1 / 8, and 1 / 16 of the original image size respectively. The feature channels of these high-level image features are all set to 256, and all the image features will be further fused with the point cloud features within the cylinder.

[0031] S20. Divide the bird's-eye view space of the point cloud into a number of cylinders with the same size along the preset coordinate system.

[0032] Process the image information within each column, the distance between the column and the sensor, the number of point clouds in the column, and the local offset of the point clouds in the histogram based on a preset bird's-eye view encoding score generation algorithm to obtain the encoding score for each column.

[0033] In an embodiment of the present application, step S20 may include the following execution process:

[0034] S201. Process the image information within the column, the distance between the column and the sensor, the number of point clouds in the column, and the local offset of the point clouds in the histogram based on a preset bird's-eye view encoding score generation algorithm to correspondingly obtain the column foreground feature, column distance feature, column point cloud number feature, and column point cloud offset feature.

[0035] S202. Concatenate the column foreground feature, column distance feature, column point cloud number feature, and column point cloud offset feature, and then generate the encoding score for each column through convolution and an activation function.

[0036] Reference Figure 2 , in the specific execution process, after obtaining the multi-scale image features, the bird's-eye view encoding score generation algorithm then generates four features for the image information within the column, the distance between the column and the sensor, the number of point clouds in the column, and the local offset of the point clouds in the histogram respectively. The four features include the column foreground feature, column distance feature, column point cloud number feature, and column point cloud offset feature. To generate the column foreground feature, the bird's-eye view encoding score generation algorithm evenly places 6 points in each column to extract the image features near their image projection positions. For the point clouds that can be projected onto multiple camera views, the bird's-eye view encoding score generation algorithm adds up the image features at all projection positions and performs normalization processing to generate the column image feature.

[0037] The bird's-eye view encoding score generation algorithm places these columns according to their initial positions in the bird's-eye view to generate the bird's-eye view image feature, and this image feature then passes through a foreground prediction module to generate the column foreground feature that can reflect the possibility of it being inside the target. Where H and W respectively represent the height and width of the bird's-eye view space.

[0038] The bird's-eye view encoding score generation algorithm calculates the distance d from the center of each column to the sensor. pillar , and then generates the column distance feature F according to formula (1). distan. :

[0039] F distan. = CBR(1 / log 10 (d pillar + 0.5)) (1)

[0040] Among them, CBR consists of a convolutional layer, a normalization layer, and a ReLu activation function.

[0041] The bird's-eye view encoding score generation algorithm counts the number of point clouds N contained in each cylinder p , and then generates the cylinder point cloud number feature F according to formula (2) quant. :

[0042] F quant. = CBR(log 10 (N p + 0.5)) (2)

[0043] The bird's-eye view encoding score generation algorithm finds the point clouds contained in each cylinder, and then calculates the local position offsets of these points to the center of the cylinder respectively Among them, (xjo, yjo, zjo) represents the center of the j-th cylinder, {p i = (x i , y i , z i )|i = 1,…, N} represents the point set in space, and M and N represent the number of non-empty cylinders and point clouds in the scene respectively. Finally, the local position offsets of these cylinders are used to obtain the cylinder point cloud offset feature through formula (3):

[0044] F off. = CBR(MLP({Δp i |p i in Pillar j})) (3)

[0045] Among them, MLP represents the multi-layer perception module proposed in PointNet, and p i in Pillar j means that the point p i is located in the j-th cylinder.

[0046] After obtaining these four cylinder features, the bird's-eye view encoding score generation algorithm cascades these features along the feature channels of the cylinders, and then passes the concatenated features through a CBR module and a sigmoid function in sequence, thereby generating the cylinder encoding score.

[0047] S30. Use the bird's-eye view hybrid feature encoding algorithm to compare each cylinder encoding score with multiple set thresholds respectively, obtain the classification to which each cylinder belongs, and adopt corresponding feature encoding strategies for the cylinders in each classification to generate each cylinder feature, and obtain the multi-modal bird's-eye view feature based on each cylinder feature.

[0048] In an embodiment of the present application, the classifications to which each columnar body belongs include background columnar bodies, dense columnar bodies, moderately sparse columnar bodies, and extremely sparse columnar bodies. Step S30 may include the following execution process:

[0049] S301. For the background columnar bodies, retain the sparse features of the background columnar bodies as the background columnar body features.

[0050] Reference Figure 3 , the bird's-eye view hybrid feature encoding algorithm first classifies the columnar bodies according to the encoding scores of each columnar body, and then adopts a targeted feature encoding strategy for each type of columnar body to generate columnar body features. After obtaining all the columnar body features, these features are combined together and sent into a convolutional layer to obtain the final bird's-eye view feature.

[0051] In the specific execution process, the range of the columnar body encoding scores generated by the bird's-eye view hybrid feature encoding algorithm is between 0 and 1. At the same time, the bird's-eye view hybrid feature encoding algorithm sets three thresholds of 0.9, 0.4, and 0.35 to compare with these columnar body encoding scores, so as to divide the columnar bodies in the space into four categories, including background columnar bodies, dense columnar bodies, moderately sparse columnar bodies, and extremely sparse columnar bodies. The background columnar bodies contain a large amount of background point clouds, and their corresponding encoding scores are generally lower than 0.35. For these columnar bodies, no additional feature extraction and fusion operations are required, and only the sparsity at the positions of these columnar bodies needs to be retained.

[0052] S302. For the dense columnar bodies, learn the point cloud features of the dense columnar bodies to obtain the dense columnar body features.

[0053] In an embodiment of the present application, step S302 may include the following execution process:

[0054] S3021. Adopt the PointPillar algorithm to perform feature encoding on the dense columnar bodies to generate dense bird's-eye view features.

[0055] S3022. Obtain the dense columnar body features through the two-dimensional index of the dense columnar bodies in the dense bird's-eye view features.

[0056] Continue to refer to Figure 3 , in the specific execution process, the encoding scores of the dense columnar bodies should be greater than 0.9. These columnar bodies contain rich point cloud data. Therefore, directly learning these point cloud features can fully represent the target information therein. Correspondingly, the bird's-eye view hybrid feature encoding algorithm first uses PointPillar to perform simple feature encoding on the columnar bodies in the space to generate dense bird's-eye view features F dense, and then the corresponding dense cylinder feature is obtained through the two-dimensional index of the dense cylinder in the bird's-eye view.

[0057] S303. For medium-sparse cylinders, fuse the multi-scale image features and point cloud features corresponding to the medium-sparse cylinders to obtain medium-sparse cylinder features.

[0058] In an embodiment of the present application, step S303 may include the following execution process:

[0059] S3031. Use the bilinear interpolation algorithm to extract the point cloud features of the medium-sparse cylinders to obtain medium-sparse cylinder point cloud features.

[0060] S3032. Use the medium-sparse cylinder point cloud features to query the multi-scale image features of the point set inside at the multi-view image features.

[0061] S3033. Add the queried multi-scale image features and the medium-sparse cylinder point cloud features to obtain medium-sparse cylinder features.

[0062] Continue to refer to Figure 3 , where the encoding score of the medium-sparse cylinders should be between 0.9 and 0.4. Although these cylinders contain a certain number of point clouds, these point clouds are difficult to completely describe the spatial distribution state of the target. Therefore, the module uses bilinear interpolation according to formula (4) First, extract the corresponding cylinder point cloud features in the bird's-eye view feature F dense Then use the obtained cylinder features to query the point set inside the cylinder The set of image features at the multi-view image projection, and finally add the queried cylinder image features and the cylinder point cloud features, so as to help the medium-sparse cylinders obtain the final features:

[0063]

[0064] Here, represents the feature of the jth cylinder classified as medium-sparse, v represents the view index of the image, p i represents the point located in the jth cylinder, V hit represents p i The number of images that can be effectively projected on the multi-view image, K represents the number of point clouds contained in the jth cylinder, and MHAttn represents the multi-head attention proposed in the Transformer structure.

[0065] S304. For extremely sparse cylinders, use the reference points distributed within the extremely sparse cylinders to learn the nearby multi-scale image features, generate the extremely sparse cylinder image features, and use the background cylinder features, dense cylinder features, moderately sparse cylinder features, and extremely sparse cylinder image features as the bird's-eye view features of each point cloud.

[0066] Specifically, step S304 may include the following execution process:

[0067] S3041. Uniformly place multiple reference points along the z-axis direction of the preset coordinate system, and obtain the features of each reference point.

[0068] S3042. Extract the point cloud features of the extremely sparse cylinders as the point cloud voxel query features.

[0069] S3043. Input the point cloud voxel query features, multi-scale image features, and reference point features into the deformable attention encoder, and output the attention weights and bird's-eye view offsets corresponding to each reference point.

[0070] S3044. For each reference point, adjust the reference point position to the target area based on the bird's-eye view offset.

[0071] S3045. Weightedly sum the multi-scale image features in the target area to obtain the multi-scale aggregation features corresponding to each reference point.

[0072] S3046. Weightedly sum the multi-scale aggregation features of all reference points based on the attention weights to obtain the extremely sparse cylinder image features.

[0073] Continue to refer to Figure 3 , where the encoding scores of the extremely sparse cylinders should be between 0.4 and 0.35. These cylinders contain extremely sparse point clouds, which are difficult to provide effective target features. At this time, the corresponding image features of these cylinders should be fully utilized to help these cylinders generate effective features. Therefore, the bird's-eye view hybrid feature encoding module uniformly places 6 reference points along the z-axis direction of the point cloud coordinate system in these cylinders, then defines a set of cylinder query features for these cylinders, and then uses the cylinder query features to generate the corresponding attention weights and bird's-eye view offsets for these six points respectively according to the deformable attention. Then, useful image target information is aggregated for these six reference points in the above-obtained three multi-scale image features respectively. Finally, these six aggregated features are combined according to the corresponding attention to generate the final cylinder features for the extremely sparse cylinders.

[0074] In an embodiment of the present application, step S30 may further include the following execution process:

[0075] S305. Pass the feature channels of each bird's-eye view feature of the point cloud through a linear layer to obtain point cloud bird's-eye view features with the same number of channels.

[0076] S306. Arrange the point cloud bird's-eye view features with the same number of channels according to their positions in the original point cloud bird's-eye view space to obtain multi-modal bird's-eye view features.

[0077] After encoding the point cloud and image features in all cylinders, the feature channels of these cylinder features are lifted to the same number of channels through a linear layer. The bird's-eye view mixed feature encoding algorithm then re-positions these cylinders according to their positions in the bird's-eye view, and the re-positioned cylinder features are combined again to form multi-modal bird's-eye view features. The obtained bird's-eye view features are then fed into a two-dimensional convolution with a convolution kernel of 1 to generate the final multi-modal bird's-eye view features. The generated bird's-eye view features can be further combined with a region proposal network to meet the requirements of advanced tasks such as object detection.

[0078] Reference Figure 4 , based on the above embodiments, the present application also provides a point cloud and image feature fusion system based on bird's-eye view hybrid encoding to solve the same technical problems as the method embodiments. The point cloud and image feature fusion system 100 based on bird's-eye view hybrid encoding may include a bird's-eye view partitioning module 101. Among them, the bird's-eye view partitioning module 101 is used to divide the point cloud bird's-eye view space along a preset coordinate system into multiple cylinders of the same size. The bird's-eye view encoding score generation module 102 is used to process the image information, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point clouds in the histogram in each cylinder based on a preset bird's-eye view encoding score generation algorithm to obtain the encoding scores of each cylinder. The bird's-eye view mixed feature encoding module 103 is used to compare the encoding scores of each cylinder with multiple set thresholds respectively by using the bird's-eye view mixed feature encoding algorithm to obtain the classification to which each cylinder belongs, and adopt corresponding feature encoding strategies for the cylinders in each classification to generate the features of each cylinder, and obtain multi-modal bird's-eye view features based on the features of each cylinder.

[0079] Specifically, the technical solutions adopted by the device embodiments of the present invention to solve its technical problems are as follows:

[0080] Construct a point cloud and image feature fusion system, and the point cloud and image feature fusion system includes a bird's-eye view encoding score generation module and a bird's-eye view mixed feature encoding module.

[0081] The bird's-eye view encoding score generation module comprehensively analyzes the corresponding image features, point cloud features, and position information of the cylinder in each cylinder in the bird's-eye view space, so as to generate the corresponding encoding score of the cylinder.

[0082] The bird's-eye view hybrid feature encoding module classifies the histograms in the bird's-eye view according to the columnar body encoding scores generated by the bird's-eye view encoding score generation module, and then adopts a targeted feature encoding strategy for each type of columnar body, so as to generate columnar body features that can fully represent the target information.

[0083] The bird's-eye view hybrid feature encoding module relocates the obtained columnar body features according to the positions of the respective columnar bodies in the bird's-eye view, so as to construct a bird's-eye view feature based on multi-modal feature fusion. Finally, the obtained bird's-eye view feature is generated into the final bird's-eye view feature through a two-dimensional convolution with a convolution kernel size of 1.

[0084] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0085] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

[0086] The above is only the preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for fusing point cloud and image features based on mixed coding of bird's-eye view, characterized in that Including: Dividing the point cloud bird's-eye view space into multiple cylinders of the same size along a preset coordinate system; Processing the image information, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point clouds in the histogram within each cylinder based on a preset bird's-eye view encoding score generation algorithm to obtain the encoding score of each cylinder; Using the bird's-eye view hybrid feature encoding algorithm to compare the encoding scores of each cylinder with multiple set thresholds respectively to obtain the classification to which each cylinder belongs, and adopting corresponding feature encoding strategies for the cylinders in each classification to generate the features of each cylinder, and obtaining the multi-modal bird's-eye view features based on the features of each cylinder.

2. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 1, wherein, The method further includes: Using the Swin Transformer image feature extraction network to extract the multi-view image features of the pre-input image; Extracting the multi-view image features through the feature pyramid network to obtain the multi-scale image features.

3. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 1, wherein The processing of the image information, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point clouds in the histogram within each cylinder based on a preset bird's-eye view encoding score generation algorithm to obtain the encoding score of each cylinder includes: Processing the image information, the distance between the cylinder and the sensor, the number of point clouds in the cylinder, and the local offset of the point clouds in the histogram within the cylinder based on a preset bird's-eye view encoding score generation algorithm to correspondingly obtain the foreground feature of the cylinder, the distance feature of the cylinder, the point cloud number feature of the cylinder, and the point cloud offset feature of the cylinder; Cascading the foreground feature of the cylinder, the distance feature of the cylinder, the point cloud number feature of the cylinder, and the point cloud offset feature of the cylinder, and then generating the encoding score of each cylinder through convolution and activation functions.

4. The method for fusing point cloud and image features based on mixed coding of bird's-eye view as claimed in claim 2, wherein The classifications to which each cylinder belongs include background cylinders, dense cylinders, moderately sparse cylinders, and extremely sparse cylinders; The adopting corresponding feature encoding strategies for the cylinders in each classification to generate the features of each cylinder includes: For background cylinders, retaining the sparse features of the background cylinders as the background cylinder features; For dense cylinders, learning the point cloud features of the dense cylinders to obtain the dense cylinder features; For moderately sparse cylinders, fusing the corresponding multi-scale image features and point cloud features of the moderately sparse cylinders to obtain the moderately sparse cylinder features; For extremely sparse cylinders, using the reference points distributed within the extremely sparse cylinders to learn the nearby multi-scale image features to generate the extremely sparse cylinder image features, and combining the background cylinder features, the dense cylinder features, the moderately sparse cylinder features, and the extremely sparse cylinder features together to generate the point cloud bird's-eye view features.

5. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 4, wherein The learning the point cloud features of the dense cylinders to obtain the dense cylinder features includes: Adopting the PointPillar algorithm to perform feature encoding on the dense cylinders to generate the dense bird's-eye view features; Obtaining the dense cylinder features through the two-dimensional index of the dense cylinders in the dense bird's-eye view features.

6. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 4, wherein The obtaining the moderately sparse cylinder features based on the multi-scale image features corresponding to the moderately sparse cylinders includes: Using the bilinear interpolation algorithm to extract the point cloud features of the moderately sparse cylinders to obtain the moderately sparse cylinder point cloud features; Query the point set inside the medium-sparse columnar point cloud feature at the multi-scale image features of the multi-view image features; Add the queried multi-scale image features and the medium-sparse columnar point cloud features to obtain medium-sparse columnar features.

7. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 4, wherein The method of using the reference points distributed inside the extremely sparse columnar body to learn the nearby multi-scale image features to generate extremely sparse columnar image features includes: Uniformly place multiple reference points along the z-axis direction of the preset coordinate system, and obtain the features of each reference point; Extract the point cloud features of the extremely sparse columnar body as the point cloud voxel query features; Input the point cloud voxel query features, multi-scale image features and reference point features into the deformable attention encoder, and output the attention weights and bird's-eye view offsets corresponding to each reference point; For each reference point, adjust the reference point position to the target area based on the bird's-eye view offset; Weightedly sum the multi-scale image features in the target area to obtain the multi-scale aggregation features corresponding to each reference point; Based on the attention weights, weightedly sum the multi-scale aggregation features of all reference points to obtain extremely sparse columnar image features.

8. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to any one of claims 1-7, characterized in that The method of obtaining the multi-modal bird's-eye view features based on each columnar feature includes: Convert the feature channels of the columnar features of each category into multi-modal columnar features with the same number of channels through a linear layer; Arrange the point cloud bird's-eye view features with the same number of channels according to their positions in the original point cloud bird's-eye view space to obtain multi-modal bird's-eye view features.

9. The method for fusing point cloud and image features based on bird's-eye view hybrid coding according to claim 1, wherein The method of dividing the point cloud bird's-eye view space into several columnar bodies of the same size along the preset coordinate system includes: Take the position of the lidar as the coordinate origin, take the front of the lidar as the x-axis, and establish a lidar coordinate system according to the right-hand rule; Uniformly divide the preset point cloud bird's-eye view space in the point cloud space into several columnar bodies along the x-axis and y-axis at a fixed step size.

10. A point cloud and image feature fusion system based on bird's-eye view hybrid coding, characterized in that, It includes: A bird's-eye view division module for dividing the point cloud bird's-eye view space into multiple columnar bodies of the same size along the preset coordinate system; A bird's-eye view encoding score generation module for processing the image information, the distance between the columnar body and the sensor, the number of point clouds in the columnar body, and the local offset of the point clouds in the columnar diagram in the columnar body based on a preset bird's-eye view encoding score generation algorithm to obtain the encoding scores of each columnar body; A bird's-eye view mixed feature encoding module for using a bird's-eye view mixed feature encoding algorithm to compare the encoding scores of each columnar body with multiple set thresholds respectively to obtain the classification to which each columnar body belongs, and adopting corresponding feature encoding strategies for the columnar bodies in each classification to generate the features of each columnar body, and obtaining multi-modal bird's-eye view features based on the features of each columnar body.

Images