A feature fusion and environment perception method, device, equipment, medium and product

Abstract

This application provides a feature fusion and environmental perception method, apparatus, device, medium, and product. According to one example of this application, the method may include: acquiring multiple spatial units in a three-dimensional space, each spatial unit's multimodal features including visual features and point cloud features; for each spatial unit, generating geometric perception messages of neighboring units based on the relative geometric information between the spatial unit and its neighboring units; updating the multimodal features of the spatial unit based on the geometric perception messages of the neighboring units to generate fused features. By generating and aggregating geometric perception messages using relative geometric information in three-dimensional space, the relative geometric constraints between spatial units remain unchanged. This allows for accurate fusion of multimodal features in three-dimensional space without projecting visual features and point cloud features onto a two-dimensional plane for fusion, thereby improving the accuracy of point cloud data and visual data fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobiles, and in particular to a feature fusion and environment perception method, device, equipment, medium and product. BACKGROUND

[0002] In the field of intelligent driving environment perception, vehicles are usually equipped with multiple sensors to obtain features of different modalities. These multi-modal features each carry complementary geometric and semantic clues, and fusion of them can significantly improve overall perception accuracy.

[0003] In related technologies, the fusion method is usually to align and splice multi-modal features in a two-dimensional plane. However, when the vehicle pitch, roll, etc. occurs, the two-dimensional plane will tilt with the vehicle, while the objects in the scene do not move in the world coordinate system, resulting in misalignment of the originally aligned multi-modal features, and low fusion accuracy. SUMMARY

[0004] To overcome the problems in the related art, the present application provides a feature fusion and environment perception method, device, equipment, medium and product.

[0005] According to a first aspect of any one of the embodiments of the present application, a feature fusion method is provided, the method comprising:

[0006] obtaining a plurality of spatial units in a three-dimensional space, the multi-modal features of each spatial unit including visual features and point cloud features;

[0007] For each spatial unit, based on the relative geometric information between the spatial unit and the adjacent units, generating a geometric perception message of the adjacent units, the geometric perception message being used to reflect the spatial geometric constraints of the adjacent units relative to the spatial unit, the adjacent units being the spatial units that satisfy a preset geometric relationship with the spatial unit in the three-dimensional space;

[0008] Based on the geometric perception message of the adjacent units, updating the multi-modal features of the spatial unit to generate fusion features.

[0009] According to a second aspect of any one of the embodiments of the present application, an environment perception method is provided, the method comprising:

[0010] obtaining fusion features by the feature fusion method according to any one of the embodiments of the present application;

[0011] mapping the fusion features to a BEV plane to generate a BEV feature map;

[0012] performing an environment perception task based on the BEV feature map to generate an environment perception result.

[0013] According to a third aspect of any of the embodiments of the present application, there is provided a feature fusion device, the device comprising:

[0014] a unit obtaining module configured to obtain a plurality of spatial units in a three-dimensional space, a multi-modal feature of each spatial unit comprising a visual feature and a point cloud feature;

[0015] a generating module configured to, for each spatial unit, generate a geometric perception message of a neighboring unit based on relative geometric information between the spatial unit and the neighboring unit, the geometric perception message being used to reflect a spatial geometric constraint of the neighboring unit relative to the spatial unit, the neighboring unit being a spatial unit that satisfies a preset geometric relationship with the spatial unit in the three-dimensional space;

[0016] a fusion module configured to update the multi-modal feature of the spatial unit based on the geometric perception message of the neighboring unit, and generate a fused feature.

[0017] According to a fourth aspect of any of the embodiments of the present application, there is provided an environment perception device, the device comprising:

[0018] a feature obtaining module configured to obtain a fused feature by using the feature fusion method according to any of the embodiments of the present application;

[0019] a mapping module configured to map the fused feature to a BEV plane, and generate a BEV feature map;

[0020] a perception module configured to perform an environment perception task based on the BEV feature map, and generate an environment perception result.

[0021] According to a fifth aspect of any of the embodiments of the present application, there is provided an electronic device, comprising:

[0022] a processor;

[0023] a memory for storing processor-executable instructions;

[0024] wherein the processor implements the method according to any of the embodiments of the present application by running the executable instructions.

[0025] According to a sixth aspect of any of the embodiments of the present application, there is provided a computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, implement the method according to any of the embodiments of the present application.

[0026] According to a seventh aspect of any of the embodiments of the present application, there is provided a computer program product having stored thereon a computer program / instructions, which, when executed by a processor, implement the method according to any of the embodiments of the present application.

[0027] The technical scheme provided in the application can have the following beneficial effects.

[0028] According to the above embodiment, by obtaining a plurality of space units in a three-dimensional space, the multi-modal features of each space unit include visual features and point cloud features, for each space unit, based on the relative geometric information between the space unit and the adjacent unit, the geometric perception message of the adjacent unit is generated, the multi-modal features of the space unit are updated based on the geometric perception message of the adjacent unit, the fusion features are generated, by generating and aggregating the geometric perception message in the three-dimensional space using the relative geometric information, the relative geometric constraint between the space units remains unchanged, the multi-modal features can be accurately fused in the three-dimensional space, without projecting the visual features and the point cloud features into a two-dimensional plane for fusion, avoiding the projection misalignment caused by the complex attitude changes such as vehicle pitch, roll or yaw, thereby improving the fusion accuracy of the point cloud data and the visual data.

[0029] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are incorporated into the specification and constitute a part of the application, show embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0031] Figure 1 is a flowchart of a feature fusion method according to an exemplary embodiment of the application;

[0032] Figure 2 is a schematic diagram of a graph structure data according to an exemplary embodiment of the application;

[0033] Figure 3 is a flowchart of an environment perception method according to an exemplary embodiment of the application;

[0034] Figure 4 is a flowchart of a feature fusion and perception method according to an exemplary embodiment of the application;

[0035] Figure 5 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the application;

[0036] Figure 6 is a block diagram of a feature fusion device according to an exemplary embodiment of the application;

[0037] Figure 7 is a block diagram of an environment perception device according to an exemplary embodiment of the application. DETAILED DESCRIPTION

[0038] The detailed description set forth below describes exemplary embodiments that represent the best currently known embodiments of the application. The description of the exemplary embodiments is not intended, unless otherwise indicated, to limit the scope of the application. Rather, the description is intended to present various aspects of the application in the context of the exemplary embodiments. The exemplary embodiments described below are not necessarily to be considered as limitations of the scope of the application. Contrarily, they are to be considered as examples of apparatuses and methods consistent with some aspects of the application as detailed in the appended claims.

[0039] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0040] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one type of information from another. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present application. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining".

[0041] In the related art, the feature fusion method is usually to project multi-modal features into a two-dimensional plane for alignment and splicing. For example, both visual features and point cloud features can be projected into a Bird's Eye View (BEV) plane with the vehicle body coordinate system as the reference, and then the alignment and fusion of features are performed in the plane.

[0042] However, this method has a significant defect. When the vehicle undergoes a pitch, roll or other attitude change, the two-dimensional plane will tilt with the vehicle. Since the objects in the scene do not move in the world coordinate system, the multi-modal features originally aligned at the same pixel or grid position are misaligned, resulting in low fusion accuracy. This misalignment directly affects the accuracy of subsequent target detection, recognition and tracking tasks, thereby reducing the performance and reliability of the entire perception system.

[0043] To solve the above problems, the present application provides a feature fusion method. In order to further illustrate the present application, the following embodiments are provided:

[0044] Please refer to Figure 1 , Figure 1is a flowchart of a feature fusion method according to an exemplary embodiment of the present application. The feature fusion method can be exemplarily executed by a perception system, which can be applied to a vehicle, and can also be applied to a server, a cluster server, a cloud server, and the like. The environment perception method can also be executed by other systems or devices in different application scenarios, which is not limited in the embodiments of the present application.

[0045] As shown in Figure 1 , the feature fusion method can include the following steps:

[0046] Step 101: Obtain a plurality of spatial units in a three-dimensional space, and the multi-modal features of each spatial unit include visual features and point cloud features.

[0047] In this step, the perception system can obtain multi-modal environment data by using vehicle-mounted sensors. The vehicle-mounted sensors can include three-dimensional sensors and image sensors, and the three-dimensional sensors can be lidars, and the image sensors can be cameras. The environment data can be multi-modal data, which can include two-dimensional visual data and three-dimensional point cloud data.

[0048] The collected visual data and point cloud data can be preprocessed, including data cleaning, denoising, and preliminary feature extraction, to ensure the accuracy of subsequent processing. The preprocessed visual data and point cloud data are mapped into a three-dimensional space to form a plurality of spatial units. The spatial unit is a basic unit for representing perception data in a three-dimensional space, which can correspond to a point or a region.

[0049] The multi-modal features are joint feature vectors bound to the spatial units, and the multi-modal features of each spatial unit can include visual features and point cloud features. The visual features can be extracted from the visual data by image processing algorithms, which can be texture, RGB mean and variance, etc. The point cloud features can be directly obtained from the point cloud data, which can be distance, reflection intensity, etc.

[0050] In an embodiment, when obtaining a plurality of spatial units in a three-dimensional space, the point cloud data and visual data of the environment around the vehicle can be obtained; each data point in the point cloud data is mapped to a spatial unit; based on the point cloud data, the point cloud features of each data point are extracted; based on the spatial position of each data point, the projection area of the data point in the visual data is determined, and the projection area is feature extracted to obtain the visual features of the data point; the visual features and point cloud features of each data point are associated with the corresponding spatial unit to obtain the multi-modal features of each spatial unit.

[0051] The projection area is a two-dimensional image area corresponding to the data point in the visual data, which is used to extract visual features and associate with the point cloud features of the data point.

[0052] Exemplarily, the point cloud data and visual data of the environment around the vehicle can be acquired through a camera and a laser radar. The camera collects high-resolution two-dimensional image data to obtain visual data, which can provide rich texture, color and semantic information, such as the appearance features of traffic signs, lane lines, pedestrians, vehicles and the like. The laser radar emits laser beams and receives reflected light to generate accurate point cloud data, which can reflect the accurate geometric distance and spatial profile of the environment, and clearly depict the shape and position of the vehicle, obstacles and road and the like.

[0053] The collected visual data and point cloud data can be preprocessed to ensure the accuracy of subsequent processing. For the visual data, image processing algorithms such as histogram equalization and bilateral filtering can be used for image enhancement, denoising, color correction and other preliminary processing, and deep learning methods such as convolutional neural networks can also be used for feature extraction.

[0054] For the point cloud data, filtering, downsampling and ground removal and other processing can be performed to reduce the amount of calculation and retain key features. Each data point in the preprocessed point cloud data can be mapped to a spatial unit, and each spatial unit corresponds to a three-dimensional coordinate point.

[0055] Deep learning methods such as PointNet and PointCNN can be used to extract point cloud features of each data point based on the point cloud data. The point cloud features can include reflection intensity and other information.

[0056] The three-dimensional data point can be projected onto a two-dimensional image plane based on the spatial position of each data point through the camera's internal and external parameter matrix to determine its projection area in the visual data. Feature extraction methods such as Scale-invariant feature transform (SIFT), Oriented FAST and Rotated BRIEF (ORB) can be used to extract features from the projection area to obtain the visual features of the data point.

[0057] After extracting the visual features and point cloud features corresponding to each data point, the visual features and point cloud features corresponding to the data point are associated with the spatial unit corresponding to the data point.

[0058] As described above, by mapping the point cloud data into spatial units in three-dimensional space and combining visual data for feature extraction and association, multi-modal features containing rich geometric and semantic information can be generated. This construction method can ensure that the multi-modal features have high quality and high information content, laying a solid foundation for subsequent feature fusion and perception tasks, and significantly improving the overall performance of the perception system.

[0059] Step 102: For each spatial unit, generate a geometric awareness message of the neighboring units based on the relative geometric information between the spatial unit and the neighboring units, the geometric awareness message being used to reflect the spatial geometric constraints of the neighboring units relative to the spatial unit, the neighboring units being the spatial units that satisfy the preset geometric relationship with the spatial unit in the three-dimensional space.

[0060] In this step, for each spatial unit, it can be determined whether the geometric relationship between the spatial unit and each of the other spatial units satisfies the preset geometric relationship, and the corresponding neighboring units of each spatial unit are determined.

[0061] For each spatial unit, the relative geometric information between each spatial unit and its neighboring units can be calculated to reflect the position, direction, and other information of the neighboring units relative to the current spatial unit. Using the calculated relative geometric information, the geometric awareness message of the neighboring units is generated.

[0062] The preset geometric relationship is a predefined geometric condition between spatial units, used to determine the neighboring units and control the range and objects of feature fusion, for example, classification by K-Nearest Neighbor (KNN) classification algorithm, the relative distance between two spatial units being less than a preset distance threshold, two spatial units being within a preset line-of-sight range, two spatial units belonging to the same semantic segmentation region, etc.

[0063] The relative geometric information is information describing the relative position between two spatial units, which can include distance information and direction information. The relative geometric information can provide spatial constraints for feature fusion to ensure the geometric consistency of the fused features. The geometric awareness message is information reflecting the spatial geometric relationship of the neighboring units relative to the current spatial unit, which can guide the multi-modal feature update of the current spatial unit and ensure that the fused features contain correct spatial relationships.

[0064] The spatial geometric constraint is a rule that limits the relative position between spatial units, such as a distance threshold or a direction range, which can maintain the geometric rationality of feature fusion.

[0065] Step 103: Update the multi-modal features of the spatial unit based on the geometric awareness messages of the neighboring units to generate fused features.

[0066] In this step, for each spatial unit, the feature update can be achieved by methods such as weighted summation, feature splicing, or attention mechanism, and the geometric awareness messages of all neighboring units of the spatial unit are fused with the multi-modal features of the spatial unit to generate the fused features of the spatial unit.

[0067] The fused feature is a spatial unit feature updated through the geometric perception message, integrates multi-modal information, contains visual information and geometric information, and can more comprehensively represent the environmental information of the spatial unit, and can be used for subsequent environmental perception tasks such as target detection, identification and tracking.

[0068] Exemplarily, each node can be used to represent each spatial unit in the three-dimensional space, and the edges between the nodes can be used to represent the spatial adjacency relationship between the spatial units and the corresponding adjacent units, so as to construct the graph structure data. The graph structure data can provide topological constraints between the spatial units for subsequent feature fusion.

[0069] A learnable graph neural network that is equivariant to three-dimensional special Euclidean group SE(3) transformation can be constructed in advance. The graph neural network is equivariant to the input rigid body transformation, so that it is not sensitive to the attitude change of the input data. For example, the graph neural network can be an equivariant graph neural network (EGNN).

[0070] The graph structure data can be input into the graph neural network, and the graph neural network can be used for feature fusion. The graph neural network can include multiple equivariant convolution layers. Each equivariant convolution layer uses the multi-modal features corresponding to the nodes and the spatial adjacency relationship reflected by the edges to perform message passing: for each node, based on the relative geometric information between the node and the adjacent nodes connected to the node, the geometric perception messages of the adjacent nodes are generated; the node receives the geometric perception messages from the adjacent nodes, and then fuses the geometric perception messages of all adjacent nodes to update the multi-modal features of the node, to obtain the fused features of each node.

[0071] In the next equivariant convolution layer, the fused features obtained in the last layer are used to perform feature fusion again through the above message passing method. Finally, the fused features of each node with transformation equivariance are output.

[0072] Please refer to Figure 2 , Figure 2 A schematic diagram of graph structure data is shown. Exemplarily, a plurality of spatial units in a three-dimensional space can be abstracted into graph structure data G = (V, E), the nodes V in the graph structure data include node Vi, node Vj and node Vk, each node corresponds to a spatial unit, and the multi-modal features of the corresponding spatial unit are associated.

[0073] The edges E in the graph structure data can be established according to the spatial adjacency relationship between the spatial units, and the nodes corresponding to the spatial units and adjacent units satisfying the preset geometric relationship are connected. The edges E in the graph structure data can include edge E1 and edge E2.

[0074] The adjacent units of the spatial unit corresponding to the node Vi include the spatial unit corresponding to the node Vj and the spatial unit corresponding to the node Vk, the node Vi is connected with the node Vj through the edge E1, and the node Vi is connected with the node Vk through the edge E2.

[0075] The graph structure data shown in FIG. 1 is input into a pre-constructed graph neural network. Figure 2 In each isometric convolution layer of the graph neural network, for the node Vi, a geometric perception message of the node Vj is generated based on the relative geometric information between the spatial unit corresponding to the node Vi and the spatial unit corresponding to the node Vj.

[0076] A geometric perception message of the node Vk is generated based on the relative geometric information between the spatial unit corresponding to the node Vi and the spatial unit corresponding to the node Vk. The geometric perception message of the node Vj, the geometric perception message of the node Vk, and the multi-modal feature of the node Vi are fused to generate a fusion feature of the node Vi.

[0077] For the node Vj, a geometric perception message of the node Vi is generated based on the relative geometric information between the spatial unit corresponding to the node Vj and the spatial unit corresponding to the node Vi. The geometric perception message of the node Vi and the multi-modal feature of the node Vj are fused to generate a fusion feature of the node Vj.

[0078] For the node Vk, a geometric perception message of the node Vi is generated based on the relative geometric information between the spatial unit corresponding to the node Vk and the spatial unit corresponding to the node Vi. The geometric perception message of the node Vi and the multi-modal feature of the node Vk are fused to generate a fusion feature of the node Vk.

[0079] In the next isometric convolution layer, the fusion features of the node Vi, the node Vj and the node Vk generated in the previous layer are fused again. After the feature fusion of the multiple isometric convolution layers, the final fusion features of the node Vi, the node Vj and the node Vk can be obtained.

[0080] It can be understood that, Figure 2 The number of nodes and the topological structure in the graph structure data shown in FIG. 1 are only examples, and can be constructed according to the spatial units in the application scenario. In addition to the graph structure data, the multiple spatial units in the three-dimensional space can also be represented as tree structure, adjacency matrix and other forms of data, and the embodiments of the present application do not limit this.

[0081] The feature fusion method of the embodiment is characterized in that a plurality of space units in a three-dimensional space are obtained, the multi-modal features of each space unit include visual features and point cloud features, for each space unit, based on the relative geometric information between the space unit and the adjacent unit, a geometric perception message of the adjacent unit is generated, based on the geometric perception message of the adjacent unit, the multi-modal features of the space unit are updated to generate fused features, by generating and aggregating the geometric perception message by using the relative geometric information in the three-dimensional space, the relative geometric constraints between the space units remain unchanged, the multi-modal features can be accurately fused in the three-dimensional space, it is not necessary to project the visual features and the point cloud features into a two-dimensional plane for fusion, the projection misalignment caused by the complex attitude changes such as vehicle pitching, rolling or yawing is avoided, and thus the fusion accuracy of the point cloud data and the visual data is improved.

[0082] In the foregoing embodiments, by obtaining a plurality of space units in a three-dimensional space, generating a geometric perception message by using relative geometric information, updating the multi-modal features of the space unit to generate fused features, accurate feature fusion is realized, the projection misalignment problem caused by vehicle attitude changes is avoided, and thus the accuracy and robustness of feature fusion are improved. In the following embodiments, the fusion process of multi-modal features will be described in more detail, and any of the above embodiments can be applied.

[0083] In an embodiment, the relative geometric information can include relative distance features and relative direction features. When generating the geometric perception message of the adjacent unit, the relative distance features can be first geometrically distance coded to obtain distance feature coding, the multi-modal features of the adjacent unit are rotated and processed according to the relative direction features to obtain direction weighted features, then, based on the distance feature coding, the multi-modal features of the space unit and the multi-modal features of the adjacent unit, a geometric perception weight is generated, finally, the direction weighted features of the adjacent unit are modulated by using the geometric perception weight to generate the geometric perception message of the adjacent unit.

[0084] The relative distance features can reflect the distance between two space units and are used to encode the distance information between the space units. The relative direction features are the direction of the adjacent unit relative to the current unit and are used to encode the direction information between the space units.

[0085] The distance feature coding is a high-dimensional feature vector reflecting the distance between two space units, which can enhance the expression ability of the distance information and make it more suitable for feature fusion. The direction weighted features have rotational invariance, which can ensure that the direction features remain consistent under rotation and improve the robustness of the features.

[0086] The geometric perception weight is a weight calculated based on the relative geometric information and the features, which is used to reflect the influence degree of the adjacent unit on the features of the current unit.

[0087] Exemplarily, the relative distance feature can be geometrically distance coded in a manner of a radial basis function, a piecewise linear kernel function, or a piecewise Gaussian kernel function, so as to map the one-dimensional distance into a high-dimensional distance feature code. The multi-modal features of the neighboring cells can be rotationally invariantly processed in a manner of a spherical harmonic function, a rotation transformation network, or a rotation transformation matrix, based on the relative direction feature, so as to obtain direction weighted features.

[0088] The distance feature code, the multi-modal feature of the current spatial cell, and the multi-modal features of the neighboring cells are input into a multilayer perceptron (MLP), and the multilayer perceptron outputs geometric perception weights through a fully connected layer, an activation function, and a normalization layer. The geometric perception weights are multiplied with or weighted-summed with the direction weighted features element by element or in a manner of modulation processing, so as to generate geometric perception messages of the neighboring cells.

[0089] As described above, by introducing the geometric distance coding and the rotational invariant processing, the high-dimensional distance feature code is obtained by geometrically coding the relative distance feature, so as to enhance the expression ability of the distance information; the direction weighted features are obtained by rotationally invariantly processing the relative direction feature, so as to ensure the rotational invariance of the direction information, and the geometric perception weights generated based on the distance feature code and the relative direction feature can more accurately reflect the influence of the neighboring cells on the current spatial cell, thereby improving the accuracy and adaptability of the feature fusion.

[0090] In an embodiment, when the relative distance feature is geometrically distance coded, a plurality of preset distance features uniformly distributed at the center can be obtained; difference information between the relative distance feature and each preset distance feature can be obtained; each difference information can be converted into a distance weight coefficient; and the distance weight coefficients corresponding to all the difference information can be combined to obtain the distance feature code.

[0091] The preset distance feature is a set of predefined distance values, which can be used as a radial basis center of the radial basis function. The difference information can reflect the difference between the relative distance feature and the preset distance feature, and is used to calculate the distance weight coefficient.

[0092] The distance weight coefficient is a weight calculated based on the difference information, and is used to combine to form the distance feature code. The extended feature is a feature mapped to a high-dimensional space by the spherical harmonic function, which can enhance the expression ability of the direction feature and support the rotational invariant processing.

[0093] Exemplarily, the radial basis function can be used for the geometric distance coding. A plurality of preset distance features uniformly distributed at the center can be obtained first. ​​), as the radial basis center. These preset distance features are distributed within a preset range, which can be set according to experience or statistical results, for example, 0 to the maximum search radius of the spatial unit and the adjacent unit.

[0094] For the relative distance feature between any pair of spatial units and adjacent units, difference information between the relative distance feature and each preset distance feature is obtained. The difference information is mapped by a Gaussian radial basis function to convert into a corresponding distance weight coefficient. The distance weight coefficients corresponding to all difference information are sequentially spliced to obtain a distance feature code with a dimension of K.

[0095] For example, the distance feature code can be determined according to the following formula 1:

[0096] Formula 1

[0097] wherein, denotes the distance feature code, , denotes a real number field, denotes the relative distance feature, denotes the preset distance feature, denotes a Gaussian kernel width hyperparameter for controlling the sensitivity of the basis function to the distance difference, denotes the total number of preset distance features, denotes the difference information, denotes the distance weight coefficient.

[0098] As described above, by using the radial basis function to geometrically encode the relative distance feature, one-dimensional distance information can be expanded into a multi-dimensional feature vector, which can not only enhance the expression ability of the distance feature, but also make the feature more sensitive to distance changes. The uniform distribution of the preset distance feature center can ensure the comprehensiveness of the encoding, and the process of converting the difference information into the distance weight coefficient can further refine the expression of the distance feature. The distance feature code obtained by combining all distance weight coefficients can more accurately reflect the relative distance relationship between nodes and improve the spatial resolution of feature fusion.

[0099] In an embodiment, when geometrically encoding the relative distance feature, the multi-modal feature of the adjacent unit can be subjected to rotation equivariant processing, and the relative direction feature can be mapped onto a spherical harmonic function. Based on the multi-modal feature of the adjacent unit and the spherical harmonic function, an extended feature containing direction information is obtained. According to the weight of each spherical harmonic degree, the extended features corresponding to all orders under the spherical harmonic degree are weighted and summed to obtain a direction weighted feature.

[0100] wherein the spherical harmonic function is an orthogonal basis defined on a spherical surface, has a learnable, rotation equivariant convolution kernel base, and the spherical harmonic function can include a plurality of spherical harmonic degrees ) corresponding basis functions. Each spherical harmonic includes a complete order from 0 to 2n. The weight of each spherical harmonic can be pre-learned in the network training phase.

[0101] Exemplarily, the relative direction feature between the spatial unit and the neighboring unit is mapped onto the spherical harmonics, and the relative direction feature originally containing only direction information is converted into a series of spherical harmonic coefficients to form a high-dimensional representation of the direction feature.

[0102] Multiplying the multi-modal feature of the neighboring unit with the spherical harmonics obtains an extended feature containing direction information, which can promote the scalar feature to a direction-related feature space so that the subsequent processing can perceive the change of spatial orientation.

[0103] According to the weight of each spherical harmonic, the extended features corresponding to all orders of the spherical harmonic are weighted and summed to obtain a direction-weighted feature. The direction-weighted feature not only retains the intensity information of the multi-modal feature, but also encodes the rotation and other variable characteristics caused by the change of direction.

[0104] As described above, by processing the relative direction feature by using the spherical harmonics, the direction feature of the neighboring unit has rotation invariance, the relative direction feature is mapped to the spherical harmonic basis, and the features of different spherical harmonics are weighted and summed to generate an extended feature containing rich direction information. Not only the geometric information of the original direction feature is retained, but also the direction expression ability of the feature is enhanced through the weighted sum, so that the fused feature has natural adaptability to the rotation change, further improving the stability and accuracy of the perception result.

[0105] In an embodiment, the multi-modal feature of each spatial unit can further include spatial coordinate information. The spatial coordinate information is used to describe the spatial position (such as Cartesian coordinates) of the spatial unit in the three-dimensional space, to calculate the relative position feature, and to construct the spatial geometric constraint between the spatial units.

[0106] Before generating the geometric perception message of the neighboring unit, the relative position feature between the spatial unit and the neighboring unit can be determined according to the spatial coordinate information of the spatial unit and the spatial coordinate information of the neighboring unit; and the relative position feature is decomposed to obtain a relative distance feature and a relative direction feature.

[0107] Exemplarily, according to the spatial coordinate information of the spatial unit and the spatial coordinate information of the neighboring unit, the relative position feature between the two is calculated.

[0108] ​​The relative position feature vector is decomposed into two independent geometric components. By calculating the length of the relative position feature, the relative distance feature between the space unit and the adjacent unit is obtained . By normalizing the relative position feature, the relative direction feature pointing to the adjacent unit is obtained , to represent the direction of the adjacent unit relative to the space unit.

[0109] The obtained relative distance feature and relative direction feature are used for subsequent geometric distance coding and rotation invariant processing, respectively. The relative distance feature is encoded into a high-dimensional distance feature by a radial basis function, and the relative direction feature is transformed into a direction weighted feature by a spherical harmonic function. These two features jointly participate in the generation of geometric perception messages, ensuring that the fusion process is robust to vehicle pose changes.

[0110] As described above, by introducing spatial coordinate information and decomposing the relative position feature, the separation of the relative distance feature and the relative direction feature enables the distance and direction information to be encoded separately and more accurately capture the geometric relationship between space units. This not only improves the accuracy of feature fusion, but also enhances the perception of spatial geometric structure, so that the fused features can more truly reflect the spatial layout of the vehicle's surrounding environment, thereby improving the performance of perception tasks.

[0111] Please refer to Figure 3 , Figure 3 is a flowchart of an environment perception method according to an example embodiment. The environment perception method can be exemplarily executed by a perception system, and the environment perception method can also be executed by other systems or devices in different application scenarios. The environment perception method can include the following steps:

[0112] Step 301: obtaining the fused feature by the feature fusion method described in any embodiment of the present application.

[0113] In this step, the perception system can fuse the visual feature and the point cloud feature to generate the fused feature by the feature fusion method described in any embodiment of the present application. The fused feature can integrate visual information and geometric information to provide a more accurate data basis for subsequent environment perception tasks.

[0114] Step 302: mapping the fused feature to the BEV plane to generate a BEV feature map.

[0115] In this step, the fused feature corresponding to each space unit can be mapped from three-dimensional space to a predefined BEV plane by projection, voxelization, etc., to generate a BEV feature map. Converting the fused feature into a representation suitable for environment perception tasks facilitates subsequent processing and analysis. The BEV plane can be a two-dimensional bird's eye view grid plane.

[0116] Step 303: Perform an environment perception task based on the BEV feature map to generate an environment perception result.

[0117] In this step, the generated BEV feature map can be used to perform downstream environment perception tasks such as object detection, semantic segmentation, or path planning. The perception system generates environment perception results based on the BEV feature map to provide decision support for autonomous vehicles, ensuring safe and efficient driving.

[0118] The environment perception method of this embodiment maps the fused features to the BEV plane to generate a BEV feature map suitable for downstream environment perception tasks, converting three-dimensional fused features into a two-dimensional representation that retains key information in three-dimensional space and facilitates efficient computation and analysis in a two-dimensional plane. Based on the BEV feature map, the environment perception task can fully utilize the high precision and robustness of the fused features to generate reliable environment perception results, providing solid data support for the decision-making of autonomous vehicles.

[0119] To further introduce the feature fusion and perception process, Figure 4 A flowchart of a feature fusion and perception method is shown. The method can include the following steps:

[0120] Step 401: Obtain point cloud data and visual data of the environment around the vehicle.

[0121] In this step, high-resolution point cloud data and visual data can be collected by the laser radar and camera mounted on the vehicle. The point cloud data can form a 120000x5 tensor containing accurate geometric information of the environment, and the visual data can have a resolution of 1920x1080, providing rich visual details. The extrinsic calibration between the laser radar and the camera is performed in advance to ensure data consistency and accuracy.

[0122] Step 402: Map each data point in the point cloud data to a spatial unit.

[0123] In this step, each data point in the laser radar point cloud data is converted into a spatial unit, and each spatial unit corresponds to a three-dimensional coordinate point.

[0124] Step 403: Associate the visual features and point cloud features of each data point to the corresponding spatial unit to obtain the multi-modal features of each spatial unit.

[0125] In this step, the camera's intrinsic and extrinsic parameter matrix is used to project each data point onto a two-dimensional image plane, determine the projection area of each data point in the visual data, and extract the visual features of the projection area. The extracted point cloud features based on the point cloud data are combined to form the multi-modal features of the corresponding spatial unit.

[0126] Step 404: For each spatial unit, decompose the relative position feature between the spatial unit and the neighboring unit into a relative distance feature and a relative direction feature.

[0127] In this step, the neighboring data points of each data point in the point cloud data can be determined by a radius neighbor search method (for example, the radius is 0.5 meters, and the maximum number of neighbors is 32), and a k-NN graph is constructed. The local geometric structure between data points can be effectively captured by using the k-NN graph.

[0128] For each spatial unit, the neighboring unit of the spatial unit that satisfies the preset geometric relationship in the three-dimensional space can be determined according to the constructed k-NN graph.

[0129] According to the spatial coordinate information of the spatial unit and the spatial coordinate information of the neighboring unit, the relative position vector between the spatial unit and the neighboring unit is calculated. The relative position feature is decomposed to obtain a relative distance feature and a relative direction feature.

[0130] Step 405: Geometric distance coding is performed on the relative distance feature by using a radial basis function to obtain a distance feature code.

[0131] In this step, a plurality of preset distance features uniformly distributed at the center are obtained. The difference information between the relative distance feature and each preset distance feature is obtained by using a radial basis function. Each difference information is converted into a distance weight coefficient, and the distance weight coefficients corresponding to all difference information are combined to obtain a high-dimensional distance feature code, thereby enhancing the expression ability of distance information.

[0132] Step 406: A spherical harmonic function is used to perform rotation equivariant processing on the multi-modal feature of the neighboring unit according to the relative direction feature, to obtain a direction weighted feature of the neighboring unit.

[0133] In this step, the relative direction feature is mapped to a spherical harmonic function, and the multi-modal feature of the neighboring unit is multiplied by the spherical harmonic function to obtain an extended feature. The extended features corresponding to all orders under each spherical harmonic degree are weighted and summed according to the weight of each spherical harmonic degree to obtain the direction weighted feature.

[0134] Step 407: A geometric perception weight is generated based on the distance feature code, the multi-modal feature of the spatial unit, and the multi-modal feature of the neighboring unit.

[0135] In this step, the distance feature code, the multi-modal feature of the spatial unit, and the multi-modal feature of the neighboring unit are input into a multi-layer perception machine, and the multi-layer perception machine is used to generate the geometric perception weight of the neighboring unit.

[0136] Step 408: Modulate the direction weighted feature using the geometric perception weight to generate the geometric perception message of the neighboring cell.

[0137] In this step, the geometric perception weight is multiplied by the direction weighted feature to generate the geometric perception message containing geometric and visual information. If there are multiple neighboring cells satisfying the preset geometric relationship for the space cell, the geometric perception message of each neighboring cell can be generated by executing steps 404-408 for each neighboring cell of the space cell.

[0138] Step 409: Update the multi-modal feature of the space cell based on the geometric perception message of the neighboring cell to generate the fusion feature.

[0139] In this step, the geometric perception messages of all neighboring cells and the multi-modal feature of the space cell can be aggregated to generate the fusion feature of the space cell. By executing steps 404-409 for each space cell, the multi-modal features of each space cell are fused to obtain the fusion feature of each space cell, for example, dimensional scalar feature.

[0140] Each space cell can be represented as graph structure data, and the graph structure data is input into the isometric graph neural network. Steps 404-409 are executed using the graph neural network to fuse the multi-modal features of each space cell, and the fusion features of each space cell output by the graph neural network are obtained.

[0141] Step 410: Map the fusion feature to the BEV plane to generate the BEV feature map.

[0142] In this step, the fusion feature of each space cell is projected from three-dimensional space to the BEV plane. The fusion features of the space cells falling within the same BEV grid cell are averaged and pooled to generate the BEV feature map suitable for downstream task processing. The BEV space range can be 80 meters in front, 40 meters on the left and right, and the grid size can be 0.2m x 0.2m. The size of the BEV feature map can be 400 x 400 x 256.

[0143] Step 411: Perform the environment perception task based on the BEV feature map to generate the environment perception result.

[0144] In this step, the generated BEV feature map is input into a CenterPoint-based detection head to perform a three-dimensional target detection task to obtain the environment perception result, which provides support for autonomous driving decision-making.

[0145] Figure 5is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application. The electronic device may, for example, be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, a car machine, etc. Referring to Figure 5 At the hardware level, the electronic device includes a processor 501, an internal bus 502, a network interface 503, a memory 504, and a non-volatile memory 505, and of course can also include other hardware required by the business. The processor 501 reads the corresponding computer program from the non-volatile memory 505 into the memory 504 and then runs, forming a feature fusion and / or environment perception device at the logical level. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logical unit, but can also be hardware or logic devices.

[0146] Figure 6 is a block diagram of a feature fusion device according to an exemplary embodiment of the present application. Referring to Figure 6 The device can include a unit acquisition module 601, a generation module 602, and a fusion module 603, wherein:

[0147] The unit acquisition module 601 is configured to acquire a plurality of spatial units in a three-dimensional space, and the multi-modal features of each spatial unit include visual features and point cloud features;

[0148] The generation module 602 is configured to, for each spatial unit, generate a geometric perception message of a neighboring unit based on relative geometric information between the spatial unit and the neighboring unit, the geometric perception message being used to reflect a spatial geometric constraint of the neighboring unit relative to the spatial unit, the neighboring unit being a spatial unit that satisfies a preset geometric relationship with the spatial unit in the three-dimensional space;

[0149] The fusion module 603 is configured to update the multi-modal features of the spatial unit based on the geometric perception message of the neighboring unit, and generate a fused feature.

[0150] Optionally, the generation module 602 includes:

[0151] A distance encoding module is configured to perform geometric distance encoding on the relative distance feature to obtain a distance feature code;

[0152] A transformation processing module is configured to perform rotation and other transformation processing on the multi-modal features of the neighboring unit according to the relative direction feature to obtain a direction weighted feature of the neighboring unit;

[0153] a weight generation module configured to generate a geometric perception weight based on the distance feature code, the multi-modal feature of the space unit and the multi-modal feature of the adjacent unit;

[0154] a message generation module configured to generate a geometric perception message of the adjacent unit by modulating the direction weighted feature using the geometric perception weight.

[0155] Optionally, the distance encoding module comprises:

[0156] a feature acquisition module configured to acquire a plurality of preset distance features uniformly distributed at the center;

[0157] a difference acquisition module configured to acquire difference information between the relative distance feature and each preset distance feature;

[0158] a coefficient conversion module configured to convert each difference information into a distance weight coefficient;

[0159] a coefficient combination module configured to combine distance weight coefficients corresponding to all difference information to obtain the distance feature code.

[0160] Optionally, the transformation processing module comprises:

[0161] a feature mapping module configured to map the relative direction feature to a spherical harmonic function, the spherical harmonic function comprising base functions corresponding to a plurality of spherical harmonic degrees;

[0162] an expansion module configured to obtain an expansion feature containing direction information based on the multi-modal feature of the adjacent unit and the spherical harmonic function;

[0163] a weighting module configured to perform weighted summation on expansion features corresponding to all orders under each spherical harmonic degree according to a weight of the spherical harmonic degree to obtain the direction weighted feature.

[0164] Optionally, the multi-modal feature of each space unit further comprises spatial coordinate information; the device further comprises:

[0165] a position determination module configured to determine a relative position feature between the space unit and the adjacent unit according to spatial coordinate information of the space unit and spatial coordinate information of the adjacent unit;

[0166] a decomposition module configured to decompose the relative position feature to obtain the relative distance feature and the relative direction feature.

[0167] Optionally, the unit acquisition module 601 comprises:

[0168] a data acquisition module configured to acquire point cloud data and visual data of an environment around a vehicle;

[0169] a data point mapping module, configured to map each data point in the point cloud data as a spatial unit;

[0170] a point cloud feature extraction module, configured to extract a point cloud feature of each data point based on the point cloud data;

[0171] a visual feature extraction module, configured to determine a projection area of each data point in the visual data based on a spatial position of the data point, and perform feature extraction on the projection area to obtain a visual feature of the data point;

[0172] a feature association module, configured to associate the visual feature and the point cloud feature of each data point to a corresponding spatial unit to obtain a multi-modal feature of each spatial unit.

[0173] Figure 7 is a block diagram of an environment perception apparatus according to an example embodiment. Referring to Figure 7 , the apparatus can include a feature acquisition module 701, a mapping module 702, and a perception module 703, wherein:

[0174] the feature acquisition module 701 is configured to acquire a fusion feature through the feature fusion method described in any of the embodiments of the present application;

[0175] the mapping module 702 is configured to map the fusion feature to a BEV plane to generate a BEV feature map;

[0176] the perception module 703 is configured to perform an environment perception task based on the BEV feature map to generate an environment perception result.

[0177] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0178] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the part of the method embodiment. The above described device embodiment is only illustrative, wherein the modules described as separate components can be or can not be physically separated, and the components displayed as modules can be or can not be physical modules, that is, they can be located in one place, or can be distributed on multiple network modules. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0179] In an example embodiment, there is also provided a non-transitory computer- readable storage medium comprising instructions, for example a memory comprising instructions, executable by a processor of a feature fusion and / or environment perception apparatus to implement a method as described in any of the above embodiments.

[0180] The non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, and the like, and the present application is not limited thereto.

[0181] In an example embodiment, there is also provided a computer program product comprising computer programs / instructions executable by a processor of a feature fusion and / or environment perception apparatus to implement a method as described in any of the above embodiments.

[0182] The above descriptions have been provided for specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish the desired results. Also, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0183] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is not intended to be limited by the above description, and modifications and variations are possible without departing from the scope of the application. The scope of the application is limited only by the following claims.

[0184] The above descriptions are only preferred embodiments of the present application and are not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A feature fusion method, characterized in that, The method comprises: obtaining a plurality of space units in a three-dimensional space, the multi-modal feature of each space unit comprising a visual feature and a point cloud feature; for each space unit, generating a geometric perception message of a neighboring unit based on a relative distance feature and a relative direction feature between the space unit and the neighboring unit, the geometric perception message being used to reflect a spatial geometric constraint of the neighboring unit relative to the space unit to guide multi-modal feature updating of the space unit, the neighboring unit being a space unit that satisfies a preset geometric relationship with the space unit in the three-dimensional space; fusing the geometric perception message of all neighboring units of the space unit with the multi-modal feature of the space unit to generate a fused feature of the space unit; the generating of the geometric perception message of the neighboring unit based on the relative distance feature and the relative direction feature between the space unit and the neighboring unit comprises: geometric distance coding the relative distance feature to obtain a distance feature code; rotating and invariating the multi-modal feature of the neighboring unit according to the relative direction feature to obtain a direction weighted feature of the neighboring unit; generating a geometric perception weight based on the distance feature code, the multi-modal feature of the space unit and the multi-modal feature of the neighboring unit; modulating the direction weighted feature using the geometric perception weight to generate the geometric perception message of the neighboring unit.

2. The method of claim 1, wherein, the geometric distance coding of the relative distance feature to obtain the distance feature code comprises: obtaining a plurality of preset distance features uniformly distributed at the center; obtaining difference information between the relative distance feature and each preset distance feature; converting each difference information into a distance weight coefficient; combining distance weight coefficients corresponding to all difference information to obtain the distance feature code.

3. The method of claim 1, wherein, the rotating and invariating of the multi-modal feature of the neighboring unit according to the relative direction feature to obtain the direction weighted feature comprises: mapping the relative direction feature onto a spherical harmonic function, the spherical harmonic function comprising a plurality of basis functions corresponding to spherical harmonic degrees; obtaining an extended feature containing direction information based on the multi-modal feature of the neighboring unit and the spherical harmonic function; weighting and summing the extended features corresponding to all orders under each spherical harmonic degree according to the weight of each spherical harmonic degree to obtain the direction weighted feature.

4. The method of claim 1, wherein, The multi-modal feature of each space unit further comprises spatial coordinate information. Before the generating of the geometric perception message of the neighboring unit based on the relative distance feature and the relative direction feature between the space unit and the neighboring unit, the method further comprises: determining a relative position feature between the space unit and the neighboring unit according to the spatial coordinate information of the space unit and the spatial coordinate information of the neighboring unit; decomposing the relative position feature to obtain the relative distance feature and the relative direction feature.

5. The method of claim 1, wherein, The obtaining of the plurality of space units in the three-dimensional space comprises: obtaining point cloud data and visual data of an environment around a vehicle; mapping each data point in the point cloud data into a space unit; extracting a point cloud feature of each data point based on the point cloud data; determine a projection region of the data point in the visual data based on a spatial position of the data point, and perform feature extraction on the projection region to obtain a visual feature of the data point; associate the visual feature and the point cloud feature of each data point to a corresponding spatial cell to obtain a multi-modal feature of each spatial cell.

6. An environmental perception method characterized by, The method comprises: obtaining the fusion feature by the feature fusion method in any one of claims 1-5; mapping the fusion feature to a BEV plane to generate a BEV feature map; performing an environment perception task based on the BEV feature map to generate an environment perception result.

7. A feature fusion device, characterized by, The device comprises: a cell obtaining module configured to obtain a plurality of spatial cells in a three-dimensional space, and a multi-modal feature of each spatial cell comprising a visual feature and a point cloud feature; a generating module configured to, for each spatial cell, generate a geometric perception message of a neighboring cell based on a relative distance feature and a relative direction feature between the spatial cell and the neighboring cell, the geometric perception message being used to reflect a spatial geometric constraint of the neighboring cell relative to the spatial cell to guide multi-modal feature updating of the spatial cell, the neighboring cell being a spatial cell that satisfies a preset geometric relationship with the spatial cell in the three-dimensional space; a fusion module configured to fuse the geometric perception message of all neighboring cells of the spatial cell with the multi-modal feature of the spatial cell to generate a fusion feature of the spatial cell. The generating module comprises: a distance encoding module configured to perform geometric distance encoding on the relative distance feature to obtain a distance feature encoding; a transformation processing module configured to perform rotation or other transformation processing on the multi-modal feature of the neighboring cell according to the relative direction feature to obtain a direction weighted feature of the neighboring cell; a weight generating module configured to generate a geometric perception weight based on the distance feature encoding, the multi-modal feature of the spatial cell, and the multi-modal feature of the neighboring cell; a message generating module configured to modulate the direction weighted feature using the geometric perception weight to generate the geometric perception message of the neighboring cell.

8. An environmental perception apparatus, characterized by, The device comprises: a feature obtaining module configured to obtain a fusion feature by the feature fusion method in any one of claims 1-5; a mapping module configured to map the fusion feature to a BEV plane to generate a BEV feature map; a perception module configured to perform an environment perception task based on the BEV feature map to generate an environment perception result.

9. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor implements the feature fusion method in any one of claims 1-5 or the environment perception method in claim 6 by running the executable instructions.

10. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the feature fusion method in any one of claims 1-5 or the environment perception method in claim 6.

11. A computer program product having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the feature fusion method in any one of claims 1-5 or the environment perception method in claim 6.

Images