Multi-sensor feature fusion method, system, electronic device, and storage medium

Abstract

Embodiments of the present application provide a multi-sensor feature fusion method and system, electronic equipment and storage medium, relating to the technical field of radar and vision fusion. The method comprises: obtaining image BEV features according to image data; obtaining initial point cloud BEV features according to point cloud data; converting the initial point cloud BEV features to a coordinate system of the image BEV features to obtain first point cloud BEV features; determining image BEV features matched with the first point cloud BEV features in the coordinate system, and performing feature fusion on the first point cloud BEV features and the matched image BEV features to obtain fused BEV features; and inputting the fused BEV features into a target detection head for target detection. Through the multi-sensor feature fusion method of the embodiments, the accuracy of the fused features can be improved in the process of multi-sensor feature fusion, so as to improve the accuracy of target detection.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of radar and vision fusion technology, and in particular to a multi-sensor feature fusion method, system, electronic device and storage medium. Background Technology

[0002] Environmental perception is a crucial module in autonomous driving, encompassing tasks including, but not limited to, 2D / 3D object detection, semantic segmentation, depth completion, and prediction. These tasks all rely on raw data collected from the environment by sensors mounted on the vehicle. Commonly used sensors for environmental perception include cameras, LiDAR, and millimeter-wave radar, each with its own advantages and disadvantages. However, to enable all-weather detection capabilities and perform multiple functions such as road obstacle detection, lane detection, object detection, speed and distance measurement, the perception capabilities of single-modal data have reached a bottleneck. Therefore, most researchers in industry and academia have begun to focus on multimodal fusion. Currently, multi-sensor fusion perception methods include pre-fusion algorithms, which primarily perform data fusion or feature fusion for LiDAR, cameras, and millimeter-wave radar.

[0003] However, in related pre-fusion schemes, there is a problem that visual sensors and radar sensors operate at different frequencies, making temporal alignment during feature fusion difficult. When the target's movement speed is too high, the error generated by feature fusion is large. Therefore, how to improve the accuracy of fused features during multi-sensor feature fusion to enhance the accuracy of target detection is a technical problem that this invention urgently needs to solve. Summary of the Invention

[0004] This invention provides a multi-sensor feature fusion method, system, electronic device, and storage medium to improve the accuracy of fused features during the multi-sensor feature fusion process.

[0005] The first aspect of this invention provides a multi-sensor feature fusion method, the method comprising:

[0006] Based on the image data, obtain the BEV features of the image;

[0007] Based on the point cloud data, obtain the initial point cloud BEV features;

[0008] The initial point cloud BEV features are transformed to the coordinate system of the image BEV features to obtain the first point cloud BEV features.

[0009] Determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature.

[0010] The fused BEV features are input into the target detection head for target detection.

[0011] Optionally, determining the image BEV features that match the first point cloud BEV features in the coordinate system, and fusing the first point cloud BEV features with the matched image BEV features to obtain fused BEV features, includes:

[0012] The search and matching part of the neighborhood search network is used to search for the image BEV features and match the image BEV features that correspond to the first point cloud BEV features in the coordinate system.

[0013] The feature fusion part of the neighborhood search network fuses the first point cloud BEV features with the matched image BEV features to obtain the fused BEV features.

[0014] Optionally, the step of searching for the image BEV features through the search matching part of the neighborhood search network, and matching the image BEV features corresponding to the first point cloud BEV features in the coordinate system, includes:

[0015] The search and matching part adds the first point cloud BEV features and the image BEV features together, and determines whether the correlation between the BEV features after feature addition exceeds a threshold.

[0016] If the correlation exceeds the threshold, a matching pair is obtained;

[0017] If the correlation does not exceed the threshold, iterative learning is performed to determine the offset of the first point cloud BEV feature;

[0018] Based on the first point cloud BEV features and the offset, determine the offset first point cloud BEV features;

[0019] The BEV features of the offset first point cloud are added to the BEV features of the image until the correlation exceeds the threshold, thus obtaining the matching pair.

[0020] The image BEV feature in the matching pair is determined as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

[0021] Optionally, the feature fusion step of fusing the first point cloud BEV features with the matched image BEV features through the feature fusion part of the neighborhood search network to obtain the fused BEV features includes:

[0022] The feature fusion part inputs the sum of the features of the matching pair into a 3x3 convolutional kernel, an encoder, and a decoder in sequence to obtain the first output;

[0023] The first output is fused with the output of the 3x3 convolution kernel to obtain the second output;

[0024] After reducing the dimensionality of the second output by inputting it into a 1x1 convolution kernel, the third output is obtained.

[0025] The third output is filtered by the judge to obtain the fourth output;

[0026] The fourth output is fused with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

[0027] Optionally, obtaining image BEV features based on image data includes:

[0028] Acquire multiple image data captured by the vehicle's panoramic camera;

[0029] By using the Dual-Swin-Tiny backbone network and FPN network in the improved LSS algorithm, 2D features are extracted and enhanced from the multiple image data to obtain multi-view 2D feature maps.

[0030] Based on the vehicle's own coordinate system, the multi-view 2D feature map is projected onto the vehicle's own coordinate system to obtain the image BEV features;

[0031] Wherein, the vehicle coordinate system is the coordinate system of the BEV feature in the image.

[0032] Optionally, obtaining the initial point cloud BEV features based on the point cloud data includes:

[0033] Acquire the point cloud data captured by the vehicle's point cloud device;

[0034] The point cloud data is converted into the initial point cloud BEV features by learning the parameterized hypervoxel method of the original points.

[0035] Optionally, the target detection head includes at least one of the following: a CenterPoint detection head, a PointPillars detection head, and a TransFusion-L detection head.

[0036] A second aspect of this invention provides a multi-sensor feature fusion system, the system comprising:

[0037] The image feature extraction module is used to obtain the BEV features of an image based on the image data.

[0038] The point cloud feature extraction module is used to obtain the initial point cloud BEV features based on the point cloud data.

[0039] The feature transformation module is used to transform the initial point cloud BEV features to the coordinate system of the image BEV features to obtain the first point cloud BEV features.

[0040] The feature fusion module is used to determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and to fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature.

[0041] The detection head module is used to input the fused BEV features into the target detection head for target detection.

[0042] Optionally, the feature fusion module includes:

[0043] The feature matching submodule is used to search for the image BEV features through the search matching part of the neighborhood search network, and to match the image BEV features that correspond to the first point cloud BEV features in the coordinate system.

[0044] The feature fusion submodule is used to fuse the first point cloud BEV features with the matched image BEV features through the feature fusion part of the neighborhood search network to obtain the fused BEV features.

[0045] Optionally, the feature matching submodule includes:

[0046] The feature addition submodule is used to add the first point cloud BEV features and the image BEV features through the search matching part, and determine whether the correlation between the BEV features after feature addition exceeds a threshold.

[0047] The first threshold determination submodule is used to obtain a matching pair if the correlation degree exceeds the threshold.

[0048] The second threshold determination submodule is used to perform iterative learning to determine the offset of the first point cloud BEV feature if the correlation degree does not exceed the threshold.

[0049] The offset submodule is used to determine the offset first point cloud BEV features based on the first point cloud BEV features and the offset amount.

[0050] The judgment submodule is used to add the BEV features of the offset first point cloud to the BEV features of the image until it is determined that the correlation exceeds the threshold, and thus obtain the matching pair;

[0051] The BEV feature matching submodule is used to determine the image BEV feature in the matching pair as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

[0052] Optionally, the feature fusion submodule includes:

[0053] The first output submodule is used to input the sum of the features of the matching pair into a 3x3 convolutional kernel, an encoder, and a decoder in sequence through the feature fusion part to obtain the first output;

[0054] The second output submodule is used to perform feature fusion between the first output and the output of the 3x3 convolutional kernel to obtain the second output;

[0055] The third output submodule is used to reduce the dimensionality of the second output input 1X1 convolution kernel to obtain the third output;

[0056] The fourth output submodule is used to filter the third output through a judge to obtain the fourth output;

[0057] The BEV feature fusion submodule is used to fuse the fourth output with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

[0058] Optionally, the image feature extraction module includes:

[0059] The image data acquisition submodule is used to acquire multiple image data captured by the vehicle's panoramic camera;

[0060] The 2D feature acquisition submodule is used to extract and enhance 2D features from the multiple image data through the backbone network Dual-Swin-Tiny and FPN network in the improved LSS algorithm, so as to obtain multi-view 2D feature maps.

[0061] The image BEV feature acquisition submodule is used to project the multi-view 2D feature map onto the vehicle coordinate system based on the vehicle's own coordinate system to obtain the image BEV features.

[0062] Wherein, the vehicle coordinate system is the coordinate system of the BEV feature in the image.

[0063] Optionally, the point cloud feature extraction module includes:

[0064] The point cloud data acquisition submodule is used to acquire the point cloud data captured by the vehicle's point cloud device.

[0065] The point cloud BEV feature acquisition submodule is used to convert the point cloud data into the initial point cloud BEV features by learning the parameterized hypervoxel method of the original points.

[0066] Optionally, the target detection head includes at least one of the following: a CenterPoint detection head, a PointPillars detection head, and a TransFusion-L detection head.

[0067] A third aspect of the present invention provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the multi-sensor feature fusion method as described in the first aspect of the present invention.

[0068] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-sensor feature fusion method of the first aspect of the present invention.

[0069] The multi-sensor feature fusion method provided in this invention first obtains image BEV features based on image data; then, it obtains initial point cloud BEV features based on point cloud data. Next, it transforms the initial point cloud BEV features to the coordinate system of the image BEV features, obtaining first point cloud BEV features. Then, it determines image BEV features that match the first point cloud BEV features in the coordinate system, and fuses the first point cloud BEV features with the matched image BEV features to obtain fused BEV features. Finally, it inputs the fused BEV features into a target detection head for target detection. In this embodiment, image BEV features and point cloud BEV features are extracted separately, and the feature extraction processes are independent and do not affect each other, thus ensuring the robustness of each sensor branch. That is, if one feature extraction branch encounters a problem, it does not affect the feature extraction process of other branches. Furthermore, after BEV feature extraction, the image BEV features and point cloud BEV features are transformed to the same coordinate system, and then spatial BEV feature matching is performed. This allows for feature fusion of the first point cloud BEV features and the image BEV features that match them in spatial location, thereby improving the accuracy of fused BEV features, increasing the accuracy of target detection, providing a higher-dimensional feature representation for target detection, and providing more reliable detection results for 3D target detection. This overcomes the technical problem of large errors caused by feature fusion due to temporal issues when the target's motion speed is too high. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating a multi-sensor feature fusion method according to an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram illustrating a feature search and matching process using a neighborhood search network according to an embodiment of the present invention;

[0073] Figure 3 This is a diagram illustrating a neighborhood search network structure according to an embodiment of the present invention;

[0074] Figure 4 This is a branch structure diagram of image BEV feature extraction shown in one embodiment of the present invention;

[0075] Figure 5 This is a diagram of the backbone network structure of an image BEV feature extraction branch, as shown in one embodiment of the present invention.

[0076] Figure 6 This is a diagram of the backbone network structure of a point cloud BEV feature extraction branch, as shown in one embodiment of the present invention.

[0077] Figure 7 This is an overall architecture design diagram of a multi-sensor feature fusion scheme shown in one embodiment of the present invention;

[0078] Figure 8 This is a structural block diagram of a multi-sensor feature fusion system provided in an embodiment of the present invention;

[0079] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] refer to Figure 1 , Figure 1 This is a flowchart illustrating a multi-sensor feature fusion method according to an embodiment of the present invention. Figure 1 As shown, the multi-sensor feature fusion method of this embodiment may include the following steps:

[0082] Step S1: Obtain the BEV features of the image based on the image data.

[0083] In this embodiment, visual image data can be acquired through a visual sensor, and feature extraction can be performed on the acquired image data to obtain BEV features. This step can be performed using any feature extraction algorithm that processes visual image data into BEV features; the feature extraction algorithm can be replaced at any time, and this embodiment does not impose any restrictions on it.

[0084] Step S2: Obtain the initial point cloud BEV features based on the point cloud data.

[0085] In this embodiment, point cloud data can be acquired using a radar sensor, and feature extraction can be performed on the acquired point cloud data to obtain initial point cloud BEV features. This step can be performed using any perception algorithm that processes point cloud data into BEV features, and can be replaced at any time. For example, the PointPillars algorithm, TransFusion algorithm, etc., can be used; this embodiment does not impose any restrictions. In this embodiment, the radar sensor can be a lidar sensor, millimeter-wave radar sensor, etc., and the initial point cloud BEV features are the BEV features extracted from the point cloud data. In this embodiment, S1 and S2 are executed simultaneously.

[0086] Step S3: Transform the initial point cloud BEV features to the coordinate system of the image BEV features to obtain the first point cloud BEV features.

[0087] In this embodiment, considering the different operating frequencies of the visual sensor and the radar sensor, there are temporal errors in the acquired image data and point cloud data, resulting in spatial differences. Therefore, after obtaining the initial point cloud BEV features, this embodiment maps the initial point cloud BEV features to the image space where the image BEV features reside; that is, it transforms the initial point cloud BEV features to the coordinate system of the image BEV features, obtaining the transformed first point cloud BEV features. In this embodiment, the first point cloud BEV feature is the point cloud BEV feature corresponding to the initial point cloud BEV features in the image BEV feature coordinate system.

[0088] Step S4: Determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature.

[0089] In this embodiment, after performing point cloud BEV feature transformation to obtain the first point cloud BEV feature, it is necessary to determine the image BEV feature that matches the first point cloud BEV feature in spatial location. That is, to determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system of the image BEV feature, so as to complete the position alignment of the two BEV features. Thus, when performing multi-sensor feature fusion, it is no longer a simple addition and fusion of the extracted image BEV feature and the initial point cloud BEV feature, but rather the feature fusion of the first point cloud feature and the image BEV feature that matches the first point cloud BEV feature in the coordinate system of the image BEV feature, to obtain the fused BEV feature after feature fusion.

[0090] Step S5: Input the fused BEV features into the target detection head to perform target detection.

[0091] In this embodiment, the obtained fused BEV features are input into the target detection head for target detection, and the target detection result is obtained. The target detection head is a pre-set detection head; this embodiment supports multiple detection heads and is not limited thereto.

[0092] In this embodiment, firstly, strict requirements are not placed on the accuracy of multi-sensor joint calibration during the feature extraction stage. The extracted multi-sensor BEV features are aligned and matched in spatial position, thereby achieving more accurate fusion. This compensates for the deficiency of low final object detection accuracy caused by excessive multi-sensor joint calibration errors. This is especially beneficial for objects moving at medium to high speeds, as it is more conducive to fusing effective features, improving the effectiveness of feature fusion and further enhancing the detection effect. Secondly, the image BEV feature extraction branch and the point cloud BEV feature extraction branch use their own independent feature extraction modules, and they are no longer interdependent during the process. If one branch unexpectedly crashes or fails during operation, other branches can continue to run, improving the stability and robustness of the multi-sensor feature fusion method. Finally, in this embodiment, the image BEV feature extraction branch and the point cloud BEV feature extraction branch do not need to be fixed with their specific schemes. Each modal branch can switch to any BEV feature extraction scheme, making the overall framework highly flexible and applicable to multi-scheme comparative analysis.

[0093] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-sensor feature fusion method. In this method, step S4 specifically includes steps S41 and S42:

[0094] Step S41: Search for the image BEV features through the search matching part of the neighborhood search network, and obtain the matching image BEV features that correspond to the first point cloud BEV features in the coordinate system.

[0095] In this embodiment, a neighborhood search network method is used for BEV feature search and matching and feature fusion. Specifically, this embodiment proposes a neighborhood search network (DSNet), which implements adaptive feature operations using a neural network algorithm during the BEV feature fusion process. The neighborhood search network mainly consists of two parts: a search matching part and a feature fusion part. The search matching part is used to search and match point cloud BEV features with image BEV features in spatial location. The feature fusion part is used to fuse the first successfully matched point cloud BEV features with the image BEV features.

[0096] In this embodiment, the search and matching part of the neighborhood search network searches for image BEV features in the image space, thereby obtaining image BEV features that correspond to the spatial location of the first point cloud BEV features.

[0097] Step S42: The first point cloud BEV features and the matched image BEV features are fused by the feature fusion part of the neighborhood search network to obtain the fused BEV features.

[0098] In this embodiment, in order to ensure the accuracy of neighborhood search matching, the feature fusion part of the neighborhood search network is used to fuse the matched first point cloud BEV features and the image BEV features that match the first point cloud BEV features in spatial location (i.e., in the coordinate system of the image BEV features) to obtain the fused BEV features.

[0099] Considering the mutual influence between laser and visual branches in traditional pre-fusion schemes, and the significant errors in joint calibration, which prevent true alignment of multi-sensor features, this embodiment aims to fuse the extracted image BEV features and point cloud BEV features at the BEV feature level to achieve a high matching degree. This embodiment uses a neighborhood search network to fuse the point cloud BEV features and corresponding visual BEV features. The neighborhood search network performs optimal feature search and matching, eliminating the need for calibration and overcoming the problem of low accuracy in target detection systems caused by calibration. It effectively avoids the feature mismatch problem caused by directly projecting the extracted visual BEV features and radar BEV features in traditional methods. This is especially beneficial for medium- and high-speed moving objects, facilitating the fusion of effective features and improving the effectiveness of feature fusion, thus further enhancing detection performance.

[0100] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-sensor feature fusion method. In this method, step S41 specifically includes steps S411 to S417:

[0101] Step S411: The search matching part adds the first point cloud BEV features and the image BEV features together, and determines whether the correlation between the BEV features after feature addition exceeds a threshold.

[0102] In this embodiment, after obtaining the first point cloud BEV features and the image BEV features, an iterative process is initiated to search and match the BEV features. First, the search and matching part of the neighborhood search network adds the first point cloud BEV features and the image BEV features. After feature addition, it is determined whether the correlation between the two BEV features (i.e., the first point cloud BEV features and the image BEV features) exceeds a threshold. That is, the correlation between the two BEV features is used to determine whether they are successfully matched. In this embodiment, the threshold is a pre-set minimum correlation value for successful matching, which can be freely set according to requirements, and there are no restrictions on the specific value of the threshold.

[0103] Step S412: If the correlation exceeds the threshold, a matching pair is obtained.

[0104] In this embodiment, if the correlation between two BEV features exceeds a threshold, the two BEV features are determined to be a successful match, and these two BEV features are identified as a matching pair for subsequent feature fusion. The matching pair includes a first point cloud BEV feature and an image BEV feature.

[0105] Step S413: If the correlation degree does not exceed the threshold, perform iterative learning to determine the offset of the first point cloud BEV feature.

[0106] In this embodiment, if the correlation between two BEV features does not exceed the threshold, it is determined that the two BEV features do not match. At this time, iterative learning of the neighborhood search network is required to learn the offset of the first point cloud BEV feature to be matched, that is, to learn the offset of the edge of the original first point cloud BEV feature.

[0107] Step S414: Based on the first point cloud BEV features and the offset, determine the offset first point cloud BEV features.

[0108] In this embodiment, after obtaining the offset, the original first point cloud BEV feature being matched is added to the obtained offset to obtain the offset first point cloud BEV feature. In essence, adding the offset to the first point cloud BEV feature changes its spatial position, allowing it to be shifted to a new location before re-matching the image BEV feature. This is the main process of search and matching in this embodiment.

[0109] Step S415: Add the offset first point cloud BEV features to the image BEV features until the correlation exceeds the threshold, and obtain the matching pair.

[0110] In this embodiment, after obtaining the offset first point cloud BEV features, the offset first point cloud BEV features are re-matched with the previously matched image BEV features. That is, the offset first point cloud BEV features and the previously matched image BEV features are added together. After the feature addition, it is re-determined whether the correlation between the two BEV features (i.e., the offset first point cloud BEV features and the image BEV features) exceeds a threshold, so as to re-determine the matching situation until a suitable matching pair is found. That is, until it is determined that the correlation between the first point cloud BEV features after N (N is a natural number greater than or equal to 1) offsets and the image BEV features exceeds the threshold, that is, a matching image BEV feature is found, and a matching pair is obtained.

[0111] Step S416: Determine the image BEV feature in the matching pair as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

[0112] In this embodiment, after obtaining a matching pair, the image BEV feature in the matching pair can be determined as: the image BEV feature that matches the first point cloud BEV feature in the matching pair in terms of spatial location. That is, the image BEV feature in the matching pair is the image BEV feature that matches the first point cloud BEV feature in the matching pair, thereby completing the search and matching of image BEV features and obtaining the matching image BEV feature that corresponds to the first point cloud BEV feature in the coordinate system.

[0113] In this embodiment, by using the neighborhood search network DSNet, a simple network structure is used to overcome the misalignment of high-speed targets in the multi-sensor feature fusion stage to a certain extent. By using network parameter learning, the corresponding features can automatically search for the most suitable relevant features in the spatial neighborhood, thereby improving the accuracy of feature correspondence in the pre-fusion algorithm.

[0114] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the feature search and matching process of a neighborhood search network according to an embodiment of the present invention. Figure 2In this process, the BEV features of the image are determined through the Domain Search Network (DSNet). The BEV features of the image remain unchanged during each round of matching. The BEV features of the laser point cloud are gradually shifted so that the shifted BEV features of the laser point cloud are successfully matched with the BEV features of the image. The matched BEV features of the image correspond to the BEV features of the laser point cloud in spatial location.

[0115] In one embodiment, such as Figure 3 As shown, Figure 3 This is a diagram illustrating a neighborhood search network structure according to an embodiment of the present invention. Figure 3 In this process, point cloud BEV features (LiDAR BEV Features) and image BEV features (Camera BEV Features) are first processed by the feature summing module in the neighborhood search network (i.e., Figure 3 In The two BEV features are added together. After addition, the correlation between the two BEV features is determined. If the correlation determines that they are not a normal matching pair, the offset of the edge of the point cloud BEV feature is learned iteratively. Then, the offset is added to the original point cloud BEV feature, which is equivalent to obtaining a new point cloud location box. The point cloud BEV feature with the changed position is added to the original image BEV feature again, and the matching is re-determined based on the correlation until a suitable matching pair is found. This means that a suitable image BEV feature has been found for the point cloud BEV feature. After obtaining a suitable matching pair, the feature addition result output by the feature addition module is input to the convolutional fusion module of the neighborhood search network (i.e., Figure 3 The Conv Fusion Module in the network performs BEV feature fusion to obtain the output of the entire neighborhood search network: fused BEV features.

[0116] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-sensor feature fusion method. In this method, step S42 specifically includes steps S421 to S425:

[0117] S421: The feature fusion part inputs the sum of the features of the matching pair into the 3x3 convolution kernel, encoder and decoder in sequence to obtain the first output.

[0118] In this embodiment, after obtaining the matching pair, the feature fusion part of the neighborhood search network can obtain the feature sum of the matching pair and input the feature sum into the 3x3 convolution kernel to obtain the output of the 3x3 convolution kernel. Then, the output of the 3x3 convolution kernel is input into the encoder and decoder to obtain the output of the decoder. In this embodiment, the first output is the output of the decoder.

[0119] S422: Perform feature fusion between the first output and the output of the 3x3 convolution kernel to obtain the second output.

[0120] In this embodiment, after obtaining the first output, the first output can be fused with the output of the 3x3 convolution kernel at the feature level. At this time, the feature fusion is not a simple feature addition, but a series of feature fusion operations such as averaging to obtain the second output.

[0121] S423: After reducing the dimensionality of the second output by inputting a 1x1 convolution kernel, the third output is obtained.

[0122] In this embodiment, after obtaining the second output, the second output is passed through a 1X1 convolution kernel for feature dimensionality reduction. That is, after the second output is input into the 1X1 convolution kernel for dimensionality reduction, the third output of the 1X1 convolution kernel is obtained.

[0123] S424: The third output is filtered by the judge to obtain the fourth output.

[0124] In this embodiment, after obtaining the third output, it will pass through a judge. This judge is used to determine which elements in the dimensionality-reduced third output are needed and to perform filtering processing on the third output. For example, a threshold value is set. If the element is greater than the threshold value, it indicates that the element is needed. If the element is not greater than the threshold value, it indicates that the element is not needed. This increases the weight of the needed elements and decreases the weight of the unnecessary elements, and a new matrix is ​​obtained, which is the fourth output.

[0125] S425: The fourth output is fused with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

[0126] In this embodiment, after obtaining the fourth output, the fourth output is fused with the output of the 3x3 convolution kernel again to obtain the final output of the neighborhood search network: fused BEV features.

[0127] In one embodiment, the neighborhood search network is not trained alone, but is trained together with the models of two branches (image BEV feature extraction branch and point cloud BEV feature extraction branch). The final output of the neighborhood search network, which fuses BEV features, also passes through a detection head, and the prediction results of the detection head are used to tune the parameters of the entire three models.

[0128] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-sensor feature fusion method. In this method, step S1 may specifically include steps S11 to S13:

[0129] Step S11: Acquire multiple image data captured by the vehicle's panoramic camera.

[0130] In this embodiment, the visual sensor is a panoramic camera on the vehicle, which is a multi-view panoramic camera, such as a 6-view or 4-view panoramic camera. Thus, this embodiment can acquire multiple image data captured by the vehicle's panoramic camera.

[0131] Step S12: Using the backbone network Dual-Swin-Tiny and FPN network in the improved LSS algorithm, 2D features are extracted and enhanced from the multiple image data to obtain multi-view 2D feature maps.

[0132] In this embodiment, the LSS algorithm for BEV feature extraction in images is described. The LSS algorithm was initially proposed for BEV semantic segmentation rather than 3D object detection, therefore its performance is poor when used directly. To adapt it to 3D object detection tasks, this embodiment optimizes deployment time and feature extraction, and modifies part of the LSS algorithm's structure to better suit the traditional 3D Bounding Box detection task.

[0133] In other words, this embodiment improves upon the traditional LSS algorithm, extracting BEV features from images using the improved LSS algorithm. Specifically, the image backbone of the LSS algorithm in this embodiment mainly consists of a 2D backbone network for basic feature extraction and a module for scale variables. A Dual-Swin-Tiny network is used as the 2D backbone network in the improved LSS algorithm, and a standard Feature Pyramid Network (FPN) is used after the Dual-Swin-Tiny backbone network to extract features at multiple scale resolutions, thereby enhancing features. Dual-Swin-Tiny is essentially CBNet, an improved version of the ResNet network. This model is more suitable for multimodal feature fusion, that is, more suitable for extracting 2D image features, and the goal of the extracted features is to convert them into BEV. It mainly uses multiple combinations of classic networks to construct new networks, which can extract more effective features and eliminates the need for ImageNet pre-training, allowing direct use of pre-trained parameters from other networks (ResNet or ResNeXt), making it simpler and more efficient. Therefore, this embodiment can extract 2D features from multiple image data and enhance the 2D features through the backbone network Dual-Swin-Tiny and FPN network in the improved LSS algorithm, and obtain multi-view 2D feature maps.

[0134] Step S13: Based on the vehicle's own coordinate system, project the multi-view 2D feature map onto the vehicle's own coordinate system to obtain the image BEV features.

[0135] In this embodiment, after obtaining the multi-view 2D feature map, the multi-view 2D feature map can be projected onto a 3D coordinate system. Specifically, the multi-view 2D feature map can be projected onto the vehicle coordinate system, thus transforming the multi-view 2D feature map into the vehicle's coordinate system, thereby obtaining the BEV image features. Here, the vehicle's coordinate system is the coordinate system of the BEV image features.

[0136] In this embodiment, the extraction of BEV features from an image is based on the improved LSS algorithm, which can improve the extraction efficiency and accuracy of BEV features.

[0137] like Figure 4 As shown, Figure 4 This is a branch structure diagram for image BEV feature extraction, as illustrated in one embodiment of the present invention. Figure 4 In the process, multiple image data are acquired and processed through a 2D backbone network (i.e., Figure 4 2D Backbone) Dual-Swin-Tiny Network and Pyramid Network FPN (i.e. Figure 4 After FPN w / ADP, the extracted 2D features are obtained; among them, Figure 4 The dashed box in the image refers to the feature extraction stage; then the 2D features are projected into 3D (i.e., ...). Figure 4 (2D→3DProjector), based on 3D vehicle information (i.e. Figure 4 3D ego-car coordinate in and BEV coding (i.e. Figure 4 The BEV Encoder in the image is used to obtain the BEV features (i.e., BEV features). Figure 4 (Camera BEV Features).

[0138] like Figure 5 As shown, Figure 5 This is a diagram illustrating the backbone network structure of an image BEV feature extraction branch according to an embodiment of the present invention. In this embodiment, the backbone network of the image BEV feature extraction branch is a Dual-Swin-Tiny network. Figure 5 As can be seen from the diagram, the Dual-Swin-Tiny network is a standard inverted pyramid network. In order to better align these features of different resolutions, a simple feature adaptation module (ADP) is used to refine the sampled features. That is, before connecting the feature layers of different resolutions, adaptive average pooling and 1*1 convolution are used for each upsampled feature layer.

[0139] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-sensor feature fusion method. In this method, step S2 may include steps S21 and S22:

[0140] Step S21: Acquire the point cloud data captured by the vehicle's point cloud device.

[0141] In this embodiment, the radar sensor on the vehicle is a point cloud device, such as a lidar sensor, a millimeter-wave radar sensor, etc., and then the point cloud data captured by the vehicle's point cloud device is acquired.

[0142] Step S22: By learning the parameterized hypervoxel method of the original points, the point cloud data is converted into the initial point cloud BEV features.

[0143] In this embodiment, point cloud data can be converted into the initial point cloud BEV features based on the parameterized supervoxel method of learning the original points, thereby reducing the dimensionality and effectively generating point cloud BEV features in the BEV space using a network composed of sparse 3D convolutions.

[0144] like Figure 6 As shown, Figure 6 This is a backbone network structure diagram of a point cloud BEV feature extraction branch according to an embodiment of the present invention. In this embodiment, the point cloud BEV feature extraction branch adopts the CenterPoint method. Figure 6 In the process, point cloud data first passes through a 3D backbone network (i.e., Figure 6 The point cloud BEV features are obtained from the 3D Backbone, and then passed through the detection head (i.e. Figure 6 The output of Det.Head.

[0145] In conjunction with the above embodiments, in one implementation, this invention also provides a multi-sensor feature fusion method. In this method, the target detection head can use detection heads from multiple 3D target detection methods, and the target detection head may include at least one of: a CenterPoint detection head, a PointPillars detection head, and a TransFusion-L detection head.

[0146] In another embodiment, such as Figure 7 As shown, Figure 7 This is an overall architecture design diagram of a multi-sensor feature fusion scheme shown in one embodiment of the present invention.

[0147] exist Figure 7In this model, the image BEV feature extraction branch and the point cloud BEV feature extraction branch are completely independent. After each branch completes its feature extraction, the features are combined through a fusion module. Finally, the combined features are used for model inference to detect target information. Figure 7 The upper part is the image BEV feature extraction branch: the acquired multiple image data are processed by a 2D backbone network (i.e., Figure 7 2D Backbone) Dual-Swin-Tiny Network and Pyramid Network FPN (i.e. Figure 7 After FPN w / ADP, the extracted 2D features are obtained; among them, Figure 7 The dashed boxes surrounding the 2D backbone network and pyramid network FPN refer to the feature extraction stage; then the 2D features are projected from 2D to 3D (i.e., Figure 7 (2D→3D Projector), based on 3D vehicle information (i.e. Figure 7 3D ego-car coordinate in and BEV coding (i.e. Figure 7 The BEV Encoder in the image is used to obtain the BEV features (i.e., BEV features). Figure 7 (Camera BEV Features).

[0148] Figure 7 The left half of the lower section is the point cloud BEV feature extraction branch, which passes through the 3D backbone network (i.e. Figure 7 The point cloud BEV features are extracted from the 3DBackBone. Figure 7 The right half of the lower section is the neighborhood search network (i.e. Figure 7 Domain Search Network (DSM) and multiple detectors (i.e., Figure 7 The Multi-Det Head (MDT) is used to search and match the received image BEV features and point cloud BEV features. Then, the image BEV features and point cloud BEV features that successfully match in spatial location are fused to obtain fused BEV features. The multi-det head makes predictions based on the fused BEV features output by the neighborhood search network and infers the target detection results.

[0149] This embodiment provides a feature fusion scheme that allows sensors to operate independently, has high feature matching degree, high detection accuracy, and high feasibility of deployment. It not only makes up for the lack of features of single-modal sensors such as cameras and LiDAR, but also enables the fused target detection algorithm to achieve a balance between performance and real-time performance.

[0150] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0151] Based on the same inventive concept, one embodiment of the present invention provides a multi-sensor feature fusion system 800. (Reference) Figure 8 , Figure 8 This is a structural block diagram of a multi-sensor feature fusion system provided in an embodiment of the present invention. Figure 8 As shown, the system 800 includes:

[0152] The image feature extraction module 801 is used to obtain the BEV features of the image based on the image data;

[0153] The point cloud feature extraction module 802 is used to obtain the initial point cloud BEV features based on the point cloud data.

[0154] The feature conversion module 803 is used to convert the initial point cloud BEV features to the coordinate system of the image BEV features to obtain the first point cloud BEV features.

[0155] The feature fusion module 804 is used to determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and to fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature.

[0156] The detection head module 805 is used to input the fused BEV features into the target detection head for target detection.

[0157] Optionally, the feature fusion module 804 includes:

[0158] The feature matching submodule is used to search for the image BEV features through the search matching part of the neighborhood search network, and to match the image BEV features that correspond to the first point cloud BEV features in the coordinate system.

[0159] The feature fusion submodule is used to fuse the first point cloud BEV features with the matched image BEV features through the feature fusion part of the neighborhood search network to obtain the fused BEV features.

[0160] Optionally, the feature matching submodule includes:

[0161] The feature addition submodule is used to add the first point cloud BEV features and the image BEV features through the search matching part, and determine whether the correlation between the BEV features after feature addition exceeds a threshold.

[0162] The first threshold determination submodule is used to obtain a matching pair if the correlation degree exceeds the threshold.

[0163] The second threshold determination submodule is used to perform iterative learning to determine the offset of the first point cloud BEV feature if the correlation degree does not exceed the threshold.

[0164] The offset submodule is used to determine the offset first point cloud BEV features based on the first point cloud BEV features and the offset amount.

[0165] The judgment submodule is used to add the BEV features of the offset first point cloud to the BEV features of the image until it is determined that the correlation exceeds the threshold, and thus obtain the matching pair;

[0166] The BEV feature matching submodule is used to determine the image BEV feature in the matching pair as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

[0167] Optionally, the feature fusion submodule includes:

[0168] The first output submodule is used to input the sum of the features of the matching pair into a 3x3 convolutional kernel, encoder and decoder in sequence through the feature fusion part to obtain the first output;

[0169] The second output submodule is used to fuse the first output with the output of the 3x3 convolutional kernel to obtain the second output.

[0170] The third output submodule is used to reduce the dimensionality of the second output input 1X1 convolution kernel to obtain the third output;

[0171] The fourth output submodule is used to filter the third output through a judge to obtain the fourth output;

[0172] The BEV feature fusion submodule is used to fuse the fourth output with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

[0173] Optionally, the image feature extraction module 801 includes:

[0174] The image data acquisition submodule is used to acquire multiple image data captured by the vehicle's panoramic camera;

[0175] The 2D feature acquisition submodule is used to extract and enhance 2D features from the multiple image data through the backbone network Dual-Swin-Tiny and FPN network in the improved LSS algorithm, so as to obtain multi-view 2D feature maps.

[0176] The image BEV feature acquisition submodule is used to project the multi-view 2D feature map onto the vehicle coordinate system based on the vehicle's own coordinate system to obtain the image BEV features.

[0177] Wherein, the vehicle coordinate system is the coordinate system of the BEV feature in the image.

[0178] Optionally, the point cloud feature extraction module 802 includes:

[0179] The point cloud data acquisition submodule is used to acquire the point cloud data captured by the vehicle's point cloud device.

[0180] The point cloud BEV feature acquisition submodule is used to convert the point cloud data into the initial point cloud BEV features by learning the parameterized hypervoxel method of the original points.

[0181] Optionally, the target detection head includes at least one of the following: a CenterPoint detection head, a PointPillars detection head, and a TransFusion-L detection head.

[0182] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the multi-sensor feature fusion method as described in any of the above embodiments of the present invention.

[0183] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 900, such as... Figure 9 As shown. Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 902, a processor 901, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the multi-sensor feature fusion method described in any of the above embodiments of the present invention.

[0184] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0185] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0186] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0191] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0192] The foregoing has provided a detailed description of the multi-sensor feature fusion method, system, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-sensor feature fusion method, characterized in that, The method includes: Based on the image data, obtain the BEV features of the image; Based on the point cloud data, obtain the initial point cloud BEV features; The initial point cloud BEV features are transformed to the coordinate system of the image BEV features to obtain the first point cloud BEV features. Determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature. The fused BEV features are input into the target detection head for target detection; the step of determining the image BEV features that match the first point cloud BEV features in the coordinate system, and fusing the first point cloud BEV features with the matched image BEV features to obtain the fused BEV features includes: The search and matching part of the neighborhood search network is used to search for the image BEV features and match the image BEV features that correspond to the first point cloud BEV features in the coordinate system. The feature fusion part of the neighborhood search network fuses the first point cloud BEV features with the matched image BEV features to obtain the fused BEV features. The step of searching for the image BEV features through the search and matching part of the neighborhood search network, and matching the image BEV features corresponding to the first point cloud BEV features in the coordinate system, includes: The search and matching part adds the first point cloud BEV features and the image BEV features together, and determines whether the correlation between the BEV features after feature addition exceeds a threshold. If the correlation exceeds the threshold, a matching pair is obtained; If the correlation does not exceed the threshold, iterative learning is performed to determine the offset of the first point cloud BEV feature; Based on the first point cloud BEV features and the offset, determine the offset first point cloud BEV features; The BEV features of the offset first point cloud are added to the BEV features of the image until the correlation exceeds the threshold, thus obtaining the matching pair. The image BEV feature in the matching pair is determined as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

2. The multi-sensor feature fusion method according to claim 1, characterized in that, The feature fusion part of the neighborhood search network fuses the first point cloud BEV features with the matched image BEV features to obtain the fused BEV features, including: The feature fusion part inputs the sum of the features of the matching pair into a 3x3 convolutional kernel, an encoder, and a decoder in sequence to obtain the first output; The first output is fused with the output of the 3x3 convolution kernel to obtain the second output; After reducing the dimensionality of the second output by inputting it into a 1x1 convolution kernel, the third output is obtained. The third output is filtered by the judge to obtain the fourth output; The fourth output is fused with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

3. The multi-sensor feature fusion method according to claim 1, characterized in that, The step of obtaining image BEV features based on image data includes: Acquire multiple image data captured by the vehicle's panoramic camera; By using the Dual-Swin-Tiny backbone network and FPN network in the improved LSS algorithm, 2D features are extracted and enhanced from the multiple image data to obtain multi-view 2D feature maps. Based on the vehicle's own coordinate system, the multi-view 2D feature map is projected onto the vehicle's own coordinate system to obtain the image BEV features; Wherein, the vehicle coordinate system is the coordinate system of the BEV feature in the image.

4. The multi-sensor feature fusion method according to claim 1, characterized in that, The step of obtaining initial point cloud BEV features based on point cloud data includes: Acquire the point cloud data captured by the vehicle's point cloud device; The point cloud data is converted into the initial point cloud BEV features by learning the parameterized hypervoxel method of the original points.

5. The multi-sensor feature fusion method according to claim 1, characterized in that, The target detection head includes at least one of the following: CenterPoint detection head, PointPillars detection head, and TransFusion-L detection head.

6. A multi-sensor feature fusion system, characterized in that, The system includes: The image feature extraction module is used to obtain the BEV features of an image based on the image data. The point cloud feature extraction module is used to obtain the initial point cloud BEV features based on the point cloud data. The feature transformation module is used to transform the initial point cloud BEV features to the coordinate system of the image BEV features to obtain the first point cloud BEV features. The feature fusion module is used to determine the image BEV feature that matches the first point cloud BEV feature in the coordinate system, and to fuse the first point cloud BEV feature with the matched image BEV feature to obtain the fused BEV feature. The detection head module is used to input the fused BEV features into the target detection head for target detection; The feature fusion module includes: The feature matching submodule is used to search for the image BEV features through the search matching part of the neighborhood search network, and to match the image BEV features that correspond to the first point cloud BEV features in the coordinate system. The feature fusion submodule is used to fuse the first point cloud BEV features with the matched image BEV features through the feature fusion part of the neighborhood search network to obtain the fused BEV features. The feature matching submodule includes: The feature addition submodule is used to add the first point cloud BEV features and the image BEV features through the search matching part, and determine whether the correlation between the BEV features after feature addition exceeds a threshold. The first threshold determination submodule is used to obtain a matching pair if the correlation degree exceeds the threshold. The second threshold determination submodule is used to perform iterative learning to determine the offset of the first point cloud BEV feature if the correlation degree does not exceed the threshold. The offset submodule is used to determine the offset first point cloud BEV features based on the first point cloud BEV features and the offset amount. The judgment submodule is used to add the BEV features of the offset first point cloud to the BEV features of the image until it is determined that the correlation exceeds the threshold, and thus obtain the matching pair; The BEV feature matching submodule is used to determine the image BEV feature in the matching pair as the image BEV feature that matches the first point cloud BEV feature in the matching pair in the coordinate system.

7. The multi-sensor feature fusion system according to claim 6, characterized in that, The feature fusion submodule includes: The first output submodule is used to input the sum of the features of the matching pair into a 3x3 convolutional kernel, encoder and decoder in sequence through the feature fusion part to obtain the first output; The second output submodule is used to fuse the first output with the output of the 3x3 convolutional kernel to obtain the second output. The third output submodule is used to reduce the dimensionality of the second output input 1X1 convolution kernel to obtain the third output; The fourth output submodule is used to filter the third output through a judge to obtain the fourth output; The BEV feature fusion submodule is used to fuse the fourth output with the output of the 3x3 convolution kernel to obtain the fused BEV feature.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the multi-sensor feature fusion method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-sensor feature fusion method as described in any one of claims 1 to 5.

Images