A multi-source sensor fusion method, device and equipment based on feature distillation

Abstract

The application discloses a multi-source sensor fusion method and device based on feature distillation, equipment and a readable storage medium, and relates to the technical field of data fusion. The method comprises the following steps: first, obtaining a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map; then, performing up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map; finally, distilling the features of the first feature map to the second feature map, and fusing the first feature map and the second feature map to obtain a fused feature map. The features of the high-performance sensor guide the feature extraction of the low-performance sensor, the perception accuracy of the low-performance sensor is improved, and the consistency of the effect of a subsequent perception algorithm platform on high-performance sensors and low-performance sensors is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion, in particular to a multi-source sensor fusion method, device and equipment based on feature distillation and a readable storage medium. BACKGROUND

[0002] The intelligent networked automobile platform supports multiple vehicle models and various sensor configurations. High-performance sensor configurations and low-performance sensor configurations coexist in the intelligent networked automobile perception algorithm platform support list. The differences in resolution and sampling rate between low-performance sensor combinations (such as low-cost millimeter wave radar, low-precision laser radar, and low-precision camera) and high-performance sensor combinations (such as high-precision laser radar and high-precision camera) result in inconsistent perception results, which is one of the important problems that the intelligent networked automobile platform perception algorithm needs to solve.

[0003] Existing multi-modal fusion methods do not compensate for hardware differences, resulting in large differences in the effects of fusion algorithms under different sensor combinations. Therefore, there is an urgent need for a multi-source sensor fusion method based on feature distillation that can overcome the above-mentioned defects. SUMMARY

[0004] The present application relates to the technical field of data fusion, in particular to a multi-source sensor fusion method, device and equipment based on feature distillation and a readable storage medium.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a multi-source sensor fusion method based on feature distillation, which comprises:

[0007] obtaining a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map;

[0008] performing up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map;

[0009] distilling the features of the first feature map to the second feature map, and fusing the first feature map and the second feature map to obtain a fused feature map.

[0010] In some embodiments, distilling the features of the first feature map to the second feature map comprises:

[0011] extracting backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features;

[0012] distilling the first backbone features to the second backbone features based on activation feature distillation;

[0013] extracting first multi-scale features and second multi-scale features in the first backbone features and the second backbone features;

[0014] distilling the first multi-scale features to the second multi-scale features based on candidate box feature distillation.

[0015] In some embodiments, distilling the first backbone features to the second backbone features based on activation feature distillation comprises:

[0016] calculating an activation mask of distilling the first backbone features to the second backbone features;

[0017] generating a distillation region mask based on the activation mask of distilling the first backbone features to the second backbone features;

[0018] generating a distillation loss weight based on the distillation region mask, and generating a loss function of activation distillation to realize distilling the first backbone features to the second backbone features.

[0019] In some embodiments, distilling the first multi-scale features to the second multi-scale features based on candidate box feature distillation comprises:

[0020] generating a candidate box based on the first multi-scale features and the second multi-scale features;

[0021] generating a distillation loss weight and a loss function based on the candidate box feature based on the candidate box;

[0022] training a total loss function of the second sensor to realize distilling the first multi-scale features to the second multi-scale features.

[0023] In some embodiments, upsampling the second feature map comprises:

[0024] interpolating, de-pooling and de-convolving the second feature map.

[0025] In some embodiments, the feature method further comprises:

[0026] establishing a relationship mapping table between the sensors and the resolutions;

[0027] obtaining at least two candidate feature maps based on at least two candidate sensors;

[0028] determining at least one group of sensors from the at least two candidate sensors based on the resolutions of the at least two feature maps; each group of sensors includes the first sensor and the second sensor.

[0029] In a second aspect, the present application further provides a multi-source sensor fusion device based on feature distillation, comprising:

[0030] a feature extraction module, configured to acquire a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map;

[0031] a feature sampling module, configured to perform up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to that of the first feature map;

[0032] a feature distillation module, configured to distill the features of the first feature map to the second feature map, and fuse the first feature map and the second feature map to obtain a fused feature map.

[0033] In a third aspect, the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-source sensor fusion method based on feature distillation provided in the first aspect when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the multi-source sensor fusion method based on feature distillation provided in the first aspect.

[0035] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the multi-source sensor fusion method based on feature distillation provided in the first aspect.

[0036] The multi-source sensor fusion method based on feature distillation provided in the present application acquires a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map; then performs up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to that of the first feature map; finally distills the features of the first feature map to the second feature map, and fuses the first feature map and the second feature map to obtain a fused feature map. The features of the high-performance sensor guide the feature extraction of the low-performance sensor, improve the perception accuracy of the low-performance sensor, and improve the consistency of the subsequent perception algorithm platform for high and low performance sensors.

[0037] The above description is only a summary of the technical solutions of the present application. In order to make the technical means of the present application clearer and can be implemented according to the content of the description, the preferred embodiments of the present application are described in detail as follows with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flowchart of a multi-source sensor fusion method based on feature distillation is shown in an embodiment of the present application.

[0039] Figure 2 A flowchart of another multi-source sensor fusion method based on feature distillation is shown in an embodiment of the present application.

[0040] Figure 3 A structural schematic diagram of a multi-source sensor fusion device based on feature distillation is shown in an embodiment of the present application.

[0041] Figure 4 A structural schematic diagram of another multi-source sensor fusion device based on feature distillation is shown in an embodiment of the present application.

[0042] Figure 5 An electronic device structure schematic diagram is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0044] It should be noted that the description of "one embodiment", "embodiment", "example embodiment" and the like in the specification means that the described embodiment can include a specific feature, structure or characteristic, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not mean the same embodiment. Further, when a specific feature, structure or characteristic is described in combination with an embodiment, it is indicated that such a feature, structure or characteristic is combined with other embodiments within the knowledge of those skilled in the art, whether or not it is explicitly described.

[0045] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as there is no conflict.

[0046] In some embodiments, as shown in Figure 1 A multi-source sensor fusion method based on feature distillation is provided, and the specific method includes:

[0047] S101, acquire a first feature map extracted by a first sensor and a second feature map extracted by a second sensor.

[0048] The performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map.

[0049] Specifically, when sensor fusion is performed, a high-performance sensor and a low-performance sensor are required for fusion necessity, and the process of extracting a feature map by a sensor can be based on point cloud or camera to extract a feature map.

[0050] The method of extracting a feature map based on point cloud mainly includes: extracting a feature map based on a Voxel-Based point cloud extraction backbone network or a Pollar-Based point cloud extraction backbone network.

[0051] The specific steps of the Voxel-Based point cloud extraction backbone network are: a. voxelization processing of the laser radar point cloud; b. voxel feature coding; c. multi-scale feature extraction of the voxel feature coded features by using a 3D convolution backbone network; and d. projecting the features extracted by the 3D convolution backbone network into a BEV space to obtain BEV features.

[0052] The specific steps of the Pillar-Based point cloud extraction backbone network are: a. voxelization processing of the laser radar point cloud; b. voxel feature coding; c. projecting the results of the voxel feature coding into a BEV space to obtain BEV features.

[0053] The method of extracting a feature map based on a camera mainly includes: LSS (Lift, Splat, Shoot) and a mapping method based on Transformer.

[0054] The specific steps of the LSS method are: Lift: predicting a depth distribution (such as a discrete depth interval probability) for each pixel; Splat: mapping image features to a 3D point cloud through internal and external parameters, and voxelizing to generate BEV features; and performing task reasoning in the BEV space.

[0055] The specific steps of the mapping method based on Transformer are: constructing a query (Query) region in the BEV space; each of the multiple cameras performs high-level feature extraction through a backbone network. The query (Query) and the multi-view high-level features perform attention interaction, and aggregate the information of the multiple cameras.

[0056] To better determine the first sensor and the second sensor, it is further needed to: establish a relationship mapping table between sensors and resolutions; obtain at least two candidate feature maps based on the at least two candidate sensors; determine at least one group of sensors from the at least two candidate sensors based on resolutions of the at least two feature maps; and each group of sensors includes the first sensor and the second sensor.

[0057] Specifically, since there are many types of sensors, for example, when the number of candidate sensors is three, the candidate sensor corresponding to the highest resolution can form two groups of sensors with the other two candidate sensors respectively, and the candidate sensor corresponding to the highest resolution is the first sensor in the two groups of sensors, and the other two candidate sensors are the second sensors.

[0058] S102, performing up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map.

[0059] Optionally, the up-sampling processing on the second feature map includes interpolation processing, de-pooling processing and de-convolution processing.

[0060] Specifically, the resolution of the second feature map can be adjusted according to the relationship mapping table between sensors and resolutions to increase the resolution to be equal to the resolution of the first feature map.

[0061] S103, performing feature distillation from the first feature map to the second feature map, and fusing the first feature map and the second feature map to obtain a fused feature map.

[0062] Optionally, the fusion process includes activation-based feature distillation and bounding box-based feature distillation, specifically: extracting backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features; performing activation-based feature distillation to distill the first backbone features to the second backbone features; extracting first multi-scale features and second multi-scale features in the first backbone features and the second backbone features; and performing bounding box-based feature distillation to distill the first multi-scale features to the second multi-scale features.

[0063] Specifically, the first backbone feature is distilled to the second backbone feature based on activation feature distillation, including: calculating an activation mask of the first backbone feature distilled to the second backbone feature; generating a distillation region mask based on the activation mask of the first backbone feature distilled to the second backbone feature; generating a distillation loss weight based on the distillation region mask, and generating a loss function of activation distillation, to realize the distillation of the first backbone feature to the second backbone feature. The feature is characterized in that the first multi-scale feature is distilled to the second multi-scale feature based on candidate box feature distillation, including: generating a candidate box based on the first multi-scale feature and the second multi-scale feature; generating a distillation loss weight based on the candidate box and a loss function based on the candidate box feature; training a total loss function of the second sensor to realize the distillation of the first multi-scale feature to the second multi-scale feature.

[0064] Exemplarily, the backbone features of the first feature map and the second feature map can be extracted by using a 2D CNN backbone network (such as ResNet, VGGNet, RegNet, etc.), to obtain the first backbone feature and the second backbone feature. The activation mask of the first backbone feature distilled to the second backbone feature is generated by using the following formula:

[0065] ;

[0066] ;

[0067] wherein, represents a feature vector, i , j is a spatial dimension index (height, width direction), c is a channel index, (high-precision, low-precision sensor).

[0068] The distillation region mask is generated again:

[0069] ;

[0070] The distillation loss weight is generated again:

[0071] ;

[0072] The loss function of activation distillation is generated again:

[0073] ;

[0074] The first multi-scale feature and the second multi-scale feature are extracted from the first backbone feature and the second backbone feature, and the candidate box (denoted as and ) is generated based on the first multi-scale feature and the second multi-scale feature. The distillation loss weight is generated based on the candidate box:

[0075] ;

[0076] The loss function based on the candidate box features is generated based on the candidate box:

[0077] ;

[0078] wherein, , represents a feature vector, i, j is a spatial dimension index (height, width direction), and c is a channel index, .

[0079] The total loss function of training the second sensor is:

[0080] ;

[0081] wherein, , respectively refer to the loss function based on activation distillation and the loss function based on the candidate box features; refers to the loss function of a normal training task; , is a constant weight set.

[0082] Finally, the first feature map and the second feature map are fused to obtain a fused feature map for subsequent decision generation.

[0083] The multi-source sensor fusion method based on feature distillation in the above embodiment first acquires a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map; then the second feature map is up-sampled to increase the resolution of the second feature map to be equal to that of the first feature map; finally, the features of the first feature map are distilled to the second feature map, and the first feature map and the second feature map are fused to obtain a fused feature map. The features of the high-performance sensor guide the feature extraction of the low-performance sensor, improve the perception accuracy of the low-performance sensor, and improve the consistency of the subsequent perception algorithm platform for high and low performance sensors.

[0084] In order to more comprehensively show the present scheme, the present embodiment gives an optional way of the multi-source sensor fusion method based on feature distillation, as shown in Figure 2 .

[0085] S201, a relationship mapping table between sensors and resolutions is established.

[0086] S202, at least two candidate feature maps are acquired based on at least two candidate sensors.

[0087] S203, determining at least one group of sensors from the at least two candidate sensors based on resolutions of the at least two feature maps.

[0088] wherein the first sensor and the second sensor are included in each group of sensors.

[0089] S204, obtaining a first feature map extracted by the first sensor and a second feature map extracted by the second sensor.

[0090] wherein the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map.

[0091] S205, performing interpolation processing, de-pooling processing and de-convolution processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map.

[0092] S206, extracting backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features.

[0093] S207, calculating an activation mask of distillation of the first backbone features to the second backbone features.

[0094] S208, generating a distillation region mask based on the activation mask of distillation of the first backbone features to the second backbone features.

[0095] S209, generating a distillation loss weight based on the distillation region mask, and generating a loss function of activation distillation to realize distillation of the first backbone features to the second backbone features.

[0096] S210, extracting first multi-scale features and second multi-scale features in the first backbone features and the second backbone features.

[0097] S211, generating a candidate box based on the first multi-scale features and the second multi-scale features.

[0098] S212, generating a distillation loss weight and a loss function based on candidate box features based on the candidate box.

[0099] S213, training a total loss function of the second sensor to realize distillation of the first multi-scale features to the second multi-scale features.

[0100] S214, fusing the first feature map and the second feature map to obtain a fused feature map.

[0101] The specific process of S201-S214 described above can refer to the description of the method embodiments described above, and the implementation principle and technical effects are similar, which will not be repeated here.

[0102] Based on the same inventive concept, the embodiments of the present application also provide a feature distillation based multi-source sensor fusion device for implementing the feature distillation based multi-source sensor fusion method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more feature distillation based multi-source sensor fusion device embodiments provided below can be referred to the limitations of the feature distillation based multi-source sensor fusion method described above, which will not be repeated here.

[0103] In one embodiment, as shown in Figure 3 , a feature distillation based multi-source sensor fusion device is provided, which comprises:

[0104] a feature extraction module 30 configured to obtain a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map;

[0105] a feature sampling module 31 configured to perform up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map;

[0106] a feature distillation module 32 configured to distill the features of the first feature map to the second feature map, and fuse the first feature map and the second feature map to obtain a fused feature map.

[0107] In another embodiment, as shown in Figure 4 , the feature distillation module 32 in the above Figure 3 comprises:

[0108] a first extraction unit 320 configured to extract backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features;

[0109] a first distillation unit 321 configured to distill the first backbone features to the second backbone features based on activation feature distillation;

[0110] a second extraction unit 322 configured to extract first multi-scale features and second multi-scale features in the first backbone features and the second backbone features;

[0111] a second distillation unit 323 configured to distill the first multi-scale features to the second multi-scale features based on candidate box feature distillation.

[0112] In another embodiment, the above Figure 4The first distillation unit 321 is specifically used for: calculating the activation mask of the first backbone feature distilled to the second backbone feature; generating a distillation region mask based on the activation mask of the first backbone feature distilled to the second backbone feature; generating distillation loss weights based on the distillation region mask, and generating a loss function for activation distillation, so as to realize the distillation of the first backbone feature to the second backbone feature.

[0113] In another embodiment, the above Figure 4 The second distillation unit 323 is specifically used for: generating candidate boxes based on the first multi-scale features and the second multi-scale features; generating distillation loss weights and loss functions based on the candidate boxes; and training the total loss function of the second sensor to achieve distillation of the first multi-scale features into the second multi-scale features.

[0114] In another embodiment, the above Figure 3 The feature sampling module 31 is specifically used for: interpolating, unpooling, and deconvolution processing the second feature map. In another embodiment, the above... Figure 3 The feature-based multi-source sensor fusion device is also specifically used for: establishing a mapping table between sensors and resolution; acquiring at least two candidate feature maps based on at least two candidate sensors; determining at least one group of sensors from at least two candidate sensors based on the resolution of at least two feature maps; each group of sensors includes a first sensor and a second sensor.

[0115] This application also provides an electronic device, in some embodiments, referring to... Figure 5 As shown, the electronic device 700 includes an input unit 710, a memory 720, a processor 730, and an output unit 740. The memory 720 stores program instructions that can be executed on the processor 730. The processor 730 can execute the multi-source sensor fusion method and / or technical solution based on feature distillation in the foregoing embodiments by calling the program instructions. The electronic device 700 can be a mobile terminal device such as a mobile phone or a computer.

[0116] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program that performs a multi-source sensor fusion method based on feature distillation. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions that invoke the methods of this application may be stored in a fixed or removable storage medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in a storage medium that operates according to the program instructions.

[0117] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by universal computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and can be realized by program codes executable by the computing devices, so as to be stored in storage devices and executed by the computing devices, or be respectively manufactured into integrated circuit modules, or be manufactured into a single integrated circuit module to realize multiple modules or steps. Therefore, the present application is not limited to any specific combination of hardware and software.

[0118] The technical features of the above embodiments can be integrated in any manner. In order to make the description simple, all possible integrations of the technical features in the above embodiments are not described, however, as long as the integration of the technical features does not exist contradictions, it should be considered as the scope of the present application.

[0119] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-source sensor fusion method based on feature distillation, characterized in that, The method comprises: obtaining a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map; performing up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map; extracting backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features; performing activation feature distillation to distill the first backbone features to the second backbone features; extracting first multi-scale features and second multi-scale features in the first backbone features and the second backbone features; performing candidate box feature distillation to distill the first multi-scale features to the second multi-scale features, and fusing the first feature map and the second feature map to obtain a fused feature map.

2. The feature distillation based multi-source sensor fusion method of claim 1, wherein, The activation feature distillation from the first backbone features to the second backbone features comprises: calculating an activation mask for the distillation of the first backbone features to the second backbone features; generating a distillation area mask based on the activation mask for the distillation of the first backbone features to the second backbone features; generating a distillation loss weight based on the distillation area mask, and generating a loss function for activation distillation to realize the distillation of the first backbone features to the second backbone features.

3. The feature distillation based multi-source sensor fusion method of claim 1, wherein, The candidate box feature distillation from the first multi-scale features to the second multi-scale features comprises: generating a candidate box based on the first multi-scale features and the second multi-scale features; generating a distillation loss weight and a loss function based on the candidate box feature based on the candidate box; training a total loss function of the second sensor to realize the distillation of the first multi-scale features to the second multi-scale features.

4. The feature distillation based multi-source sensor fusion method of claim 1, wherein, The up-sampling processing on the second feature map comprises: performing interpolation processing, de-pooling processing and de-convolution processing on the second feature map.

5. The feature distillation based multi-source sensor fusion method of claim 1, wherein, The method further comprises: establishing a relationship mapping table between sensors and resolutions; obtaining at least two candidate feature maps based on at least two candidate sensors; determining at least one group of sensors from the at least two candidate sensors based on the resolutions of the at least two candidate feature maps; each group of sensors includes a first sensor and a second sensor.

6. A multi-source sensor fusion apparatus based on feature distillation, characterized by, The device comprises: a feature extraction module configured to obtain a first feature map extracted by a first sensor and a second feature map extracted by a second sensor; the performance of the first sensor is higher than that of the second sensor, and the resolution of the first feature map is higher than that of the second feature map; a feature sampling module configured to perform up-sampling processing on the second feature map to increase the resolution of the second feature map to be equal to the resolution of the first feature map; The feature distillation module is configured to extract backbone features of the first feature map and the second feature map to obtain first backbone features and second backbone features; perform activation feature distillation to distill the first backbone features to the second backbone features; extract first multi-scale features and second multi-scale features from the first backbone features and the second backbone features; perform candidate box feature distillation to distill the first multi-scale features to the second multi-scale features, and fuse the first feature map and the second feature map to obtain a fused feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the feature distillation based multi-source sensor fusion method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the feature distillation based multi-source sensor fusion method in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the feature distillation based multi-source sensor fusion method in any one of claims 1 to 5.

Images