Multi-modal fusion 3D target detection method and system based on personalized federal learning

Abstract

The invention discloses a multi-modal fusion 3D target detection method and system based on personalized federal learning, and the method comprises the steps: collecting data information around a plurality of user vehicles, and carrying out the personalized preprocessing of the data information in a vehicle system of each user vehicle; each user vehicle inputs the data information subjected to personalized preprocessing into a preset 3D target detection model of the user vehicle, 3D target detection is completed, and local parameters are obtained; uploading the plurality of local parameters to a server for weighted federated average processing to obtain global model parameters, and updating a 3D target detection model of the server; the updated 3D target detection model of the server is sent back to the multiple user vehicles, the 3D target detection model of the user vehicles is updated according to the sent-back 3D target detection model, the steps are repeated for multiple times of updating, and a final 3D target detection model is obtained. According to the invention, the communication burden is reduced, and the accuracy of the 3D target detection model is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle autonomous driving environment perception technology, and in particular to a multimodal fusion 3D target detection method and system based on personalized federated learning. Background Art

[0002] With the rapid development of autonomous driving technology for smart cars, it has shown great potential in alleviating urban traffic congestion, significantly improving road safety and optimizing traffic flow efficiency. As a core key technology of intelligent driving systems, 3D object detection relies on high-precision on-board sensors such as cameras, radars and lidars to accurately identify and understand the location and category of objects in the surrounding environment. This technology significantly enhances the intelligent driving system's ability to identify traffic obstacles and participants by performing complex tasks such as object classification, two-dimensional and three-dimensional semantic segmentation, two-dimensional and three-dimensional object detection and depth completion, thereby improving the accuracy, safety and intelligence of the system. What is particularly critical is that in the core link of object detection, the system needs to accurately identify and continuously track a variety of targets such as surrounding vehicles, pedestrians and bicycles, which plays an irreplaceable and important role in ensuring road safety and ensuring the stable operation of intelligent driving systems in various complex traffic scenarios.

[0003] However, there are still the following problems in 3D object detection: it is difficult to fully consider the data differences between users when improving the 3D object detection model through federated learning; when performing target recognition through point cloud data, the point cloud data is in three-dimensional space and has a sparse nature, making it difficult for the model to process point cloud data; the accuracy of the 3D object detection model is not high. Summary of the invention

[0004] In order to at least partially solve the problem that federated learning is difficult to fully consider the data differences between users and the accuracy of 3D target detection model detection is not high enough, the present invention provides a multimodal fusion 3D target detection method and system based on personalized federated learning, in which each vehicle performs personalized training (personalized preprocessing) based on local data and uploads local parameters to the server. The server aggregates all local parameters, optimizes the global model, and redistributes the global model parameters to each vehicle, achieving a dual improvement in personalization and generalization. At the same time, the introduction of new point cloud processing technology significantly enhances the performance of local models. The present invention effectively protects data privacy, reduces communication burden, ensures vehicle data security, provides solid support for the intelligence and security of autonomous driving technology, and improves the accuracy of 3D target detection models.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] The first aspect of the present invention proposes a multimodal fusion 3D object detection method based on personalized federated learning, comprising:

[0007] Step 1: collecting data information around multiple user vehicles, and each of the multiple user vehicles performs personalized pre-processing on the data information in its own vehicle system, so as to effectively integrate differentiated information in the autonomous driving scenario;

[0008] Step 2: Each user vehicle inputs the personalized pre-processed data information into its own preset 3D object detection model to complete 3D object detection and obtain local parameters, so as to improve the accuracy of the preset 3D object detection model;

[0009] Step 3: Upload multiple local parameters to the server for weighted federated averaging to obtain global model parameters, and update the server's 3D object detection model according to the global model parameters to obtain more appropriate global model parameters;

[0010] Step 4: Send the updated 3D object detection model of the server back to multiple user vehicles, update the preset 3D object detection model of the user vehicle according to the updated 3D object detection model of the server, repeat the above steps for multiple updates to obtain the final 3D object detection model, so as to enhance the adaptability of the 3D object detection model to the local data distribution, thereby improving its generalization ability and overall performance.

[0011] Furthermore, the step 1 specifically includes:

[0012] Collect 3D point cloud data around the user's vehicle through the laser radar carried by the user's vehicle;

[0013] Map the 3D point cloud data to the 2D camera space, and then input the mapped features into the Transformer model to obtain pseudo LiDAR points, which facilitates the effective fusion and feature extraction of multimodal information;

[0014] The pseudo-LiDAR points are mapped into the LiDAR space to obtain personalized preprocessed data information, which facilitates the unification of multimodal representation in three-dimensional space.

[0015] Furthermore, the 3D point cloud data is mapped into the 2D camera space, and then the mapped features are input into the Transformer model to obtain pseudo LiDAR points, which specifically includes:

[0016] Project all lidar markers onto a 2D plane based on camera intrinsic and extrinsic parameters;

[0017] Map the 3D point cloud data into the 2D camera space according to the projection of the LiDAR marker on the 2D plane to obtain 2D image data;

[0018] The 2D image data is partitioned into cross-modal 2D local sets using dynamic set partitioning techniques and a modality-agnostic approach;

[0019] The two-dimensional local set is input into the Transformer model for two-dimensional feature learning, and then the 2D camera data is mapped to the 3D lidar space to obtain pseudo LiDAR points, which facilitates the effective fusion and feature extraction of multimodal information.

[0020] Furthermore, the pseudo LiDAR points are mapped into the LiDAR space to obtain personalized pre-processed data information, specifically including:

[0021] Sampling the pseudo LiDAR points, obtaining pseudo image points and projecting them into the corresponding image virtual coordinates;

[0022] A depth estimate of each pseudo image point is obtained according to the image virtual coordinates of the pseudo image point, and each pseudo image point is projected into the lidar space according to the depth estimate, so as to facilitate accurate projection of the pseudo image point into the lidar space;

[0023] According to the distance of each pseudo image point on the two-dimensional plane, the corresponding offset feature is obtained. According to the offset feature, the pseudo image points are divided into three-dimensional cross-modal local sets. The three-dimensional cross-modal local sets are input into the Transformer model for three-dimensional feature learning to obtain personalized preprocessed data information.

[0024] Further, the 3D object detection model includes a modality marker, a Transformer backbone network, and a bird's-eye view backbone network;

[0025] The modality marker is used to integrate the input personalized preprocessed data information to obtain image sequence markers and point cloud feature markers; wherein the modality marker includes an image block marker and a voxel feature coding marker; the image block marker is used to convert the image in the personalized preprocessed data information into an image sequence marker; the voxel feature coding marker is used to convert the personalized preprocessed data information into voxels for feature extraction;

[0026] The Transformer backbone network is used to simultaneously process the input image sequence labels and point cloud feature labels to obtain rich features;

[0027] The bird's-eye view backbone network is used to process based on rich features to complete 3D object detection.

[0028] Furthermore, the Transformer backbone network includes a plurality of processing blocks, each of which includes a balanced window attention mechanism, a first layer normalization layer, a feedforward neural network, and a second layer normalization layer connected in sequence;

[0029] The balanced window attention mechanism performs attention calculation on the point cloud data in the output of the modal marker to improve the processing capability of the point cloud sequence;

[0030] The first layer normalization layer is used to perform layer normalization on the output of the balanced window attention mechanism;

[0031] The feedforward neural network is used to perform feature classification on the output of the first layer standardization layer;

[0032] The second layer normalization layer is used to perform layer normalization on the output of the feedforward neural network.

[0033] Furthermore, the balanced window attention mechanism performs attention calculation on the point cloud data in the output of the modal marker, specifically including:

[0034] Convert the point cloud data in the personalized pre-processed data information into voxel representation, and divide the voxels into a plurality of local sets of predetermined size and non-overlapping, each of which corresponds to a 3D window, so as to facilitate subsequent attention calculation;

[0035] Attention calculation is performed based on the 3D window, which facilitates the reasonable allocation of computing resources and significantly improves the computing reasoning speed and learning efficiency;

[0036] The number of local sets is expressed as follows:

[0037]

[0038] Where μ is the maximum value of valid voxels assigned to each local set, N is the number of valid voxels in the window, To obtain the maximum integer operation, ξ[N%μ>0] means that when the remainder after the modulo operation of N and μ is greater than zero, it is 1, otherwise it is 0, and M is the number of local sets.

[0039] Furthermore, uploading the multiple local parameters to the server for weighted federated averaging processing specifically includes:

[0040] Optimize and update multiple local parameters separately;

[0041] The global model parameters are obtained according to the optimized and updated local parameters, and the 3D object detection model of the server is updated according to the global model parameters, so as to obtain more appropriate global model parameters.

[0042] Furthermore, the optimization and updating of the multiple local parameters are respectively performed, which is expressed by the following formula:

[0043]

[0044] in, To optimize the updated local parameters, is the number of user vehicles, is the gradient of the subfunction at a specific point θ, α is the step size, θ is the local parameter, f i () is the local loss function, F() is the global loss function;

[0045] The global model parameters are obtained according to the optimized and updated local parameters, and the 3D object detection model of the server is updated according to the global model parameters, which is expressed by the following formula:

[0046]

[0047] Among them, θ global is the global model parameter, ω i is the user vehicle parameter weight, is the local parameter of the i-th user vehicle.

[0048] The second aspect of the present invention proposes a multimodal fusion 3D object detection system based on personalized federated learning, comprising:

[0049] A preprocessing module, used to collect data information around multiple user vehicles, and the multiple user vehicles perform personalized preprocessing on the data information in their own vehicle systems, so as to effectively integrate differentiated information in the autonomous driving scene;

[0050] The 3D object detection module is used for each user vehicle to input personalized pre-processed data information into its own preset 3D object detection model, complete 3D object detection and obtain local parameters, so as to improve the accuracy of the preset 3D object detection model;

[0051] A calculation module is used to upload multiple local parameters to the server for weighted federated averaging to obtain global model parameters, and update the server's 3D object detection model according to the global model parameters to obtain more appropriate global model parameters;

[0052] The updating module is used to send the updated 3D object detection model of the server back to multiple user vehicles, update the preset 3D object detection model of the user vehicle according to the updated 3D object detection model of the server, repeat the above steps for multiple updates, and obtain the final 3D object detection model, so as to enhance the adaptability of the 3D object detection model to the local data distribution, thereby improving its generalization ability and overall performance.

[0053] Beneficial effects of the present invention:

[0054] (1) The present invention adopts a federated learning method to realize 3D target detection in autonomous driving. Under the premise of ensuring data privacy, it can effectively avoid the risk of leakage of a large amount of driving data generated during the autonomous driving process, and at the same time realize high-precision 3D target detection, significantly improving the safety and reliability of the autonomous driving system.

[0055] (2) The present invention adopts the method of stochastic gradient descent to optimize the global model of federated learning according to the data characteristics of the users, fully considers the data differences between users, significantly improves the generalization ability of the global model, and ensures that it can show stable performance on each user, thereby effectively improving the overall effect and user experience of the autonomous driving system.

[0056] (3) This paper proposes an innovative balanced window attention mechanism. This mechanism cleverly divides the point cloud data into equal parts and places them in a set within the window, solving the processing challenges brought about by the sparse nature of point cloud data in three-dimensional space, achieving efficient parallel processing of point cloud data, greatly improving processing speed and efficiency, and providing strong technical support for 3D object detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a multimodal fusion 3D object detection method based on personalized federated learning provided in an embodiment of the present invention.

[0058] Figure 2 A specific flow chart of a multimodal fusion 3D target detection method based on personalized federated learning provided in an embodiment of the present invention.

[0059] Figure 3 A schematic diagram of a balanced window attention mechanism provided for an embodiment of the present invention.

[0060] Figure 4 An architectural diagram of a multimodal fusion 3D object detection system based on personalized federated learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Example 1

[0063] like Figure 1As shown in the figure, the multimodal fusion 3D target detection method based on personalized federated learning includes:

[0064] S101: collecting data information around multiple user vehicles, and each of the multiple user vehicles performs personalized pre-processing on the data information in its own vehicle system.

[0065] Specifically, the 3D point cloud data around the user's vehicle is collected by the laser radar carried by the user's vehicle itself. In order to effectively integrate the differentiated information in the autonomous driving scene, the present invention proposes a novel fusion method. This method first unifies the laser radar feature information into the camera space for feature learning, and then maps the feature information into the laser radar space for further feature learning. In this way, the present invention can take into account the inherent structural relationship between the camera and the laser radar, and realize the effective fusion of multi-modal target information of autonomous driving.

[0066] In order to project 3D point cloud data into 2D camera space for feature learning, all LiDAR markers (tokens) are first accurately projected onto a 2D plane using the camera’s internal and external parameters. Subsequently, the projected images of the LiDAR markers and LiDAR tokens on the 2D plane are unified into the corresponding 2D camera space, and the 2D image data is divided into cross-modal 2D local sets in a modality-agnostic manner using dynamic set partitioning techniques. Finally, the 2D local sets are input into the Transformer model for 2D feature learning, after which the 2D camera data is mapped into the 3D LiDAR space to obtain pseudo LiDAR points for effective fusion and feature extraction of multimodal information.

[0067] In order to achieve the unification of multimodal representations in three-dimensional space, an efficient and robust view projection method is required, which can uniquely and accurately map image patches to three-dimensional space. However, due to the inaccurate depth information associated with pixels, the conversion of images to LiDAR space becomes an ill-posed problem. To overcome this limitation, the present invention proposes a pre-computation method to efficiently map image patches to LiDAR space. Specifically, a set of pseudo LiDAR points are first sampled to obtain pseudo image points, and then the obtained pseudo image points are projected into the corresponding image virtual coordinates. The image virtual coordinates of these pseudo image points are used to retrieve the depth estimate of each pseudo image point. Subsequently, these pseudo image points are projected into three-dimensional space, and the offset features are calculated based on their distances on the two-dimensional plane. Finally, these pseudo image points with offset features are divided into three-dimensional cross-modal local sets, and the three-dimensional cross-modal local sets are input into the Transformer model for three-dimensional feature learning to obtain personalized pre-processed data information for unified processing.

[0068] S102: Each user vehicle inputs the personalized pre-processed data information into its own preset 3D object detection model to complete 3D object detection and obtain local parameters.

[0069] S103: Upload multiple local parameters to the server for weighted federated averaging to obtain global model parameters, and update the 3D object detection model of the server according to the global model parameters.

[0070] Specifically, in order to reduce communication overhead and increase communication speed when uploading multiple local parameters to the server, model compression technology is used to transmit data between the server and the local user vehicle. This method not only reduces the transmission burden, but also accelerates the overall learning process.

[0071] The present invention proposes a personalized federated averaging method for weighted federated averaging, which enhances the performance of the global model and its generalization ability among different users by incorporating an optimization algorithm. Specifically, the present invention uses stochastic gradient descent as an optimization method to optimize the federated averaging process. During implementation, it is assumed that each user starts from the initial point and optimizes and updates the local parameters using the iterative steps of gradient descent according to their respective loss functions. The above process is shown in the following formula:

[0072]

[0073] in, To optimize the updated local parameters, is the number of user vehicles, is the gradient of the subfunction at a specific point θ, α is the step size, θ is the local parameter, f i () is the local loss function, and F() is the global loss function.

[0074] The global model parameters are obtained according to the optimized and updated local parameters, and the 3D object detection model of the server is updated according to the global model parameters. The above process is expressed by the following formula:

[0075]

[0076] Among them, θ global is the global model parameter, ω i is the user vehicle parameter weight, is the local parameter of the i-th user vehicle.

[0077] Finally, the server’s 3D object detection model is updated according to the global model parameters

[0078] S104: Send the updated 3D object detection model of the server back to multiple user vehicles, update the preset 3D object detection model of the user vehicle according to the updated 3D object detection model of the server, repeat the above steps for multiple updates to obtain the final 3D object detection model.

[0079] Specifically, the 3D object detection model will be redistributed back to the user vehicles for a new round of learning. It is worth noting that the present invention takes into account the inherent volatility of the Internet of Vehicles and adopts a selective client participation strategy to alleviate the deterioration of model performance during federated learning. These user vehicles as clients use their stable network connections to contribute local parameters for their local training, thereby ensuring that the aggregation process in each communication round is consistent and reliable. Before each local training, the user vehicle will use the stochastic gradient descent method to perform personalized optimization and personalized preprocessing on the global model to enhance its adaptability to local data distribution, thereby improving its generalization ability and overall performance.

[0080] like Figure 2 As shown, the present invention effectively integrates differentiated information in the autonomous driving scene by locally performing personalized preprocessing on the collected data information. Afterwards, the personalized preprocessed data information is input into the preset 3D target detection model to complete the target detection and obtain relatively accurate local parameters. Then, a weighted federal average process is performed on multiple local parameters uploaded to the server to obtain global parameters, thereby enhancing the adaptability of the 3D target detection model to the local data distribution, thereby improving its generalization ability and overall performance. Finally, the parameters of the 3D target detection model are updated multiple times to obtain the final 3D target detection model, thereby improving the accuracy of the 3D target detection model.

[0081] Example 2

[0082] Based on the above embodiments, the embodiments of the present invention provide a specific structure of a 3D object detection model, which specifically includes:

[0083] The 3D object detection model includes a modality marker, a Transformer backbone network, and a bird’s-eye view backbone network.

[0084] The modality marker is used to integrate the input personalized preprocessed data information to obtain image sequence markers and point cloud feature markers. The modality marker includes an image block marker and a voxel feature encoding marker. Specifically, the image block marker is used to segment the image in the personalized preprocessed data information into 8×8 pixel blocks and convert them into sequence markers to obtain image sequence markers.

[0085] For the point cloud data in the personalized pre-processed data information, a voxel feature encoding marker with customized grid size (0.3m, 0.3m, 8m) is used to achieve efficient representation.

[0086] The Transformer backbone network is used to simultaneously process the input image sequence labels and point cloud feature labels to obtain rich features. Specifically, the Transformer backbone network includes multiple processing blocks, which facilitate parallel processing. The multiple processing blocks each include a balanced window attention mechanism, a first layer normalization layer, a feedforward neural network, and a second layer normalization layer connected in sequence.

[0087] The balanced window attention mechanism performs attention calculation on the point cloud data in the output of the modality tagger. Specifically, Figure 3 As shown, in order to efficiently process point cloud data, the point cloud data is encapsulated in a series of well-defined, equal-sized sets, each of which corresponds to a bounded 3D window. In this way, the standard self-attention calculation can be performed in parallel, significantly improving processing efficiency. Following the traditional window attention practice, the point cloud data is first converted into a voxel representation, and these voxels are divided into a series of non-overlapping 3D windows of predetermined size. Each window encapsulates N valid voxels. In order to generate non-overlapping and equal-sized local sets, the required number of sets is first calculated in the window, as follows:

[0088]

[0089] Where μ is the maximum value of valid voxels assigned to each local set, N is the number of valid voxels in the window, To obtain the maximum integer operation, ξ[N%μ>0] means that when the remainder after the modulo operation of N and μ is greater than zero, it is 1, otherwise it is 0, and M is the number of local sets.

[0090] In this way, all voxels in the corresponding window can be covered with a minimum number of sets.

[0091] Under the action of the BWA (Balanced Window Attention) mechanism of the present invention, several sets containing equal numbers of valid voxels can be obtained in each window. This feature means that the more voxels there are in the window, the more corresponding sets there are, and each set remains equal in size. Such a design is conducive to the complete parallel processing of these sets, which can effectively promote the reasonable allocation of computing resources and significantly improve the computing reasoning speed and learning efficiency.

[0092] The output of the balanced window attention mechanism is input into the first layer normalization layer for layer normalization. The output of the first layer normalization layer is input into the feedforward neural network for feature classification. The output of the feedforward neural network is input into the second layer normalization layer for layer normalization to obtain rich features. The rich features output by the Transformer backbone network are then input into the bird's-eye view (BEV) backbone network to provide higher accuracy and efficiency for 3D perception tasks and complete 3D object detection.

[0093] Example 3

[0094] Based on the above embodiments, the embodiment of the present invention provides an experimental effect verification process of a multimodal fusion 3D object detection method based on personalized federated learning, which specifically includes:

[0095] Due to the huge demand for computing resources in this experiment, all experiments were conducted on a server equipped with four GPUs. This configuration ensures the best performance and efficiency during the experiment. The specific settings are shown in Table 1:

[0096] Table 1 Experimental environment configuration

[0097]

[0098] In the present invention, extensive 3D object detection experiments were conducted using the ONCE, NUSCENES, and WAYMO datasets. In order to simulate the inherent distributed nature of federated learning, each dataset is randomly divided into multiple subsets, each of which represents the localized data stored on a single edge device participating in the federated learning framework. In essence, these subsets reflect the local data sets of autonomous driving vehicles. This diversification of datasets not only covers a wider range of driving scenarios, but also improves the generalization ability of the present invention. These three datasets are recognized as standard benchmarks in the field of autonomous driving perception. They comprehensively cover various road conditions, different time periods, and changing weather conditions, and are therefore significantly representative. The specific structure of the dataset is shown in Table 2.

[0099] Table 2 Dataset structure

[0100]

[0101] As shown in Table 3, a comprehensive evaluation of our invention on the NUSCENES dataset is elaborated in detail and compared with nine pioneering 3D object detection methods to highlight its superior performance. Our invention stands out as a clear exception, achieving an impressive mAP (mean average precision) of 71.2% and NDS (NUSCENES detection score) of 73.6%, far surpassing the current state of the art by a significant margin. Notably, our invention outperforms baseline methods such as BEVFusion and Transfusion by more than one percentage point, strongly demonstrating the superior effect of personalized federated learning in enhancing the perception capabilities of autonomous driving systems.

[0102] Table 3 Comparative analysis of methods on the NUSCENES dataset

[0103]

[0104] Table 4 carefully presents a comprehensive comparison of the test results of the proposed method with six popular state-of-the-art methods after rigorous evaluation on the challenging ONCE dataset. The values ​​for each category given here are overall values. The results show that the proposed model achieves a mAP value of 67.14%, surpassing the outstanding CenterPoints by 5.9% and significantly outperforming other competing methods. This significant improvement highlights the superior effectiveness and robustness of the proposed personalized federated learning method in addressing the complex and multifaceted challenges inherent in the ONCE dataset.

[0105] Table 4 Comparative analysis of methods on the ONCE dataset

[0106]

[0107] Table 5 compares the test results of the present invention on the rigorous WAYMO dataset in detail and compares it with five leading 3D object detection techniques. Comprehensive analysis shows that the present invention stands out among all six competing methods, achieving 83.9% mAP and 81.8% mAPH at the L1 level, significantly surpassing other advanced methods. In addition, at the L2 level, the present invention achieved 78.3% mAP and 77.1% mAPH, which is also in a leading position. This shows that the present invention can achieve accurate perception and detection even in complex environments. It also highlights the outstanding effectiveness and advancement of personalized federated learning methods and balanced window attention mechanisms in dealing with the inherent complexity of the WAYMO dataset.

[0108] Table 5 Comparative analysis of methods on the WAYMO dataset

[0109]

[0110] In summary, the present invention proposes a multimodal fusion 3D target detection method and system based on personalized federated learning, aiming to solve the complex challenges of environmental perception in the field of autonomous driving, especially the balance between ensuring data privacy and improving detection performance. The system innovatively integrates federated learning technology, and through a distributed training mechanism, realizes the effective integration and utilization of multi-source heterogeneous data (including point clouds and images) without directly sharing the original data, thereby enhancing the 3D target detection capability of autonomous driving vehicles. The core of the present invention is: first, a personalized strategy is introduced into the federated learning framework, and a stochastic gradient descent optimization algorithm is used to perform personalized adjustments before local model training to adapt to the heterogeneity of vehicle data of different users, significantly improving the adaptability and generalization ability of the global model in diversified scenarios. Secondly, in response to the sparsity problem of point cloud data, the present invention designs an innovative balanced window attention mechanism, which can effectively fuse point cloud and image information while maintaining computational efficiency, enhance the model's perception of complex environments, and improve the accuracy and robustness of 3D target detection. The experimental results show that the multimodal autonomous driving 3D target detection system of the present invention not only significantly improves the target detection accuracy and speed, but also effectively protects the privacy and security of user data, providing important technical support for the safe and efficient development of autonomous driving technology. Therefore, the present invention has significant technical innovation and broad application prospects in the field of autonomous driving, and is an important breakthrough in autonomous driving environment perception technology.

[0111] Example 4

[0112] Based on the above embodiments, Figure 4 As shown, an embodiment of the present invention provides a multimodal fusion 3D target detection system based on personalized federated learning, including:

[0113] The preprocessing module is used to collect data information around multiple user vehicles, and the multiple user vehicles respectively perform personalized preprocessing on the data information in their own vehicle systems.

[0114] The 3D target detection module is used for each user vehicle to input personalized pre-processed data information into its own preset 3D target detection model to complete 3D target detection and obtain local parameters.

[0115] The calculation module is used to upload multiple local parameters to the server for weighted federated averaging processing to obtain global model parameters, and update the server's 3D object detection model according to the global model parameters.

[0116] The updating module is used to send the updated 3D target detection model of the server back to multiple user vehicles, update the preset 3D target detection model of the user vehicle according to the updated 3D target detection model of the server, repeat the above steps for multiple updates, and obtain the final 3D target detection model.

[0117] It should be noted that the multimodal fusion 3D target detection system based on personalized federated learning provided in the embodiment of the present invention is to implement the multimodal fusion 3D target detection method based on personalized federated learning in the above embodiment. Its specific functions can be referred to the above method embodiments and will not be repeated here.

[0118] In summary, the present invention adopts a federated learning method to realize 3D target detection in autonomous driving. It can effectively avoid the risk of leakage of a large amount of driving data generated during autonomous driving while ensuring data privacy, and at the same time realize high-precision 3D target detection, significantly improving the safety and reliability of the autonomous driving system. The present invention adopts a random gradient descent method to optimize the global model of federated learning according to the data characteristics of the user, fully considers the data differences between users, significantly improves the generalization ability of the global model, and ensures that it can show stable performance on each user, thereby effectively improving the overall effect and user experience of the autonomous driving system. The present invention proposes an innovative balanced window attention mechanism. The mechanism solves the processing challenges brought by the point cloud data being in three-dimensional space and having sparse properties by cleverly dividing the point cloud data into equal amounts and placing it in a set within the window, realizing efficient parallel processing of point cloud data, greatly improving the processing speed and efficiency, and providing strong technical support for 3D target detection tasks.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal fusion 3D object detection method based on personalized federated learning, characterized in that: include: Step 1: Collect data information around multiple user vehicles, and each of the multiple user vehicles performs personalized pre-processing on the data information in its own vehicle system; Step 2: Each user vehicle inputs the personalized pre-processed data information into its own preset 3D object detection model to complete 3D object detection and obtain local parameters; Step 3: Upload multiple local parameters to the server for weighted federated averaging to obtain global model parameters, and update the server's 3D object detection model based on the global model parameters; Step 4: Send the updated 3D object detection model of the server back to multiple user vehicles, update the preset 3D object detection model of the user vehicle according to the updated 3D object detection model of the server, repeat the above steps for multiple updates to obtain the final 3D object detection model.

2. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 1 is characterized in that: The step 1 specifically includes: Collect 3D point cloud data around the user's vehicle through the laser radar carried by the user's vehicle; Map the 3D point cloud data to the 2D camera space, and then input the mapped features into the Transformer model to obtain pseudo LiDAR points; Map the pseudo LiDAR points into the LiDAR space to obtain personalized preprocessed data information.

3. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 2 is characterized in that: The 3D point cloud data is mapped to the 2D camera space, and then the mapped features are input into the Transformer model to obtain pseudo LiDAR points, which specifically includes: Project all lidar markers onto a 2D plane based on camera intrinsic and extrinsic parameters; Map the 3D point cloud data into the 2D camera space according to the projection of the LiDAR marker on the 2D plane to obtain 2D image data; The 2D image data is partitioned into cross-modal 2D local sets using dynamic set partitioning techniques and a modality-agnostic approach; The two-dimensional local set is input into the Transformer model for two-dimensional feature learning, and then the 2D camera data is mapped to the 3D LiDAR space to obtain pseudo LiDAR points.

4. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 2 is characterized in that: The pseudo LiDAR points are mapped into the LiDAR space to obtain personalized pre-processed data information, specifically including: Sampling the pseudo LiDAR points, obtaining pseudo image points and projecting them into the corresponding image virtual coordinates; Obtaining a depth estimate of each pseudo image point according to the image virtual coordinates of the pseudo image point, and projecting the depth estimate of each pseudo image point into the lidar space; According to the distance of each pseudo image point on the two-dimensional plane, the corresponding offset feature is obtained. According to the offset feature, the pseudo image points are divided into three-dimensional cross-modal local sets. The three-dimensional cross-modal local sets are input into the Transformer model for three-dimensional feature learning to obtain personalized preprocessed data information.

5. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 1, characterized in that: The 3D object detection model includes a modality marker, a Transformer backbone network, and a bird's-eye view backbone network; The modality marker is used to integrate the input personalized preprocessed data information to obtain image sequence markers and point cloud feature markers; wherein the modality marker includes an image block marker and a voxel feature coding marker; the image block marker is used to convert the image in the personalized preprocessed data information into an image sequence marker; the voxel feature coding marker is used to convert the personalized preprocessed data information into voxels for feature extraction; The Transformer backbone network is used to simultaneously process the input image sequence labels and point cloud feature labels to obtain rich features; The bird's-eye view backbone network is used to process based on rich features to complete 3D object detection.

6. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 5, characterized in that: The Transformer backbone network includes multiple processing blocks, each of which includes a balanced window attention mechanism, a first layer normalization layer, a feedforward neural network, and a second layer normalization layer connected in sequence; The balanced window attention mechanism performs attention calculation on the point cloud data in the modality marker output; The first layer normalization layer is used to perform layer normalization on the output of the balanced window attention mechanism; The feedforward neural network is used to perform feature classification on the output of the first layer standardization layer; The second layer normalization layer is used to perform layer normalization on the output of the feedforward neural network.

7. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 6, characterized in that: The balanced window attention mechanism performs attention calculation on the point cloud data in the output of the modality marker, specifically including: Converting the point cloud data in the personalized preprocessed data information into voxel representation, and dividing the voxels into a plurality of local sets of predetermined sizes and non-overlapping with each other, each of the local sets corresponding to a 3D window; Attention calculation based on 3D window; The number of local sets is expressed as follows: Where μ is the maximum value of valid voxels assigned to each local set, N is the number of valid voxels in the window, To obtain the maximum integer operation, ξ[N%μ>0] means that when the remainder after the modulo operation of N and μ is greater than zero, it is 1, otherwise it is 0, and M is the number of local sets.

8. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 1, characterized in that: The uploading of multiple local parameters to the server for weighted federated averaging processing specifically includes: Optimize and update multiple local parameters separately; The global model parameters are obtained according to the optimized and updated local parameters, and the 3D object detection model of the server is updated according to the global model parameters.

9. The multimodal fusion 3D object detection method based on personalized federated learning according to claim 8, characterized in that: The optimization and updating of multiple local parameters are respectively performed as follows: in, To optimize the updated local parameters, is the number of user vehicles, is the gradient of the subfunction at a specific point θ, α is the step size, θ is the local parameter, and f i () is the local loss function, F() is the global loss function; The global model parameters are obtained according to the optimized and updated local parameters, and the 3D object detection model of the server is updated according to the global model parameters, which is expressed by the following formula: Among them, θ global is the global model parameter, ω i is the user vehicle parameter weight, is the local parameter of the i-th user vehicle.

10. A multimodal fusion 3D object detection system based on personalized federated learning, characterized in that: include: A preprocessing module, used to collect data information around multiple user vehicles, and the multiple user vehicles perform personalized preprocessing on the data information in their own vehicle systems; The 3D object detection module is used for each user vehicle to input personalized pre-processed data information into its own preset 3D object detection model to complete 3D object detection and obtain local parameters; A calculation module is used to upload multiple local parameters to the server for weighted federated averaging to obtain global model parameters, and update the server's 3D object detection model according to the global model parameters; The updating module is used to send the updated 3D target detection model of the server back to multiple user vehicles, update the preset 3D target detection model of the user vehicle according to the updated 3D target detection model of the server, repeat the above steps for multiple updates, and obtain the final 3D target detection model.

Images