Heterogeneity-aware federated 3d object detection method and system based on prompt communication

Abstract

The application discloses a federated 3D object detection method and system based on heterogeneous perception prompt communication, comprising the following steps: firstly, a local client collects 3D point cloud data and performs pretreatment; then a global 3D object detection model performs feature extraction and 3D object detection on the pretreated 3D point cloud data, and a detection result generates a boundary box and a category label of an object; the global 3D object detection model processes the unbalanced problem of data through local-global heterogeneous perception loss, ensures that the training effect is more accurate, in each communication round, the local client uploads updated model parameters or a prompt pool to a global server, the server aggregates the updates of all local clients, broadcasts the updated global model back to the client, and performs the next round of training, and the above process is continuously cycled, finally, a global 3D target detection model is output, and accurate 3D object detection can be performed.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, specifically to a federated 3D object detection method and system based on heterogeneous perception cue communication. Background Technology

[0002] With the rapid development of deep learning technology, 3D object detection is increasingly being used in fields such as autonomous driving, robot navigation, and smart cities. Traditional 3D object detection methods typically rely on centralized data training, requiring all datasets to be uploaded to a server for model training. However, in practical applications, data is often distributed across multiple terminal devices or sensors, which may be located in different geographical locations and have different acquisition methods and data formats. Centralized uploading of these datasets for training is often impractical due to privacy and data security concerns.

[0003] Furthermore, data in 3D object detection tasks exhibit significant heterogeneity. For example, data from different sensors may have varying resolutions, noise levels, and scene complexities, posing a significant challenge to model training. How to effectively collaborate on training distributed data while ensuring data privacy, and how to address the impact of data heterogeneity, have become crucial research issues.

[0004] Existing federated learning methods have been widely applied to solve distributed learning problems, especially in scenarios where data privacy is paramount. Federated learning effectively protects data privacy by training the model locally and transmitting only the updated parameters of the model, rather than the original data. However, traditional federated learning methods face challenges such as data heterogeneity and excessive communication overhead when handling 3D object detection tasks. Particularly in 3D object detection, how to efficiently share training information among multiple local clients, avoid transmitting large amounts of data, and effectively handle heterogeneous data from different sensors and scenes remains a pressing challenge.

[0005] To leverage distributed data while protecting data privacy, federated learning has been proposed as an emerging solution. Federated learning allows local clients to train models locally without uploading raw data, achieving collaborative learning through the exchange of model updates. Existing technologies have explored various approaches to address the communication cost challenges in federated learning. For example, in the method published by Chuhan Wu et al. (Communication-efficient federated learning via knowledge distillation), a federated learning framework called FedKD was proposed. This method reduces communication overhead through a knowledge distillation mechanism. Its core idea is to maintain a large "teacher" model and a small "student" model on each local client, and only the lightweight "student" model needs to be transferred and aggregated between local clients, thus effectively reducing communication bandwidth. While the above methods have achieved significant results in reducing communication costs, they still have shortcomings when applied to complex tasks such as large-scale 3D object detection. Although the FedKD method avoids transferring large models, its knowledge distillation framework requires local clients to maintain and train two models (teacher and student) simultaneously locally, which undoubtedly increases the computational burden on local clients. Furthermore, the communication content remains the parameters of the entire "student" model, leaving room for further optimization for scenarios seeking maximum efficiency. While the aforementioned methods mention adapting to data heterogeneity through personalized local models, their framework lacks a mechanism at the model optimization level to explicitly correct gradient biases caused by uneven data distribution. When the 3D scene data distributions of different local clients differ significantly, the "drift" problem in local model updates persists. Simple model aggregation or knowledge distillation may be insufficient to completely correct this bias, thus affecting the convergence and accuracy of the final global model.

[0006] Training high-precision 3D object detection models typically requires large amounts of data with accurate 3D annotations. The 3D annotation process is not only costly and time-consuming, but also heavily reliant on human experts. To reduce the dependence on massive amounts of labeled data, existing technologies have proposed several semi-supervised learning (SSL) methods. For example, in Junbo Yin et al.'s paper, "Semi-supervised 3D Object Detection with Proficient Teachers," a method was proposed that uses a "teacher-student" model to train on a large amount of unlabeled point cloud data. This method uses an enhanced "teacher" model to generate high-quality pseudo-labels for the unlabeled data to supervise the training of the "student" model, thus achieving excellent detection performance even with only a small amount of labeled data. However, the basic technical paradigm of these advanced semi-supervised detection methods is still centralized training. It requires all training data to be collected and stored on a single central server or computing cluster. In many real-world applications, centralized uploading of raw 3D point cloud datasets distributed across different users or organizations is either impractical or prohibited due to data privacy regulations, data security concerns, and commercial sensitivities. This leads to the so-called "data silo" problem, preventing centralized training methods from effectively utilizing massive amounts of distributed real-world data. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a federated 3D object detection method and system based on heterogeneity-aware cue communication. This method effectively balances the heterogeneity issue between the local client and the global model by introducing a local-global heterogeneity-aware loss, improving the model's adaptability to different scenes and sensor data. Furthermore, this invention designs a federated cue communication module that significantly reduces communication overhead, improves training efficiency, and ensures data privacy by transmitting only the most representative learning information from the local client (i.e., the cue pool) instead of the entire model weights. Through this method, this invention enables efficient collaborative training among multiple local clients, overcoming data heterogeneity and communication bottlenecks, thereby achieving higher accuracy and stronger robustness in complex 3D object detection tasks.

[0008] The present invention is achieved by at least one of the following technical solutions.

[0009] A federated 3D object detection method based on heterogeneity-aware cueing communication includes: a local client inputting the 3D point cloud data to be detected into a trained global 3D object detection model; and the global 3D object detection model outputting the bounding box of the object and the object's category.

[0010] The training of the global 3D object detection model includes the following steps:

[0011] Collect 3D point cloud data from multiple local clients;

[0012] Preprocess the collected 3D point cloud data;

[0013] The local client trains the local model using preprocessed 3D point cloud data. After local training is completed, each local client participating in this round of communication uploads a local hint pool containing local data knowledge to the global server.

[0014] The global server collects all local hint pools uploaded by local clients, aggregates all local hint pools into a global hint pool containing global knowledge, and broadcasts the global hint pool back to all local clients.

[0015] Each local client receives the global cue pool and uses it to update its local model for the next round of training.

[0016] Furthermore, the local model includes a vision transformer-based encoder, a decoder, and a detection head, wherein the vision transformer-based encoder is composed of multiple stacked layers, and each layer contains a multi-head self-attention block and a Transformer encoder.

[0017] Furthermore, the local cue pool is the cue embedding vector embedded in each layer of the visual transformer-based encoder, i.e., the prefix cue. During the forward propagation of local training, when computing the multi-head self-attention block in each layer of the visual transformer-based encoder, the prefix cue concatenates the original key matrix and value matrix of the multi-head self-attention block in the sequence dimension to form a new, expanded key matrix and value matrix. Subsequently, the query matrix and the expanded key matrix and value matrix are used to perform standard self-attention computation.

[0018] Furthermore, when the local client trains the local model, only the parameters of the local cue pool are learned and optimized based on the gradient of the loss function during the entire backpropagation and parameter update process of the training.

[0019] Furthermore, in each training iteration of the local client, a local correction coefficient is first calculated for each data sample in the mini-batch data. At the same time, each local client participating in this round of communication also needs to calculate and upload one or more class distribution representations to the global server. Each class distribution representation is obtained by calculating the average gradient value of all samples belonging to a specific object class in the local dataset. The gradient value is the gradient generated by the cross-entropy loss function.

[0020] Subsequently, on the global server side, the category distribution representations uploaded by all local clients are aggregated, and a global correction coefficient is calculated for each object category based on the aggregation result and the logarithmic function relationship. The global correction coefficient is then distributed to each local client.

[0021] Finally, when updating the model parameters using gradient descent on the local client, the original gradient generated by the local loss function is multiplied by the calculated local correction coefficient corresponding to the current sample and the global correction coefficient corresponding to the object category of the current sample received from the server to obtain a corrected gradient, and the corrected gradient is used to update the parameters of the local model.

[0022] Furthermore, the local correction coefficient is the normalized proportion of the gradient magnitude of a single sample in the total gradient magnitude of the entire mini-batch, and the proportion reflects the contribution of a single sample to the model parameter update.

[0023] The system for implementing the federated 3D object detection method based on heterogeneity-aware cueing communication includes:

[0024] The 3D object detection module includes a vision transformer-based encoder, decoder, and detection head, used to extract features from input 3D point cloud data and predict the bounding box and category of the object;

[0025] The federated cue communication module collaboratively trains the 3D object detection module distributed across local clients by transmitting only a lightweight, learnable pool of local cues.

[0026] Furthermore, the federated notification communication module includes the following sub-modules:

[0027] Local cue pool generation module: Generates and optimizes a local cue pool during local training. Specifically, within the encoder of the 3D object detection module, a pair of learnable prefix cues is created for each encoder level, and the prefix cues are used together to form the local cue pool.

[0028] Hint pool upload module: After local training is completed, the optimized local hint pool is used as the only model update content and sent to the global server through the communication network;

[0029] Hint pool aggregation module: responsible for receiving all local hint pools uploaded by local clients, and using a weighted average algorithm to merge the uploaded local hint pools into a global hint pool containing global knowledge;

[0030] Global cue pool broadcast module: responsible for distributing the global cue pool to all local clients to replace the original local cue pool of the local clients and to serve as the starting point for the next round of local training.

[0031] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0032] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0033] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0034] 1. This invention effectively processes heterogeneous data from different sensors and scenes by combining a deep learning framework based on visual transformers and federated learning methods, thereby improving the accuracy and robustness of 3D object detection models in multi-source data environments, and exhibiting strong adaptability, especially in complex scenes.

[0035] 2. This invention adopts a federated learning framework, which transmits only the most representative learning information learned during the local client training process (i.e., the cue pool), rather than the complete model weights. This significantly reduces communication overhead while ensuring data privacy and avoids the high bandwidth cost of transmitting large amounts of data.

[0036] 3. By introducing local-global heterogeneity-aware loss, this invention can effectively balance the data heterogeneity between different local clients and the differences between the global model, ensuring that the model can make full use of local data during local training, while maintaining the consistency and accuracy of the global model. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a federated 3D object detection method based on heterogeneity perception and cueing communication according to an embodiment of the present invention.

[0038] Figure 2 This is a flowchart of a federated 3D object detection method based on heterogeneity perception cueing communication according to an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram illustrating the performance evaluation of the method model of this invention on the SUN RGB-D and ScanNet V2 datasets with mAP@0.25. Detailed Implementation

[0040] The method of the present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 , Figure 2As shown in the figure, a federated 3D object detection method based on heterogeneity-aware cueing communication in this embodiment includes the following steps:

[0042] Step 1: The local client preprocesses the 3D point cloud data to be detected. The preprocessing includes downsampling and standardization of the point cloud data. Then, the preprocessed 3D point cloud data is input into the trained global 3D object detection model.

[0043] Step 2: The global 3D object detection model outputs the bounding boxes of the objects and the object categories.

[0044] The training of the global 3D object detection model includes the following steps:

[0045] Step S1: Collect 3D point cloud data from multiple local clients.

[0046] Step S2: Preprocess the collected 3D point cloud data, including the following steps:

[0047] First, the input 3D point cloud data undergoes downsampling. This process selectively retains some points to reduce the data volume, thereby lowering computational burden and memory consumption. The downsampled point cloud data still retains sufficient spatial features, ensuring the model can extract useful information in subsequent stages.

[0048] Subsequently, the downsampled 3D point cloud data is standardized: the coordinate values ​​in the point cloud are normalized to a uniform scale. This step, by standardizing the coordinates of each point, makes the data more stable during training and avoids gradient imbalance caused by different data scales. Standardized data can effectively improve the training efficiency of the model and accelerate model convergence.

[0049] Finally, to enhance the model's generalization ability, data augmentation techniques were applied, including random flipping, rotation, and scaling. These data augmentation strategies, by increasing data diversity, further improved the model's robustness to different scenarios and perspectives, thereby enhancing its adaptability to unknown data. Through these preprocessing steps, the quality and diversity of the input data were improved, further laying the foundation for subsequent feature extraction and object detection.

[0050] Step S3: The local client adopts and trains its local model.

[0051] The local model includes a Vision Transformer (ViT)-based encoder, decoder, and detection head. The Vision Transformer (ViT)-based encoder consists of multiple stacked layers, with each layer containing a multi-head self-attention block and a Transformer encoder.

[0052] During local training, the local model processes 3D point cloud data to achieve 3D object detection as follows: First, the encoder extracts features containing high-level semantic information from the preprocessed 3D point cloud data; then, the decoder upsamples and fuses the features output by the encoder to generate a set of feature vectors with rich semantics and accurate spatial information; finally, the detection head receives the feature vectors from the decoder and performs bounding box regression and category classification on these feature vectors to ultimately output the bounding box and category label of the object. The specific feature extraction and detection steps are as follows:

[0053] The preprocessed 3D point cloud data is encoded into feature representations. ,in Indicates the first Feature vectors of a point cloud This refers to the number of points in the point cloud data. The self-attention mechanism of multi-head self-attention blocks calculates the attention weight and weighted representation of each point using the following formula:

[0054] ;

[0055] The input to the self-attention mechanism is a set of identical point feature vectors. These identical feature vectors undergo linear transformations to generate three different matrices: a query matrix, a lookup matrix, and a lookup matrix. Key matrix Sum matrix First, by querying the matrix... transpose of the key matrix Perform matrix multiplication to calculate each point in the point cloud (from...). One row in the middle represents) and all other points (by One column in the table represents the original correlation score between the keys. These scores are then divided by the key vector dimension. square root This step is a scaling operation used to stabilize the gradient, making the training process more stable. Next, the scaled score matrix is ​​applied along the sequence dimension. The function is normalized to obtain an attention weight matrix. Each value in this matrix is ​​between 0 and 1, representing how much weight a given point should "pay attention to" or "allocate" to other points when generating new features. Finally, this attention weight matrix is ​​compared with the value matrix. Perform matrix multiplication. Therefore, this The final output of the operation is a new feature matrix. Each row of this matrix is ​​a subset of the original value matrix. The feature vector is a weighted sum of all rows in the point cloud, and the weights are determined by the correlation (attention) between the corresponding point in that row and all other points. Simply put, the output feature vector is a weighted aggregation of the original feature vectors. It not only contains the original information of the point itself, but also dynamically incorporates the most relevant contextual information in the entire point cloud, thus making the feature representation more globally perceptive and semantically rich.

[0056] First, a self-attention mechanism is employed to model the input 3D data, capturing the global dependencies between different points in the data. The self-attention mechanism calculates the similarity between input points to obtain a weighted representation for each point, enabling the extraction of more semantic features for subsequent object detection tasks. In this way, the Transformer encoder can generate new representations for each point based on local and global contextual information in the input point cloud data. These features encompass the semantic information and spatial structure of the point cloud.

[0057] Next, the feature matrix output by the self-attention mechanism is input into the Transformer encoder, which extracts feature vectors. The decoder then processes the features output by the encoder, specifically by upsampling the features containing high-level semantic information to gradually restore the spatial resolution and geometric details of the point cloud. During feature upsampling, the decoder also fuses feature information from different levels of the encoder, thus combining high-level semantic features with low-level geometric features. Finally, the decoder outputs a set of feature vectors that possess both rich semantic and accurate spatial information. and the feature vector The data is then passed to a subsequent detection head for processing. The detection head receives the feature vector from the decoder and performs object detection. The detection head consists of a regression head and a classification head. The regression head predicts the bounding box location of the object, and the classification head predicts the object's category. The regression head's role is to predict the object's bounding box location based on the output of the Transformer encoder. high-dimensional feature vectors This is used to predict the 3D bounding box parameters of the object. This process is implemented by a regression network, which can be expressed as:

[0058] ;

[0059] in, In one embodiment of the present invention, a multilayer perceptron is used. It is a high-dimensional, semantically rich feature vector As input, through nonlinear mapping and dimensionality transformation of the multilayer perceptron, a low-dimensional vector is finally output. The low-dimensional vector The dimensions are predefined, and their elements directly correspond to all the geometric parameters required to predict the bounding box.

[0060] The classification head is used to predict the category of each object based on the features extracted by the Transformer encoder. The activation function normalizes the probability of each class, and finally outputs the probability distribution of each class, as shown in the formula:

[0061] ;

[0062] in, To represent objects The probability distribution of belonging to each category. It is the weight matrix of the classification layer. It is a bias term. The function is used to transform the output into a probability distribution. Ultimately, the classification head outputs... Represents objects The probability of belonging to each category.

[0063] To address the high communication costs associated with transmitting the complete model weights in traditional federated learning, this invention designs an efficient federated cue communication module. The core of this module lies in maintaining the network parameters of the encoder and decoder in the local model unchanged (i.e., frozen) during local training, and only creating and optimizing a very small set of additional learnable parameters, known as the local cue pool. During communication, local clients only need to exchange this lightweight local cue pool, rather than the complete model weights containing the encoder, decoder, and detector head. The specific implementation of this module includes the following steps:

[0064] During local training, the local client updates only one set of learnable parameters, known as the local cue pool. In each round of communication, this local cue pool is uploaded to the global server as the sole update. Specifically, the local cue pool consists of multiple pairs of learnable cue embedding vectors, each pair being embedded into each layer of the encoder. The local cue pool initializes a pair of learnable cue embedding vectors for each layer of the encoder; this pair is called a prefix cue. Each pair of prefixes indicates... Specifically, this includes: a key hint vector and a value hint vector , wherein the key hint vector The value hint vector is used for concatenation with the key matrix in this level of self-attention mechanism. Used for concatenation with the value matrix in the self-attention mechanism of this level. The key hint vectors and value hint vectors of all levels within the encoder together constitute the local hint pool of the local client, denoted as... .

[0065] Each level within the encoder contains a multi-head self-attention block. The structure of this multi-head self-attention block includes multiple components for generating the query matrix. Key matrix Sum matrix A linear projection layer, a computational unit for performing multi-head self-attention computation, and a residual connection that adds the input of the block to the output of the computational unit.

[0066] During the forward propagation process of local training, when data flows through the multi-head self-attention block, the original query matrix is ​​first generated based on the input features through a linear projection layer. Key matrix Sum matrix .

[0067] Subsequently, the prefix suggestion described in this invention The original key matrix Sum matrix This will be expanded. Specifically, the key hint vector... With the original bond matrix The sequences are concatenated to form a new, expanded key matrix. Value hint vector With the original value matrix Concatenate along the sequence dimension This forms a new, expanded value matrix. The process can be represented as follows:

[0068] ;

[0069] ;

[0070] Subsequently, a query matrix was used. With this pair of extended key matrices Sum matrix Perform standard self-attention calculations: In this way, the local suggestion pool The parameters are deeply integrated into every step of feature extraction, and the feature extraction process can be fine-tuned based on local data.

[0071] Crucially, throughout the backpropagation and parameter update process of local training, the backbone network parameters of the encoder and decoder remain frozen and do not participate in gradient updates. Only those constituting the local cue pool... These prefix cue parameters are learned and optimized based on the gradient of the local loss function. Since the number of parameters in the cue pool is much smaller than that of the entire model, this greatly improves the efficiency of local training.

[0072] After local training is completed, each local client participating in this round of communication... Its optimized local hint pool, incorporating local data knowledge Uploaded to the global server. Since only a lightweight local hint pool is uploaded, the communication bandwidth requirements are significantly reduced.

[0073] Step S4: On the global server side, the hint pool aggregation module is responsible for collecting all local hint pools uploaded by local clients. And through a weighted average, it is aggregated into a global hint pool containing global knowledge. Let the set of local clients participating in this round of communication be . The total number of local clients in the collection is Global suggestion pool The aggregation formula is defined as follows:

[0074]

[0075] in, For the first The aggregate weight of each local client, For the first Local prompt pool uploaded by local clients, The number of local clients in the local prompt pool for uploaded content. This weight is based on the number of local clients. The number of training samples from each local client is determined by the proportion of the total number of local client samples participating in this round of communication. The calculation formula is as follows:

[0076] ;

[0077] In this weighting formula, Indicates the first The number of local training samples possessed by each local client. Indicates all Number of local clients participating in this round of communication Sum the results to get the total number of samples.

[0078] Finally, the global server will aggregate this global suggestion pool. The message is broadcast back to all local clients. Upon receiving it, the local clients use this global hint pool. Replace its own local cue pool as the starting point for the next round of local training.

[0079] Through the closed-loop process of creating, training, uploading, aggregating, and broadcasting the above-mentioned prompt pool, this invention can achieve effective knowledge collaboration among multiple local clients without transmitting large model weights, thereby significantly reducing communication costs while ensuring the final performance of the federated 3D object detection model.

[0080] Step S5: Each local client receives the updated global hint pool obtained in step S4. It then uses this global suggestion pool to replace its own local suggestion pool, thereby synchronizing the updated global knowledge to its local machine for the next round of training.

[0081] To address the data heterogeneity problem in federated 3D object detection, this invention proposes a model training method for local-global heterogeneity perception.

[0082] The core of this method is not to define a new loss function, but to adapt the loss function (i.e., the local loss function) commonly used in this field when updating model parameters using gradient descent on the local client. The gradient calculated by the loss function is dynamically weighted and corrected. In one embodiment of the invention, the loss function... This includes a cross-entropy loss function for classification tasks and a regression loss function for bounding box regression tasks. The gradient correction method described in this invention specifically applies to the gradient generated by the cross-entropy loss function. Specifically, the parameter update formula for the model parameters is as follows:

[0083] ;

[0084] in A set of learnable parameters representing the local client model. This indicates the current parameters of the model before this update. This indicates the new model parameters after this update. This represents the preset learning rate, a hyperparameter used to control the update step size. This represents a data sample currently used to compute the gradient, where For the input 3D point cloud data, Set its corresponding real label (including object category and bounding box position). Represents the local loss function In the current parameters and current sample The original gradient is calculated at [location]. This invention introduces a method specifically for samples. The local correction factor is designed to address the data imbalance problem within a single local client. This invention introduces a method specific to object categories. The global correction coefficient is designed to address the issue of heterogeneous data distribution among different local clients.

[0085] By introducing a local correction factor and global correction coefficient Each data sample The contribution of data is dynamically adjusted when updating model parameters, thereby effectively mitigating the negative impact of data imbalance and data heterogeneity.

[0086] Local correction factor This aims to solve the data imbalance problem within a single local client. For the first [unclear - possibly a specific type of batch processing]... Sample Local correction factor The calculation method is as follows:

[0087] ;

[0088] in , Represents the cross-entropy loss function The gradient relative to the output of the last layer of the classification head in the detection head, and this gradient is relative to the sample. The neurons corresponding to the true categories are calculated, where This is the output value of the neuron corresponding to the last layer of the classification head. Represents the gradient Take the absolute value. Represents all samples in this mini-batch The corresponding gradient The absolute values ​​are summed. Therefore, the local correction coefficient is... In essence, it is the first The normalized proportion of the gradient magnitude of an individual sample to the total gradient magnitude of the entire mini-batch. This proportion directly reflects the contribution of a single sample to the model parameter update. By... Multiplying into the parameter update formula can effectively balance the contributions of samples with different levels of difficulty or different categories within a small batch, thereby alleviating the problem of local data imbalance.

[0089] Global correction coefficient The aim is to address the heterogeneity of data distribution across different local clients in a way that protects data privacy. Global correction coefficient. The calculation consists of two steps:

[0090] First, in the Each local client needs to compute a representation that can represent the class distribution of its local dataset. :

[0091] ;

[0092] in, Represents a specific category of objects. This represents the category in the local client dataset. The formula calculates the total number of samples belonging to the category on this local client. The average gradient of the sample, this average gradient value It can reflect the local client's understanding of categories without uploading any raw point cloud data. The data ownership and training status. After each local client completes local training, it will store all the data it owns for all categories. The value is uploaded to the global server.

[0093] Then, on the global server side, the average gradient values ​​representing the distribution of each category uploaded by all local clients participating in this round of communication are aggregated. A collection of data. Based on this aggregated data, the server will assign each object category... Calculate the final global correction coefficients The calculation formula is as follows:

[0094]

[0095] in, It is a local client. Total number of samples This represents the total number of samples from all selected local clients. The formula is a logarithmic function whose value depends on the ratio between the aggregate representation of a specific category across all local clients and the total aggregate representation of all categories across all local clients. It also takes into account the data volume weights of different local clients to calculate a value that can represent the category. The coefficient representing the degree of imbalance under the current global data distribution. This coefficient will then be distributed back to each local client.

[0096] Through the aforementioned multiple rounds of local client-server interaction (i.e., local training, hint pool upload, global aggregation and broadcast), the global hint pool... This allows for continuous optimization. After completing all preset communication rounds, the server obtains a final global cue pool. This final global cue pool, combined with the pre-trained, parameter-frozen encoder and decoder, and the detection head, forms a final, deployable global 3D object detection model. Any local client can download and use this final global 3D object detection model to perform 3D object detection tasks. The goal of this detection task is to identify objects from input 3D point cloud data and accurately predict their bounding boxes and class labels.

[0097] Specifically, after the 3D point cloud data is processed by the final global 3D object detection model, the output includes the bounding box of the object (i.e., the position and size of the object in 3D space) and the object's category label (i.e., the category to which the object belongs). These outputs will serve as the final results of the detection task and will be used for subsequent applications, such as autonomous driving, robot navigation, or other tasks based on 3D object detection.

[0098] This invention provides a federated 3D object detection system based on anomaly perception and cueing communication, the system comprising:

[0099] The 3D object detection module, deployed on various local clients, is responsible for performing specific object detection tasks. This module includes a vision transformer-based encoder, decoder, and detection head, which are used to extract features from the input 3D point cloud data and predict the bounding boxes and categories of objects.

[0100] The Federated Cue Communication Module is responsible for efficient and secure information exchange between local clients and the global server. This module collaboratively trains the 3D object detection module distributed across various local clients by transmitting only a lightweight, learnable pool of local cues, rather than the complete model weights. The Federated Cue Communication Module specifically includes the following sub-modules:

[0101] Local cue pool generation module: Deployed on each local client. This module's function is to generate and optimize a local cue pool during local training. Specifically, within the encoder of the 3D object detection module, this module creates a pair of learnable prefix cues for each encoder level. These prefix hints together constitute the local hint pool. During local training, the backbone network parameters of the 3D object detection module remain frozen, with only the local cue pool used. The parameters participate in gradient updates. It works by using prefix hints... Key to the attention mechanism within the encoder Matrix sum value The matrices are concatenated to fine-tune the feature extraction process based on local data without changing the weights of the backbone network.

[0102] Hint pool upload module: Deployed on each local client. After local training is complete, this module is responsible for uploading the optimized local hint pool. As the sole model update content, it is sent to the global server via the communication network.

[0103] Hint Pool Aggregation Module: Deployed on the global server. This module is responsible for receiving local hint pools uploaded by all local clients and using a weighted average algorithm to merge the uploaded local hint pools into a global hint pool containing global knowledge. The weights used for the weighted average are determined based on the proportion of training samples from each local client to the total number of samples from all local clients.

[0104] Global Notification Pool Broadcast Module: Deployed on the global server. This is used when generating the global notification pool. Then, this module is responsible for distributing it to all local clients to replace the original local hint pool of the local clients and to serve as the starting point for the next round of local training.

[0105] This embodiment studies the convergence of the HPC model in the Fed3D task and uses convergence curves to demonstrate the model's performance under different communication rounds. Figure 3 The results show the variation of mAP@0.25 and mAP@0.5 metrics with the number of communication rounds on different datasets (such as SUN RGB-D and ScanNet V2). After approximately 60 communication rounds, the convergence curve becomes more sluggish, indicating that the model has essentially converged. This phenomenon validates the effectiveness and robustness of the HPC model in solving the Fed3D task, demonstrating its ability to handle heterogeneity issues under different local clients and data settings, and ultimately achieving a stable high performance level. This convergence analysis result shows that the HPC model can achieve good convergence when handling different data distributions and has an effective ability to solve the Fed3D problem.

[0106] Figure 3 Figure (a) shows the experimental results on the SUN RGB-D dataset when the total number of local clients is 20 (K=20). As can be seen from the figure, the mean accuracy (mAP) of all methods steadily improves with increasing communication rounds. However, the curve representing the method of this invention consistently stays at the top of all curves after approximately 25 communication rounds, demonstrating optimal performance. Especially in the later stages of training (75-100 rounds), the method of this invention outperforms the suboptimal method (such as MOON) by about 2-3 percentage points, and exhibits more stable convergence with less jitter.

[0107] Figure 3 (b) shows the experimental results on the SUN RGB-D dataset when the total number of local clients is increased to 100 (K=100). This is a scenario with a larger number of local clients and potentially more heterogeneous data distribution. Under this setting, the advantages of the method of the present invention (Ours) are more pronounced. Starting from approximately the 20th communication round, its performance curve significantly outperforms all other methods and maintains this lead throughout the training process. This demonstrates that the method of the present invention is more robust and more efficient in handling large-scale, highly heterogeneous federated learning tasks.

[0108] Figure 3 (c) shows the performance on the ScanNet V2 dataset with 20 local clients (K=20). Similar to its performance on SUNRGB-D, the performance curve of our method (Ours) shows an initial rapid climb, gradually reaching and stabilizing at the highest level among all methods starting from approximately the 50th communication round. This demonstrates that the effectiveness of our method is not limited to a single dataset, but exhibits good generalization ability and superior performance in different data scenarios.

[0109] Figure 3 Figure (d) illustrates the case where the number of local clients increases to 100 (K=100) on the ScanNet V2 dataset. This is also a more challenging setting. The figure shows that the performance curve of our method (Ours) is consistently leading throughout the training process. Especially after 50 rounds, its performance steadily outperforms all other comparative methods, demonstrating consistent superiority in large-scale local client scenarios across different datasets. This further validates the powerful ability of our invention to address the problem of data heterogeneity.

[0110] The following experiments are based on the above methods. The experimental method in this embodiment is as described above, and the specific implementation steps will not be elaborated in detail. The results will only be presented to demonstrate the effectiveness of the experiments. The model is evaluated through several empirical comparisons between two benchmark datasets.

[0111] The SUN RGB-D dataset contains 10,335 RGB-D and depth images, covering 37 object categories. Following the VoteNet standard split, 5,285 images were used for training and 5,050 images for testing. All point clouds were extracted from these RGB-D images, yielding 3D bounding boxes and object category annotations from the original dataset. Evaluation metrics were mean precision (mAP) and mean recall (AR) at IoU thresholds of 0.25 and 0.5.

[0112] The ScanNet V2 dataset was divided into 1,201 scenes for training and 312 scenes for testing. Standard evaluation metrics were used: mAP and AR with IoU thresholds of 0.25 and 0.5, respectively, regardless of bounding box orientation.

[0113] This embodiment constructs a 3D detector comprising a scaled PointNet and a pre-trained 6-layer ViT encoder. It is assumed that a global server can collect public 3D data and pre-train in a self-supervised manner. All training processes use only 3D point cloud data, without supervision of categories or 3D bounding boxes. The robustness of the HPC model is evaluated in two federated local client settings (i.e., C = 100, δ = 0.1 and C = 20, δ = 0.25), where C represents the number of local clients and δ represents the proportion of local clients selected in each round of communication. To simulate the heterogeneity of 3D data, 70% of the categories and 70% of the samples are assigned to each local client. All experiments were conducted with 100 rounds of global communication and 4 rounds of local training.

[0114] To evaluate the effectiveness of the HPC model, five standard federated learning methods were applied in the Fed3D setting, including FedAvg, FedProx, SCAFFOLD, MOON, and FedDyn. The comparison results are shown in Table 1.

[0115] Table 1 shows the implementation effect of the method of the present invention on the SUNRGBD dataset.

[0116]

[0117] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A federated 3D object detection method based on heterogeneity-based perceptual cueing communication, characterized in that... include: The local client inputs the 3D point cloud data to be detected into the trained global 3D object detection model; The global 3D object detection model outputs the bounding boxes of objects and the object category; The training of the global 3D object detection model includes: Collect 3D point cloud data from multiple local clients; Preprocess the collected 3D point cloud data; The local client trains the local model using preprocessed 3D point cloud data. After local training is completed, each local client participating in this round of communication uploads a local hint pool containing local data knowledge to the global server. The global server collects all local hint pools uploaded by local clients, aggregates all local hint pools into a global hint pool containing global knowledge, and broadcasts the global hint pool back to all local clients. Each local client receives the global cue pool and uses it to update its local model for the next round of training; In each training iteration of the local client, a local correction coefficient is first calculated for each data sample in the mini-batch data. At the same time, each local client participating in this round of communication also needs to calculate and upload one or more class distribution representations to the global server. Each class distribution representation is for a specific object class in the local dataset and is obtained by calculating the average gradient value of all samples belonging to the class. Subsequently, on the global server side, the category distribution representations uploaded by all local clients are aggregated, and a global correction coefficient is calculated for each object category based on the aggregation result and the logarithmic function relationship. The global correction coefficient is then distributed to each local client. Finally, when updating the model parameters using gradient descent on the local client, the original gradient generated by the local loss function is multiplied by the calculated local correction coefficient corresponding to the current sample and the global correction coefficient corresponding to the object category of the current sample received from the server to obtain a corrected gradient, and the corrected gradient is used to update the parameters of the local model.

2. The federated 3D object detection method based on heterogeneity-based perceptual cueing communication according to claim 1, characterized in that, The local model includes a vision transformer-based encoder, a decoder, and a detection head. The vision transformer-based encoder is composed of multiple stacked layers, and each layer contains a multi-head self-attention block and a Transformer encoder.

3. The federated 3D object detection method based on heterogeneity-based perceptual cueing communication according to claim 2, characterized in that, The local cue pool consists of cue embedding vectors embedded in each layer of the visual transformer-based encoder, known as prefix cueing. During the forward propagation of local training, when computing the multi-head self-attention blocks in each layer of the visual transformer-based encoder, the prefix cueing concatenates the original key and value matrices of the multi-head self-attention blocks in the sequence dimension to form new, expanded key and value matrices. Subsequently, the query matrix and the expanded key and value matrices are used to perform standard self-attention computation.

4. The federated 3D object detection method based on heterogeneity-aware cueing communication according to claim 1, characterized in that, When training a local model on a local client, only the parameters of the local cue pool are learned and optimized based on the gradient of the loss function during the backpropagation and parameter update process throughout the training.

5. The federated 3D object detection method based on heterogeneity-aware cueing communication according to claim 1, characterized in that, The local correction coefficient is the normalized proportion of the gradient magnitude of a single sample in the total gradient magnitude of the entire mini-batch, and the proportion reflects the contribution of a single sample to the update of model parameters.

6. A system for implementing the federated 3D object detection method based on heterogeneity-based perceptual cueing communication as described in claim 1, characterized in that, include: The 3D object detection module includes a vision transformer-based encoder, decoder, and detection head, used to extract features from input 3D point cloud data and predict the bounding box and category of the object; The federated cue communication module collaboratively trains the 3D object detection module distributed across local clients by transmitting only a lightweight, learnable pool of local cues.

7. The system according to claim 6, characterized in that, The Federal Notification Communication Module Includes the following sub-modules: Local cue pool generation module: Generates and optimizes a local cue pool during local training. Specifically, within the encoder of the 3D object detection module, a pair of learnable prefix cues is created for each encoder level, and the prefix cues are used together to form the local cue pool. Hint pool upload module: After local training is completed, the optimized local hint pool is used as the only model update content and sent to the global server through the communication network; Hint pool aggregation module: responsible for receiving all local hint pools uploaded by local clients, and using a weighted average algorithm to merge the uploaded local hint pools into a global hint pool containing global knowledge; Global cue pool broadcast module: responsible for distributing the global cue pool to all local clients to replace the original local cue pool of the local clients and to serve as the starting point for the next round of local training.

8. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, the processor causes the processor to implement the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 5.

Images