Autonomous driving federated learning method and system based on perception decoupling and manifold alignment

Abstract

The application discloses an automatic driving federated learning method and system based on perception decoupling and manifold alignment. The method collects road condition data through a vehicle-mounted sensor, and after extracting basic features through a heterogeneous perception network, the data is shunted to a double-head decoupling architecture. A private control head outputs local real-time risk avoidance instructions, and a public feature projection head is mapped into a unified dimension vehicle-cloud collaborative representation vector. A cloud end maintains a global prototype set and a global traffic semantic topology atlas and issues them to vehicles. Vehicles perform point-level comparison and alignment and structure-level alignment on the representation vector, so that the similarity distribution within the batch approximates the global traffic semantic topology atlas. A feature correlation matrix is calculated for the representation matrix and subjected to an orthogonal constraint to force each dimension to be independent. Vehicles update the network with a weighted total loss and upload local prototypes. The cloud end dynamically weights and aggregates global prototypes according to cosine similarity, reconstructs the global traffic semantic topology atlas, and broadcasts it. The application realizes cross-regional perception of vehicle automatic driving.

Description

Technical Field

[0001] This invention relates to the field of cooperative perception and control technology for autonomous driving and intelligent connected vehicles, and in particular to a federated learning method and system for autonomous driving based on perception decoupling and manifold alignment. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, the safety and intelligence level of autonomous driving systems have become a core focus of the industry. The performance of autonomous driving systems heavily relies on massive amounts of diverse real-world road condition data (such as urban congestion, extreme weather conditions, and rare obstacles) to train their onboard perception and decision-making models. However, in traditional centralized machine learning architectures, transmitting massive amounts of high-definition video streams and radar point cloud data generated by millions of vehicles during daily driving to the cloud in real time not only puts unbearable pressure on communication bandwidth but also seriously infringes on the privacy and geographic location information of passengers inside the vehicle.

[0003] Against this backdrop, the Federated Learning architecture demonstrates a natural fit with autonomous driving scenarios. In this architecture, the cloud-based autonomous driving server acts as a "central aggregation server," while each autonomous vehicle traveling in various locations acts as a "distributed autonomous vehicle." Vehicles locally utilize the road condition data they collect, relying on onboard computing platforms (such as onboard computers) to run local models for localized training, including road condition analysis, pedestrian detection, and obstacle recognition. Subsequently, the vehicles only encrypt and upload the trained model parameters or feature representations to the cloud. The cloud-based autonomous driving server collects the parameters from each vehicle, performs unified aggregation and analysis, and then distributes the globally updated model to all vehicles. This mechanism breaks down data silos and enables the sharing of global driving experience while ensuring that "data does not leave the vehicle."

[0004] However, in real-world autonomous driving federated collaboration scenarios, the system inevitably enters a complex and challenging environment characterized by both "data heterogeneity" and "model heterogeneity." Existing federated learning frameworks (such as those relying solely on branch decoupling or point-level alignment methods based on class prototypes) essentially remain at the level of abstract mathematical feature space optimization, lacking deep integration with the heterogeneity of autonomous driving physical sensors (such as LiDAR point clouds and high-definition camera pixels) and the responses of underlying drive-by-wire chassis actuators. Furthermore, due to the extreme scarcity of long-tail extreme road condition data in the physical world, relying solely on real-world road condition data makes it difficult to achieve sufficient safe redundancy iteration of the vehicle perception model. These frameworks, when dealing with the complex physical world, expose the following insurmountable key engineering bottlenecks:

[0005] First, environmental confounding factors cause physical semantic topological distortion (loss of causal relationships). Existing federated learning only pursues minimizing the mathematical distance between isolated feature vectors and the global centroid. In autonomous driving, vehicles in different regions are severely affected by weather (such as blizzards and fog) and lighting conditions. If only point-level alignment is performed, heterogeneous vehicular networks are prone to fitting "environmental confounding factors" (e.g., incorrectly binding the features of snow to drivable areas), disrupting the inherent high-order physical topological relationships between physical entities such as "pedestrians," "zebra crossings," and "vehicles," leading to severe "perceptual geometry drift" when deployed across regions.

[0006] Second, the "representation collapse" of the perception channel caused by conforming to the global mean leads to blind spots in long-tail extreme road conditions (corner cases). In heterogeneous federated collaboration, vehicles with weaker computing power or simpler data exhibit "learning laziness" in order to conform to the average features of the global model, excessively compressing the high-dimensional feature space. In engineering performance, the onboard model will use most of the computational dimensions to identify the most frequent "straight lanes" or "standard passenger vehicles," causing the dimensions for extracting fine features such as body material and edge reflections to become invalid. Once encountering unknown long-tail obstacles (such as the wreckage of irregularly shaped engineering vehicles) on open roads, the vehicle will completely lose its ability to identify emergency avoidance because the multi-dimensional perception channel has essentially collapsed.

[0007] Third, the representation collapse dilemma caused by corner cases. The core of autonomous driving safety lies in the identification of rare and extreme road conditions. However, in federated collaboration lacking effective geometric constraints, heterogeneous onboard models, in order to cater to global average trends, often tend to squeeze and overlap high-dimensional features into extremely low-dimensional subspaces (i.e., extracting only the most common features such as "straight lane lines"). This leads to the invalidation of a large number of redundant features used to identify complex and rare obstacles (such as overturned irregularly shaped construction vehicles or scattered cargo), causing "representation collapse" and severely weakening the autonomous driving system's out-of-distribution generalization and crisis response capabilities on unknown open roads.

[0008] Therefore, how to preserve the local perception specificity of a single vehicle in a dual heterogeneous environment of vehicle-cloud collaboration, achieve consistency of the global complex traffic semantic structure, and avoid degradation of the long-tail feature space has become a key technical problem that urgently needs to be solved in the field of autonomous driving model training. Summary of the Invention

[0009] The purpose of this invention is to address the problems of traffic semantic-physical-topological distortion caused by the susceptibility of existing autonomous driving perception systems to interference from false environmental features when facing cross-regional meteorological environmental interference and differences in computing power of heterogeneous vehicle chips, as well as the problem of multi-dimensional perception channel representation collapse and failure when catering to global high-frequency conventional road conditions and dealing with unknown long-tail extreme road conditions (corner cases). The invention provides an autonomous driving federated learning method and system based on perception decoupling and manifold alignment, endowing heterogeneous vehicle perception models with causal interpretable perception capabilities to cope with complex open physical worlds.

[0010] The technical solution adopted in this invention is:

[0011] An autonomous driving federated learning method based on perception decoupling and manifold alignment is applied to a vehicle-to-everything (V2X) system consisting of a cloud-based autonomous driving server and multiple autonomous vehicles equipped with heterogeneous onboard computing platforms and onboard physical sensors. The method includes the following steps:

[0012] S1, each autonomous vehicle collects local real-world road condition data through onboard physical sensors, inputs it into the local heterogeneous feature extractor of the locally deployed heterogeneous perception network (adapted to its underlying hardware computing power) to extract basic road condition features; the basic road condition features are diverted to the dual-head perception decoupled architecture, and the local real-time road condition analysis results are obtained through the private control head to execute the emergency avoidance command of the drive-by-wire chassis; the basic road condition features are mapped into a unified-dimensional vehicle-cloud collaborative representation vector through the common feature projection head;

[0013] S2, a cloud-based autonomous driving server maintains a global prototype set and a global traffic semantic topology map. ,in For the number of traffic physical entity categories, Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between them; the cloud-based autonomous driving server distributes the global prototype set and the global traffic semantic topology map G to each autonomous vehicle; during local training, the autonomous vehicles perform two-level alignment on the vehicle-cloud collaborative representation vector z: point-level alignment and structure-level alignment. Point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototypes of other categories as the negative anchor, and uses contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push away the distance from the negative anchor; structure-level alignment takes the vehicle-cloud collaborative representation matrix within the local batch. Calculate the intra-batch similarity distribution and use KL divergence to constrain the intra-batch similarity distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G;

[0014] S3, calculate the feature correlation matrix from the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension mutually orthogonal.

[0015] In S4, the autonomous vehicle obtains the total loss by weighting and summing the local private perception loss, point-level alignment loss, structural alignment loss, and orthogonal decorrelation loss, and then backpropagates to update the parameters of the local heterogeneous perception network and dual-head architecture. The autonomous vehicle uploads the updated local prototypes of each category to the cloud autonomous driving server. The cloud autonomous driving server calculates the cosine similarity between the local model of each autonomous vehicle and the initial global model in order to allocate dynamic aggregation weights, weights and aggregates the local prototypes to obtain the updated global prototype set, and reconstructs the global traffic semantic topology graph G and broadcasts it to the next communication round.

[0016] Furthermore, the heterogeneous in-vehicle computing platform includes low-computing-power in-vehicle chips and high-computing-power flagship in-vehicle chips; the forward computation and parameter updates of the private control head are all completed locally in the autonomous vehicle; the network scale of the private control head is strongly bound to the underlying computing power and drive-by-wire chassis control latency requirements of the corresponding autonomous vehicle, and the processing latency meets the millisecond-level response requirements of in-vehicle emergency avoidance; the public feature projection head compresses the high-dimensional non-standardized features output by each heterogeneous perception network into a standard vector of the same low-dimensional dimension, enabling autonomous vehicles with different computing power to access the same vehicle-cloud collaborative representation space.

[0017] Furthermore, in S1, the autonomous vehicle builds a local scene generator locally, using the previous round global model issued by the cloud-based autonomous driving server as the teacher model and the current local model as the student model. It conducts adversarial training by maximizing the KL divergence between the two in the classification prediction output to update the local scene generator. This allows the local scene generator to simulate and output long-tailed difficult pseudo-road condition data that causes the greatest discrepancy between the local model and the global model prediction. The autonomous vehicle uses the long-tailed difficult pseudo-road condition data to perform local knowledge distillation on the local heterogeneous perception network, and performs closed-loop optimization of the local heterogeneous feature extractor of the local heterogeneous perception network.

[0018] Furthermore, the private control head in S1 calculates the local prediction, and the corresponding expression is:

[0019] ;

[0020] in, For private control headers; Basic road condition characteristics; For the first Trainable network parameters corresponding to the private control head of an autonomous vehicle;

[0021] Calculate the private-perception cross-entropy loss The expression is:

[0022] (1);

[0023] in, To handle the batch size of data, This represents the total number of traffic entity categories. Tag the actual road conditions; For the first The corresponding road condition sample output by the private control head is the [number]th [item]. Predicted probability values ​​for traffic-like entities; This is an indicator function that takes the value 1 when the condition within the parentheses is true and 0 when the condition is false.

[0024] Furthermore, point-level alignment loss The calculation formula is as follows:

[0025] (2);

[0026] in, This is the vehicle-cloud collaboration representation vector. and These are the global prototypes for positive and negative traffic entities, respectively. Temperature coefficient; This represents the total number of traffic entity categories.

[0027] Furthermore, the structural alignment loss is calculated using KL divergence constraints. The calculation formula is as follows:

[0028] (3);

[0029] in, This is the road condition feature matrix within the batch. It is a global traffic semantic topology graph that includes high-order relationships between traffic entities. For the Softmax function; Temperature coefficient; Tag the actual road conditions; For the first batch The vehicle-cloud collaborative representation vector corresponding to each road condition sample; This is a set of labels for traffic entities within a batch.

[0030] Furthermore, the orthogonal decorrelation regularization loss is calculated. The calculation formula is as follows:

[0031] (4);

[0032] in, The dimension of the vehicle-cloud collaborative representation vector; The first element in the feature correlation matrix Line 1 The elements of the column, that is, the elements representing the first... The representation dimension and the first The degree of linear correlation between the representation dimensions; The first element in the feature correlation matrix Line 1 The element of the column, i.e., the first Autocorrelation coefficients of each representation dimension;

[0033] Furthermore, total losses The expression is as follows:

[0034] (5);

[0035] in, For balancing weighting coefficients; For privately perceived cross-entropy loss; Point-level alignment loss; This is for structural alignment loss; The loss is the orthogonal decorrelation regularization loss.

[0036] Furthermore, the global traffic semantic topology map is constrained as follows: when facing environmental differences with strong visual interference, the divergence algorithm is used to force the similarity distribution within the local feature batch to approximate the global traffic semantic topology map, so that the high-order relative geometric distances of "pedestrians" and "zebra crossings" extracted from the underlying pixel difference data collected under sunny or snowy weather are rigidly aligned with the global consensus.

[0037] Furthermore, the implementation of orthogonal decorrelation regularization includes: penalizing the off-diagonal elements of the feature correlation matrix by approximating the operation of the identity matrix, forcing the first feature dimension to extract the physical contour of the obstacle, the second feature dimension to extract the material texture, and the third feature dimension to extract the surface reflective properties, ensuring the absolute independence of the multi-dimensional perception channels and avoiding the perception network from "representation collapse" and failure in extreme long-tail scenarios.

[0038] Furthermore, in S4, the cloud-based autonomous driving server weights and aggregates the local scene generator parameters uploaded by each autonomous vehicle to obtain a global scene generator. The global scene generator is used to generate global pseudo-road condition features. The global pseudo-road condition features are then input into the local model and the initial global model of each autonomous vehicle, respectively. Global knowledge distillation is performed on the initial global model to minimize the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

[0039] Furthermore, the cloud-based autonomous driving server in S4 updates the global prototype by aggregating the local prototypes uploaded by each autonomous vehicle. And calculate inter-class similarity based on global prototype. Reconstruct the global traffic semantic topology graph for the next round.

[0040] Furthermore, before the first federated communication with the cloud-based autonomous driving server, the autonomous vehicle performs local pre-training on the heterogeneous perception network using cross-entropy loss for several iterations.

[0041] Furthermore, the present invention also discloses an autonomous driving federated learning system based on perception decoupling and manifold alignment, comprising:

[0042] A cloud-based autonomous driving server is used to maintain a global prototype set and a global traffic semantic topology graph. ,in For the number of traffic physical entity categories, Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between the models is calculated; the global prototype set and the global traffic semantic topology map G are distributed to each autonomous vehicle; the local prototypes of each category uploaded by each autonomous vehicle are received, the cosine similarity between the local model of each autonomous vehicle and the initial global model is calculated as the dynamic aggregation weight, and the local prototypes are weighted and aggregated to update the global prototype set; the global traffic semantic topology map G is reconstructed according to the updated global prototype set and broadcast to the next communication round.

[0043] Multiple autonomous vehicles, each equipped with onboard physical sensors, a heterogeneous onboard computing platform, and a drive-by-wire chassis. The heterogeneous onboard computing platform is configured with a heterogeneous perception network and a dual-head perception decoupling architecture. The heterogeneous perception network is used to extract basic road condition features from local real-world road condition data collected by the onboard physical sensors. The dual-head perception decoupling architecture includes a private control head, a common feature projection head, a manifold geometry alignment module, an orthogonal decorrelation regularization module, and a local joint update module. The private control head is used to map basic road condition features to a traffic physical entity label space, output local real-time road condition analysis results, and drive the drive-by-wire chassis to execute emergency avoidance commands. The parameters of this private control head do not participate in the vehicle-cloud federation. Interaction; The common feature projection head is used to map basic road condition features into a unified-dimensional vehicle-cloud collaborative representation vector. The unified dimension is consistent across all heterogeneous vehicle computing platforms, serving as a common feature space for vehicle-cloud federation interaction; The manifold geometry alignment module is used to perform two-layer alignment of the vehicle-cloud collaborative representation vector during local training: point-level alignment and structure-level alignment; Point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototype of other categories as the negative anchor, employing contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push away the distance from the negative anchor; Structure-level alignment takes the vehicle-cloud collaborative representation matrix within the local batch, calculates the similarity distribution within the batch, and uses KL divergence to constrain this distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G; The orthogonal decorrelation regularization module is used to calculate the feature correlation matrix of the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension orthogonal to each other. The local joint update module is used to obtain the total loss by weighted summation of local private perception loss, point-level alignment loss, structural alignment loss and orthogonal decorrelation loss, and backpropagates to update the parameters of the local heterogeneous perception network and dual-head architecture. The updated local prototypes of each category are then uploaded to the cloud autonomous driving server.

[0044] Furthermore, the heterogeneous in-vehicle computing platform includes low-computing-power in-vehicle chips and high-computing-power flagship in-vehicle chips; the network scale of the private control head is strongly tied to the underlying computing power and drive-by-wire chassis control latency requirements of the corresponding autonomous vehicle, and its forward computation and parameter updates are completed locally in the autonomous vehicle, with processing latency meeting the millisecond-level response requirements of in-vehicle emergency avoidance; the common feature projection head compresses the high-dimensional non-standardized features output by each heterogeneous perception network into a standard vector of the same low-dimensional dimension, enabling autonomous vehicles with different computing power to access the same vehicle-cloud collaborative representation space.

[0045] Furthermore, the autonomous vehicle is also equipped with a vehicle-side scene generator. The vehicle-side scene generator uses the previous round of global perception model issued by the cloud autonomous driving server as the teacher model and the local current model as the student model. It maximizes the KL divergence between the two in the classification prediction output to update its own parameters. This allows the vehicle-side scene generator to output long-tailed difficult pseudo-road condition data that makes the local model and the global model prediction most divergent. The pseudo-road condition data is then used to perform local knowledge distillation closed-loop optimization on the local heterogeneous perception network.

[0046] The cloud-based autonomous driving server is also used to receive updated scene generator parameters uploaded by each autonomous vehicle, weighted aggregate to obtain a global scene generator, use the global scene generator to generate global pseudo-road condition features, input the global pseudo-road condition features into the local model and the initial global model of each autonomous vehicle respectively, and perform global knowledge distillation fine-tuning on the initial global model in a way that minimizes the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

[0047] The present invention adopts the above technical solution and has the following beneficial effects compared with the prior art:

[0048] (1) By adapting to the perception decoupling architecture and gentle start-up mechanism of heterogeneous computing power, physical isolation between local real-time risk avoidance of autonomous vehicles and global experience sharing between vehicle and cloud is realized. The private control head is strongly bound to the computing power of the underlying drive-by-wire chassis of the autonomous vehicle. Its forward inference latency meets the millisecond-level emergency braking requirements, ensuring that the autonomous vehicle can still independently complete the autonomous risk avoidance response to immediate dangerous situations such as pedestrians crossing and sudden obstacles in any scenario of federated communication interruption or cloud autonomous driving server aggregation latency. The public feature projection head compresses the heterogeneous features output by autonomous vehicles with different computing power levels into the same dimensional space, enabling older models and high-end models to exchange knowledge at the same semantic level.

[0049] (2) This invention defines high-order rigid geometric relationships between traffic physical entities such as "pedestrian-zebra crossing", "vehicle-lane line", and "obstacle-drivable area" through a global traffic semantic topology graph G. In scenarios with strong visual interference such as blizzards, heavy fog, and backlighting, this graph constrains the similarity distribution within local feature batches to remain consistent with the global consensus, forcing the vehicle-mounted perception network to remove false correlations from environmental confounding factors such as meteorological particles and lighting angles. This results in a lower rate of accuracy reduction in the identification of traffic participants when autonomous vehicles migrate from sunny urban areas to snow-covered areas compared to traditional federated alignment methods, fundamentally eliminating perceptual geometric drift and "semantic confusion" caused by overfitting to regional environmental backgrounds.

[0050] (3) This invention constructs multi-dimensional independent safety perception redundancy channels for autonomous driving systems under unknown long-tail extreme road conditions through orthogonal decorrelation regularization. This mechanism forces each dimension of the vehicle-cloud collaborative representation to focus on the independent physical characteristics of obstacles, such as physical contours, material texture reflectivity, and surface thermal radiation features, thereby maximizing the information entropy of the feature space. When a vehicle encounters irregularly shaped obstacles on open roads, it can still activate at least three independent feature extraction channels for parallel judgment, rather than the "representation collapse" failure of multi-dimensional perception channels caused by the dominance of high-frequency conventional road condition representations as in traditional methods. This invention significantly improves the survival rate and risk avoidance success rate of autonomous driving systems in unseen scenarios. Attached Figure Description

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0052] Figure 1 This is an overall flowchart of the autonomous driving federated learning method based on perception decoupling and manifold alignment of the present invention;

[0053] Figure 2 This is a diagram illustrating the interaction framework between the cloud-based autonomous driving server and the autonomous vehicle of this invention.

[0054] Figure 3 This is a detailed flowchart of the local training process for autonomous vehicles according to the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0056] like Figures 1 to 3 As shown, this invention discloses a federated learning method for autonomous driving based on perception decoupling and manifold alignment, applied to a vehicle network system consisting of a cloud-based autonomous driving server and multiple autonomous vehicles equipped with heterogeneous onboard computing platforms and onboard physical sensors. The method includes the following steps:

[0057] S1, each autonomous vehicle collects local real-world road condition data through onboard physical sensors (such as LiDAR and cameras), inputs it into a local heterogeneous feature extractor of a locally deployed heterogeneous perception network (adapted to its underlying hardware computing power) to extract basic road condition features; the basic road condition features are physically forced to be decoupled to a dual-head perception decoupling architecture, and a private control head isolated from the federated interaction retains millisecond-level low-latency responses to local high-frequency road conditions to obtain local real-time road condition analysis results for executing emergency avoidance commands from the drive-by-wire chassis; the high-dimensional, non-standardized basic road condition features are mapped to a unified-dimensional low-dimensional vehicle-cloud through a common feature projection head. Collaborative representation vectors are used; simultaneously, the autonomous vehicle constructs a local scene generator locally, using the previous round global model issued by the cloud-based autonomous driving server as the teacher model and the current local model as the student model. Adversarial training is conducted by maximizing the KL divergence between the classification prediction results of the local model and the global model to update the local scene generator. This allows the local scene generator to simulate and output long-tailed, difficult pseudo-road condition data that maximizes the discrepancy between the local model and the global model's predictions. The autonomous vehicle then uses the long-tailed, difficult pseudo-road condition data to perform local knowledge distillation on the local heterogeneous perception network, thus optimizing the local heterogeneous feature extractor of the local heterogeneous perception network in a closed loop.

[0058] S2, a cloud-based autonomous driving server maintains a global prototype set and a global traffic semantic topology map. ,in For the number of traffic physical entity categories, Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between them; the cloud-based autonomous driving server distributes the global prototype set and the global traffic semantic topology map G to each autonomous vehicle; the autonomous vehicles use the global prototype set to perform two levels of alignment on the vehicle-cloud collaborative representation vector z during local training: point-level alignment and structure-level alignment; in the common feature space, point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototypes of other categories as the negative anchor, and uses contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push away the distance from the negative anchor, that is, to bring the local embedded vector closer to the global prototype of the same class and push away the distance from the global prototype of the negative class; the structure-level alignment uses the global traffic semantic topology map distributed by the cloud-based autonomous driving server to take the vehicle-cloud collaborative representation matrix within the local batch. Calculate the intra-batch similarity distribution and use KL divergence to constrain the intra-batch similarity distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G;

[0059] S3, calculate the feature correlation matrix from the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension mutually orthogonal to maximize the information entropy of the feature space and break the representation collapse dilemma.

[0060] In S4, the autonomous vehicle obtains the total loss by weighting and summing the local private perception loss, point-level alignment loss, structural alignment loss, and orthogonal decorrelation loss, and then backpropagates to update the parameters of the local heterogeneous perception network and dual-head architecture. The autonomous vehicle uploads the updated local prototypes of each category to the cloud autonomous driving server. The cloud autonomous driving server calculates the cosine similarity between the local model of each autonomous vehicle and the initial global model in order to allocate dynamic aggregation weights, weights and aggregates the local prototypes to obtain the updated global prototype set, and reconstructs the global traffic semantic topology graph G and broadcasts it to the next communication round.

[0061] Furthermore, the heterogeneous in-vehicle computing platform includes low-computing-power in-vehicle chips and high-computing-power flagship in-vehicle chips; the forward computation and parameter updates of the private control head are all completed locally in the autonomous vehicle; the network scale of the private control head is strongly bound to the underlying computing power and drive-by-wire chassis control latency requirements of the corresponding autonomous vehicle, and the processing latency meets the millisecond-level response requirements of in-vehicle emergency avoidance; the public feature projection head compresses the high-dimensional non-standardized features output by each heterogeneous perception network into a standard vector of the same low-dimensional dimension, enabling autonomous vehicles with different computing power to access the same vehicle-cloud collaborative representation space.

[0062] Furthermore, the implementation of the dual-head perception decoupling architecture in S1 includes:

[0063] S10, Gentle Start: Before the first round of federal communication, the autonomous vehicle pre-trains the feature extractor locally using only cross-entropy loss for a few iterations to ensure that the model has extracted features with basic discriminative power before participating in manifold alignment.

[0064] S11. Basic Road Condition Feature Extraction: Local Heterogeneous Feature Extractor For the input local traffic data sample (Such as real-time video streams collected by vehicle-mounted cameras or radar point cloud data) Output basic road condition features ,in These are local parameters.

[0065] S12, Private Control Header Mapping: Mapping basic road condition features The input is a local private control head that is strongly bound to the vehicle's underlying computing power. This is directly mapped to the traffic physical entity label space to calculate the predicted output for local real-time traffic conditions; the expression for calculating the local prediction by the private control head is:

[0066] ;

[0067] in, For private control headers; Basic road condition characteristics; These are the real-time control parameters of the private control head of the i-th autonomous vehicle;

[0068] Calculate the private-perception cross-entropy loss The expression is:

[0069] (1);

[0070] in, To handle the batch size of data, This represents the total number of categories of traffic entities (such as pedestrians, vehicles, obstacles, etc.). Tag the actual road conditions; For the first The corresponding road condition sample output by the private control head is the [number]th [item]. Predicted probability values ​​for traffic-like entities; This is an indicator function; it takes the value 1 when the condition within the parentheses is true and 0 when the condition is false. This path does not participate in subsequent vehicle-to-cloud interactions; its calculations are performed locally. Its purpose is to output local real-time traffic analysis results and execute avoidance commands such as emergency braking, ensuring low latency and absolute real-time performance of autonomous driving control.

[0071] S13, Common Feature Projection Head Mapping: Mapping basic road condition features Input common feature projection head Mapped into a unified-dimensional common feature space, a common low-dimensional vehicle-cloud collaborative representation vector is obtained. In the formula, This is the vehicle-cloud collaboration representation vector; These are the trainable network parameters for the common feature projection head. This space serves as a common linguistic environment for all heterogeneous autonomous vehicles to directly exchange knowledge, match dimensions, and calculate similarity.

[0072] Furthermore, the two-layer alignment in step S2 includes point-level alignment and structural alignment; specifically as follows:

[0073] S20, Point-level Alignment (Point-level Prototype Comparative Learning): Targeting Local Low-Dimensional Vehicle-Cloud Collaborative Representation Vectors and their corresponding traffic entity category labels (e.g., pedestrians, obstacles ahead), using the InfoNCE loss form, based on a global prototype issued by a cloud-based autonomous driving server. For anchor points, point-level alignment loss The calculation formula is as follows:

[0074] (2);

[0075] in, This is the vehicle-cloud collaboration representation vector. For the first The global prototype of positive-class traffic entities (i.e., entities of the same type) in round-robin communication. This serves as the global prototype for negative traffic entities. The cosine similarity function is used. Temperature coefficient; This represents the total number of traffic entity categories. A contrastive loss algorithm amplifies the weights of positive sample pairs, thus narrowing the physical perceived distance between similar traffic entities in the feature space.

[0076] S21. Structural Alignment (Structural-Level Semantic Graph Constraints): The cloud-based autonomous driving server maintains a global traffic semantic topology graph based on a global prototype. Its elements represent higher-order relationships between physical entities. During local training on the vehicle-mounted device, the road condition feature matrix within a batch is set as follows: Calculate the structural alignment loss using KL divergence constraints The similarity distribution within a batch of local features is forced to approximate the corresponding substructure of the global graph, and the corresponding structure alignment loss is... The calculation formula is as follows:

[0077] (3);

[0078] in, This is the road condition feature matrix within the batch. It is a global traffic semantic topology graph that includes high-order relationships between traffic entities. The Softmax function is used to transform perceptual similarity into a probability distribution. Temperature coefficient; Tag for real road conditions. For the first batch The vehicle-cloud collaborative representation vector corresponding to each road condition sample; This is a set of labels for traffic entities within a batch.

[0079] Specifically, in autonomous driving engineering practice, structural constraints endow perception models with strong causal interpretability at the physical level: by forcing the similarity distribution within batches of local heterogeneous in-vehicle networks to align with the global traffic physical map, the interference of cross-regional "environmental confounders" such as blizzards and backlighting on feature extraction can be effectively eliminated. This ensures that autonomous vehicles can maintain a rigid and invariant physical topological distance between entities such as "pedestrians" and "zebra crossings" in their feature space under any extreme weather conditions, fundamentally correcting system-level perception distortions caused by spurious correlations such as overfitting to environmental backgrounds.

[0080] Furthermore, the implementation of orthogonal decorrelation regularization in step S3 includes:

[0081] S30. Let the normalized vehicle-cloud collaborative representation matrix be... Calculate its feature correlation matrix .in is the dimension of the vehicle-cloud collaborative representation vector.

[0082] S31. Perform orthogonal decorrelation regularization: Force the feature correlation matrix by minimizing the Frobenius norm. Approaching unit formation Calculate the orthogonal decorrelation regularization loss. The calculation formula is as follows:

[0083] (4);

[0084] in, The first element in the feature correlation matrix Line 1 The elements of the column represent the first... The representation dimension and the first The degree of linear correlation between the representation dimensions; The first element in the feature correlation matrix Line 1 The element of the column, i.e., the first The autocorrelation coefficients of each representation dimension. This operation constructs a "safety net" mechanism for long-tail extreme road conditions within the vehicle-cloud collaborative framework.

[0085] Orthogonal decorrelation regularization forces the feature correlation matrix to approximate the identity matrix by minimizing the norm, compelling different perception channels (i.e., each feature dimension) of the vehicle sensor to extract completely independent physical characteristics (e.g., the first dimension extracts the physical contour, and the second dimension extracts the material reflectivity). This maximizes the information entropy of the multi-dimensional perception channels, effectively avoiding the "representation collapse" caused by the vehicle perception network over-catering to high-frequency common road conditions (such as clear straight lanes). It ensures that when the system encounters unknown long-tail road conditions (corner cases) such as irregular obstacles and scattered cargo, it can still retain mutually orthogonal and effective feature extraction channels, providing multi-dimensional judgment basis for subsequent emergency avoidance.

[0086] Furthermore, the specific implementation of step S4 includes:

[0087] S40. Calculate the local joint sensing loss: Configure balancing weight coefficients for each loss term. Total loss The expression is as follows:

[0088] (5);

[0089] in, For balancing weighting coefficients; For privately perceived cross-entropy loss; Point-level alignment loss; This is for structural alignment loss; The loss is the orthogonal decorrelation regularization loss.

[0090] Using this weighted joint sensing loss Perform backpropagation of network gradients to update the underlying hardware-extracted parameters of the vehicle-mounted heterogeneous perception network in a closed loop. Private real-time control parameters And the parameters of the car-cloud public projection head This completes a single evolution of the local autonomous driving perception model.

[0091] S41. Local Prototype Generation: According to the traffic entity category, the autonomous vehicle performs mean aggregation on the common feature vectors of the locally extracted entities of that category to generate a local prototype of the corresponding entity category, and uploads it to the cloud autonomous driving server.

[0092] S42. Global Prototype Aggregation and Graph Reconstruction: The cloud-based autonomous driving server performs weighted aggregation on the collected local prototypes of various autonomous vehicles, updating the global prototype set for the next round. Based on the new global prototype set, the inter-class similarity is recalculated, and the global traffic semantic topology is reconstructed. It is then broadcast to a subset of active autonomous vehicles for the next round of training.

[0093] Furthermore, in S4, the cloud-based autonomous driving server weights and aggregates the local scene generator parameters uploaded by each autonomous vehicle to obtain a global scene generator. The global scene generator is used to generate global pseudo-road condition features. The global pseudo-road condition features are then input into the local model and the initial global model of each autonomous vehicle, respectively. Global knowledge distillation is performed on the initial global model to minimize the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

[0094] The global traffic semantic topology graph is constrained as follows: when facing environmental differences with strong visual interference, a divergence algorithm is used to force the similarity distribution within local feature batches to approximate the global traffic semantic topology graph. This ensures that the high-order relative geometric distances between "pedestrians" and "zebra crossings" extracted from the underlying pixel difference data collected under both sunny and snowy weather conditions remain rigidly aligned with the global consensus. This invention maintains the correct geometric relative distances between traffic entities through the global traffic semantic topology graph, ensuring the reliable fusion of global driving experience. Orthogonal decorrelation regularization forces each feature dimension to perform its function with outward-expanding constraints, avoiding representation degradation in heterogeneous networks that try to conform to the global mean.

[0095] The model framework involved in this invention is called FedUniRep, which follows the design philosophy of representation space decoupling and geometric structure reshaping. It isolates global noise through dual-head decoupling and a gentle startup mechanism; at the feature fusion level, it not only performs conventional point-level prototype alignment but also introduces structure-level graph alignment to reshape high-dimensional topology; after feature reshaping, orthogonal decorrelation regularization is introduced to constrain feature axes to be independent, maximizing the information entropy of the feature space. This avoids topological distortion and low-dimensional subspace redundancy problems caused by heterogeneous model architectures.

[0096] Furthermore, the present invention also discloses an autonomous driving federated learning system based on perception decoupling and manifold alignment, comprising:

[0097] A cloud-based autonomous driving server is used to maintain a global prototype set and a global traffic semantic topology graph. ,in For the number of traffic physical entity categories, Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between the models is calculated; the global prototype set and the global traffic semantic topology map G are distributed to each autonomous vehicle; the local prototypes of each category uploaded by each autonomous vehicle are received, the cosine similarity between the local model of each autonomous vehicle and the initial global model is calculated as the dynamic aggregation weight, and the local prototypes are weighted and aggregated to update the global prototype set; the global traffic semantic topology map G is reconstructed according to the updated global prototype set and broadcast to the next communication round.

[0098] Multiple autonomous vehicles, each equipped with onboard physical sensors, a heterogeneous onboard computing platform, and a drive-by-wire chassis. The heterogeneous onboard computing platform is configured with a heterogeneous perception network and a dual-head perception decoupling architecture. The heterogeneous perception network is used to extract basic road condition features from local real-world road condition data collected by the onboard physical sensors. The dual-head perception decoupling architecture includes a private control head, a common feature projection head, a manifold geometry alignment module, an orthogonal decorrelation regularization module, and a local joint update module. The private control head is used to map basic road condition features to a traffic physical entity label space, output local real-time road condition analysis results, and drive the drive-by-wire chassis to execute emergency avoidance commands. The parameters of this private control head do not participate in the vehicle-cloud federation. Interaction; The common feature projection head is used to map basic road condition features into a unified-dimensional vehicle-cloud collaborative representation vector. The unified dimension is consistent across all heterogeneous vehicle computing platforms, serving as a common feature space for vehicle-cloud federation interaction; The manifold geometry alignment module is used to perform two-layer alignment of the vehicle-cloud collaborative representation vector during local training: point-level alignment and structure-level alignment; Point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototype of other categories as the negative anchor, employing contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push away the distance from the negative anchor; Structure-level alignment takes the vehicle-cloud collaborative representation matrix within the local batch, calculates the similarity distribution within the batch, and uses KL divergence to constrain this distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G; The orthogonal decorrelation regularization module is used to calculate the feature correlation matrix of the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension orthogonal to each other. The local joint update module is used to obtain the total loss by weighted summation of local private perception loss, point-level alignment loss, structural alignment loss and orthogonal decorrelation loss, backpropagation to update the parameters of local heterogeneous perception network and dual-head architecture, and upload the updated local prototypes of each category to the cloud autonomous driving server.

[0099] Furthermore, the heterogeneous in-vehicle computing platform includes low-computing-power in-vehicle chips and high-computing-power flagship in-vehicle chips; the network scale of the private control head is strongly tied to the underlying computing power and drive-by-wire chassis control latency requirements of the corresponding autonomous vehicle, and its forward computation and parameter updates are completed locally in the autonomous vehicle, with processing latency meeting the millisecond-level response requirements of in-vehicle emergency avoidance; the common feature projection head compresses the high-dimensional non-standardized features output by each heterogeneous perception network into a standard vector of the same low-dimensional dimension, enabling autonomous vehicles with different computing power to access the same vehicle-cloud collaborative representation space.

[0100] Furthermore, the autonomous vehicle is also equipped with a vehicle-side scene generator. The vehicle-side scene generator uses the previous round of global perception model issued by the cloud autonomous driving server as the teacher model and the local current model as the student model. It maximizes the KL divergence between the two in the classification prediction output to update its own parameters. This allows the vehicle-side scene generator to output long-tailed difficult pseudo-road condition data that makes the local model and the global model prediction most divergent. The pseudo-road condition data is then used to perform local knowledge distillation closed-loop optimization on the local heterogeneous perception network.

[0101] The cloud-based autonomous driving server is also used to receive updated scene generator parameters uploaded by each autonomous vehicle, weighted aggregate to obtain a global scene generator, use the global scene generator to generate global pseudo-road condition features, input the global pseudo-road condition features into the local model and the initial global model of each autonomous vehicle respectively, and perform global knowledge distillation fine-tuning on the initial global model in a way that minimizes the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

[0102] This invention abandons simple point-level feature alignment and innovatively introduces a global traffic semantic topology graph. By completely eliminating meteorological confounding factors through structural manifold physical topology alignment, the accuracy of the model in identifying core traffic participants (pedestrians, non-standard vehicles) is reduced by more than 40% compared to traditional federated alignment methods when deployed across regions (such as directly migrating from sunny climate zones to snow-covered areas), achieving true environmental decoupling.

[0103] This invention constructs a multi-dimensional orthogonal perception channel, greatly expanding the safety response redundancy in long-tail extreme scenarios (corner cases): Thanks to the maximization of feature space information entropy brought about by the orthogonal decorrelation mechanism, when facing unknown irregular obstacles with a frequency of less than 1% on open roads, the system can still ensure that at least three independent orthogonal physical feature dimensions (such as material, contour, and thermal radiation) are active. This enables the system to achieve a leapfrog improvement in recall for long-tail road conditions, providing the vehicle-mounted AEB system with hundreds of milliseconds of golden redundancy response time, completely breaking the technical curse of "instant failure when the unseen is seen" in traditional methods.

[0104] This invention's heterogeneous architecture and algorithm mechanism generate a non-obvious synergy: traditional federated feature alignment is mathematically a centripetal squeezing force, which is precisely the cause of multi-dimensional perception channel failure (representation collapse) in heterogeneous networks for autonomous driving. This invention cleverly utilizes "global traffic semantic topology graph constraints" as a bridge to construct an ingenious physical balance system. Macroscopically, the graph maintains the correct geometric relative distances between traffic entities to ensure global experience fusion; microscopically, orthogonal decorrelation, with its outward-expanding force, forces each feature dimension to perform its specific function. These three mechanisms produce an unexpected synergistic effect at the underlying logic—"both macroscopically aligned and microscopically independent"—successfully resolving the engineering paradox in autonomous driving that requires both high generalization and protection against rare obstacles.

[0105] The present invention adopts the above technical solution and has the following beneficial effects compared with the prior art:

[0106] (1) By adapting the perception decoupling architecture and gentle start-up mechanism to heterogeneous computing power, physical isolation between local real-time risk avoidance of autonomous vehicles and global experience sharing between vehicle and cloud is realized. The private control head is strongly bound to the computing power of the vehicle's underlying drive-by-wire chassis. Its forward inference latency meets the millisecond-level emergency braking requirements, ensuring that the vehicle can still independently complete the autonomous risk avoidance response to immediate dangerous situations such as pedestrian crossing and obstacle emergence in any scenario of federated communication interruption or cloud aggregation delay, without relying on the availability of the vehicle-cloud link; the public feature projection head compresses the heterogeneous features output by vehicles with different computing power levels into the same dimensional space, enabling older models and high-end models to exchange knowledge at the same semantic level, breaking through the vehicle fleet collaboration barrier caused by hardware generation differences.

[0107] (2) By employing a manifold geometry alignment strategy, the autonomous driving perception model is endowed with causal interpretability and environmental decoupling capabilities across regions and weather conditions. The global traffic semantic topology map G defines high-order rigid geometric relationships between traffic physical entities such as "pedestrian-zebra crossing", "vehicle-lane line", and "obstacle-drivable area". In scenarios with strong visual interference such as blizzards, heavy fog, and backlighting, the map constrains the similarity distribution within local feature batches to remain consistent with the global consensus, forcing the vehicle perception network to remove false correlations from environmental confounding factors such as meteorological particles and lighting angles. This results in a reduction of more than 40% in the accuracy of identifying core traffic participants (pedestrians, irregular vehicles, and construction cones) when vehicles migrate from sunny urban areas to snow-covered areas compared to traditional federated alignment methods, fundamentally eliminating "perceptual geometry drift" and "semantic confusion" caused by overfitting to regional environmental backgrounds.

[0108] (3) Through orthogonal decorrelation regularization, a multi-dimensional independent safety perception redundancy channel is constructed for the autonomous driving system under unknown long-tail extreme road conditions. This mechanism forces each dimension of the vehicle-cloud collaborative representation to focus on the physical contours of obstacles, material texture reflectivity, surface thermal radiation characteristics, and other independent physical properties, maximizing the information entropy of the feature space. When the vehicle encounters irregularly shaped obstacles with a frequency of less than 1% on open roads (such as the wreckage of an overturned irregularly shaped engineering vehicle or scattered irregular cargo), since each perception dimension has been forcibly decoupled and is not redundant, the system can still activate at least three independent feature extraction channels for parallel judgment, rather than the "representation collapse" failure of the multi-dimensional perception channels caused by the representation dominance of high-frequency conventional road conditions (such as standard straight lanes) as in traditional methods. This mechanism provides the onboard automatic emergency braking system (AEB) with an additional golden response time of 200–500 milliseconds, significantly improving the survival rate and risk avoidance success rate of the autonomous driving system in "unseen" scenarios.

[0109] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A federated learning method for autonomous driving based on perception decoupling and manifold alignment, applied to a vehicle-to-everything (V2X) system consisting of a cloud-based autonomous driving server and multiple autonomous vehicles equipped with heterogeneous onboard computing platforms and onboard physical sensors, characterized in that... The method includes the following steps: S1, each autonomous vehicle collects local real-world road condition data through onboard physical sensors, inputs it into the local heterogeneous feature extractor of the locally deployed heterogeneous perception network to extract basic road condition features; the basic road condition features are diverted to the dual-head perception decoupled architecture, and the local real-time road condition analysis results are obtained through the private control head to execute the emergency avoidance command of the drive-by-wire chassis; the basic road condition features are mapped into a unified-dimensional vehicle-cloud collaborative representation vector through the public feature projection head; S2, a cloud-based autonomous driving server maintains a global prototype set and a global traffic semantic topology map. ,in The number of traffic physical entity categories, and the global traffic semantic topology graph. elements Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between them; the cloud-based autonomous driving server distributes the global prototype set and the global traffic semantic topology map G to each autonomous vehicle; during local training, the autonomous vehicles perform two-level alignment on the vehicle-cloud collaborative representation vector z: point-level alignment and structure-level alignment. Point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototypes of other categories as the negative anchor, and uses contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push away the distance from the negative anchor; structure-level alignment takes the vehicle-cloud collaborative representation matrix within the local batch. Calculate the intra-batch similarity distribution and use KL divergence to constrain the intra-batch similarity distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G; S3, calculate the feature correlation matrix from the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension mutually orthogonal. S4, the autonomous vehicle obtains the total loss by weighted summing of local private perception loss, point-level alignment loss, structural alignment loss and orthogonal decorrelation loss, and then backpropagates to update the parameters of the local heterogeneous perception network and dual-head architecture. Autonomous vehicles upload updated local prototypes of each category to the cloud-based autonomous driving server. The cloud-based autonomous driving server calculates the cosine similarity between the local model of each autonomous vehicle and the initial global model in order to allocate dynamic aggregation weights. The local prototypes are weighted and aggregated to obtain the updated global prototype set, and the global traffic semantic topology graph G is reconstructed and broadcast to the next communication round.

2. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 1, characterized in that, The heterogeneous in-vehicle computing platform includes low-computing-power in-vehicle chips and high-computing-power flagship in-vehicle chips; the forward computation and parameter updates of the private control head are all completed locally in the autonomous vehicle; the network scale of the private control head is strongly tied to the underlying computing power and drive-by-wire chassis control latency requirements of the corresponding autonomous vehicle, and the processing latency meets the millisecond-level response requirements of in-vehicle emergency avoidance; the common feature projection head compresses the high-dimensional non-standardized features output by each heterogeneous perception network into a standard vector of the same low-dimensional dimension, enabling autonomous vehicles with different computing power to access the same vehicle-cloud collaborative representation space.

3. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 1, characterized in that, In S1, the private control head calculates the local prediction, and the corresponding expression is: ； in, For private control headers; Basic road condition characteristics; For the first Trainable network parameters corresponding to the private control head of an autonomous vehicle; Calculate the private-perception cross-entropy loss The expression is: （1）； in, To handle the batch size of data, This represents the total number of traffic entity categories. Tag the actual road conditions; For the first The corresponding road condition sample output by the private control head is the [number]th [item]. Predicted probability values ​​for traffic-like entities; This is an indicator function that takes the value 1 when the condition within the parentheses is true and 0 when the condition is false.

4. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 3, characterized in that, Point-level alignment loss The calculation formula is as follows: （2）； in, This is the vehicle-cloud collaboration representation vector. and These are the global prototypes for positive and negative traffic entities, respectively. Temperature coefficient; This represents the total number of traffic entity categories; Calculating structural alignment loss using KL divergence constraints The calculation formula is as follows: （3）； in, This is the road condition feature matrix within the batch. It is a global traffic semantic topology graph that includes high-order relationships between traffic entities. For the Softmax function; Temperature coefficient; Tag the actual road conditions; For the first batch The vehicle-cloud collaborative representation vector corresponding to each road condition sample; This is a set of labels for traffic entities within a batch.

5. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 1, characterized in that, Calculate the orthogonal decorrelation regularization loss The calculation formula is as follows: （4）； in, The dimension of the vehicle-cloud collaborative representation vector; The first element in the feature correlation matrix Line number The elements of the column, that is, the elements representing the first... The representation dimension and the first The degree of linear correlation between the representation dimensions; The first element in the feature correlation matrix Line number The element of the column, i.e., the first The autocorrelation coefficients of each representation dimension.

6. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 1, characterized in that, Total loss The expression is as follows: （5）； in, For balancing weighting coefficients; For privately perceived cross-entropy loss; Point-level alignment loss; This is for structural alignment loss; The loss is the orthogonal decorrelation regularization loss.

7. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 1, characterized in that, In S1, the autonomous vehicle builds a local scene generator locally, using the previous global model from the cloud-based autonomous driving server as the teacher model and the current local model as the student model. It performs adversarial training by maximizing the KL divergence between the two models in the classification prediction output to update the local scene generator. This allows the local scene generator to simulate long-tailed difficult pseudo-road condition data that causes the greatest discrepancy between the local model and the global model's prediction. The autonomous vehicle then uses the long-tailed difficult pseudo-road condition data to perform local knowledge distillation on the local heterogeneous perception network, thus optimizing the local heterogeneous feature extractor of the local heterogeneous perception network in a closed loop.

8. The autonomous driving federated learning method based on perception decoupling and manifold alignment according to claim 7, characterized in that, In S4, the cloud-based autonomous driving server weights and aggregates the local scene generator parameters uploaded by each autonomous vehicle to obtain a global scene generator. The global scene generator is used to generate global pseudo-road condition features. The global pseudo-road condition features are then input into the local model and the initial global model of each autonomous vehicle, respectively. Global knowledge distillation is performed on the initial global model to minimize the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

9. An autonomous driving federated learning system based on perception decoupling and manifold alignment, as described in any one of claims 1 to 8, characterized in that the system... include: A cloud-based autonomous driving server is used to maintain a global prototype set and a global traffic semantic topology graph. ,in For the number of traffic physical entity categories, Represents the global prototype of the m-th class With the nth global prototype The cosine similarity between the models is calculated; the global prototype set and the global traffic semantic topology map G are distributed to each autonomous vehicle; the local prototypes of each category uploaded by each autonomous vehicle are received, the cosine similarity between the local model of each autonomous vehicle and the initial global model is calculated as the dynamic aggregation weight, and the local prototypes are weighted and aggregated to update the global prototype set; the global traffic semantic topology map G is reconstructed according to the updated global prototype set and broadcast to the next communication round. Multiple autonomous vehicles, each equipped with onboard physical sensors, a heterogeneous onboard computing platform, and a drive-by-wire chassis. The heterogeneous onboard computing platform is configured with a heterogeneous perception network and a dual-head perception decoupling architecture. The heterogeneous perception network is used to extract basic road condition features from local real-world road condition data collected by the onboard physical sensors. The dual-head perception decoupling architecture includes a private control head, a public feature projection head, a manifold geometry alignment module, an orthogonal decorrelation regularization module, and a local joint update module. The private control head is used to map basic road condition features to the traffic physical entity label space, output local real-time road condition analysis results, and drive the drive-by-wire chassis to execute emergency avoidance commands. The parameters of the private control head do not participate in the vehicle-cloud federated interaction. The common feature projection head maps basic road condition features to a unified-dimensional vehicle-cloud collaborative representation vector. This unified dimension remains consistent across all heterogeneous vehicle computing platforms, serving as a common feature space for vehicle-cloud federated interaction. The manifold geometry alignment module performs two-layer alignment—point-level and structure-level—on the vehicle-cloud collaborative representation vector during local training. Point-level alignment uses the global prototype of the corresponding category as the positive anchor and the global prototypes of other categories as the negative anchor, employing contrastive loss to bring the vehicle-cloud collaborative representation vector closer to the positive anchor and push it further away from the negative anchor. Structure-level alignment takes the local batch-level vehicle-cloud collaborative representation matrix, calculates the batch-level similarity distribution, and uses KL divergence to constrain the batch-level similarity distribution to approximate the global similarity distribution of the corresponding category rows in the global traffic semantic topology graph G. The orthogonal decorrelation regularization module calculates the feature correlation matrix on the normalized vehicle-cloud collaborative representation matrix. d is the representation dimension; minimize the Frobenius norm. The forced feature correlation matrix C is approximated to the identity matrix I, making each representation dimension orthogonal to each other. The local joint update module is used to obtain the total loss by weighted summation of local private perception loss, point-level alignment loss, structural alignment loss and orthogonal decorrelation loss, and backpropagates to update the parameters of the local heterogeneous perception network and dual-head architecture. The updated local prototypes of each category are then uploaded to the cloud autonomous driving server.

10. The autonomous driving federated learning system based on perception decoupling and manifold alignment according to claim 9, characterized in that, The autonomous vehicle is also equipped with a vehicle-side scene generator. The vehicle-side scene generator uses the previous round of global perception model issued by the cloud autonomous driving server as the teacher model and the local current model as the student model. It maximizes the KL divergence between the two in the classification prediction output to update its own parameters. The vehicle-side scene generator outputs long-tailed difficult pseudo-road condition data that makes the local model and the global model prediction most divergent. It also uses the pseudo-road condition data to perform local knowledge distillation closed-loop optimization on the local heterogeneous perception network. The cloud-based autonomous driving server is also used to receive updated scene generator parameters uploaded by each autonomous vehicle, weighted aggregate to obtain a global scene generator, use the global scene generator to generate global pseudo-road condition features, input the global pseudo-road condition features into the local model and the initial global model of each autonomous vehicle respectively, and perform global knowledge distillation fine-tuning on the initial global model in a way that minimizes the KL divergence of the two classification predictions. After completion, the global traffic semantic topology graph G is reconstructed and broadcast.

Images