Vehicle-mounted large model-oriented edge deployment and federated mask learning method
By using edge deployment and federated mask learning methods, the weight matrix of the large vehicle model is stacked into a fourth-order tensor and decomposed by Tucker. Combined with binary masking and cloud aggregation mechanisms, the problems of poor computing power adaptation and high communication overhead in the deployment of large models on vehicle intelligent terminals are solved, and efficient cross-vehicle perception knowledge sharing and decision quality improvement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN FIRST NORMAL UNIV
- Filing Date
- 2026-04-30
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies make it difficult to efficiently deploy large models on in-vehicle intelligent terminals, and traditional federated learning faces challenges in communication and computing power adaptation, failing to meet the real-time, safety, and cost constraints of intelligent driving.
By employing edge deployment and federated mask learning methods, the weight matrix of the pre-trained vehicle large model is stacked into a fourth-order parameter tensor along the layer and module dimensions. The low-rank core tensor is extracted through Tucker decomposition as a holographic base, and binary masks are used to achieve selective activation and sparse communication. Combined with a cloud-based logical superposition and aggregation mechanism, cross-vehicle perception knowledge is solidified.
It improves model compression efficiency, reduces communication overhead, is compatible with heterogeneous computing terminals, realizes the sharing and transfer of driving perception knowledge across vehicle nodes, and improves the quality of perception decision-making.
Smart Images
Figure CN122114086A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving technology, and specifically discloses an edge deployment and federated mask learning method for large vehicle models. Background Technology
[0002] With the rapid development of intelligent driving technology, the demand for high-precision environmental perception and decision-making capabilities provided by large-scale models in vehicle intelligent terminals is becoming increasingly urgent. Large-scale models, with their superior multimodal understanding, cross-scenario reasoning, and complex environment modeling capabilities, have become a core technology driving vehicle intelligent driving systems. However, the parameter scale of large-scale models typically ranges from billions to trillions, placing demands on computing power, storage, and energy consumption far exceeding the capabilities of vehicle intelligent terminals. Simultaneously, driving perception data from different vehicle nodes involves user driving privacy and automaker trade secrets. Data privacy, vehicle communication bandwidth limitations, and low-latency constraints make it difficult to meet the real-time, safety, and cost constraints of autonomous driving scenarios by simply relying on cloud-based inference. Therefore, how to efficiently deploy large-scale models to resource-constrained vehicle intelligent terminals and support privacy-preserving collaborative training to improve perception and decision-making quality has become a key technical issue driving the development of vehicle intelligent driving systems.
[0003] Against this backdrop, model compression technology is considered an important approach to enabling the deployment of large models on vehicle edges. Existing model compression methods mainly include knowledge distillation, network pruning, parameter quantization, and low-rank decomposition. However, when compressing large models, these methods often process the weight matrices of each layer in isolation, ignoring the structured semantic relationships across layers and modules in deep neural networks, thus limiting compression efficiency.
[0004] In edge collaborative training, federated learning, as a privacy-preserving distributed machine learning paradigm, enables in-vehicle intelligent terminals distributed across various vehicle nodes to collaboratively optimize the global perception model while protecting privacy. However, traditional federated learning requires edge nodes to upload all gradients or model updates, and the communication volume is linearly related to the model parameter size. For large models with billions of parameters, the communication overhead is unbearable in the limited bandwidth of in-vehicle communication in intelligent driving environments. Furthermore, traditional federated learning uses arithmetic averaging for parameter aggregation, requiring all participating nodes to share the same model structure, which contradicts the reality of highly heterogeneous computing power among in-vehicle intelligent terminals of different vehicle models.
[0005] In terms of task adaptation, different vehicle nodes face different driving scenarios. For example, highway scenarios emphasize high-speed following and lane-changing decisions, urban road scenarios emphasize complex intersection perception and pedestrian detection, and low-speed parking scenarios emphasize obstacle perception and precise localization in narrow spaces. The distribution of driving perception data varies significantly among vehicle nodes. Existing efficient parameter fine-tuning methods still require learning and storing independent adaptation parameters for each node's perception task. When in-vehicle intelligent terminals face diverse driving scenario requirements, the cumulative storage overhead of adaptation parameters remains considerable. Furthermore, there is a lack of explicit correlation modeling between adaptation parameters of different vehicle nodes, making it difficult to achieve effective transfer and sharing of cross-vehicle perception knowledge.
[0006] This invention provides an edge deployment and federated mask learning method for large vehicle-mounted models to address the aforementioned problems. Summary of the Invention
[0007] The purpose of this invention is to solve the problems of poor computing power adaptation and lack of efficient collaborative training when deploying large models on heterogeneous vehicle-mounted intelligent terminals.
[0008] To achieve the above objectives, the basic solution of this invention provides an edge deployment and federated mask learning method for large vehicle-mounted models, comprising the following steps: Step A1: Obtain the pre-trained vehicle-mounted large model for the driving perception and decision-making task, and stack the weight matrices of the multi-head self-attention modules and feedforward network modules of all its layers along the layer dimension and module dimension respectively to construct a fourth-order parameter tensor. Step A2: Perform Tucker decomposition on the fourth-order parameter tensor, extract the low-rank core tensor and modal factor matrices, and jointly construct the holographic base. The cloud server broadcasts the holographic base to each vehicle intelligent terminal. Each vehicle intelligent terminal performs model reconstruction based on the holographic base to deploy an executable perception and decision model. Step A3: Each vehicle-mounted intelligent terminal introduces a mask fraction with the same dimension as the core tensor, and obtains a binary mask through Heaviside step function mapping. The core tensor is selectively activated by performing Hadamard product between the binary mask and the core tensor. Step A4: Based on the locally stored driving perception dataset, each vehicle-mounted intelligent terminal uses a pass-through estimator to calculate the gradient of the mask score, and controls the mask sparsity based on a composite regularization strategy. The mask score is then optimized and trained to obtain the trained binary mask. The positions with a value of 1 in the trained binary mask are extracted to form an activation index set, which is then uploaded to the cloud server. Step A5: The cloud server performs logical superposition and aggregation on the binary masks corresponding to the activation index set to obtain a global mask. Based on the activation frequency threshold, the global mask is path-filtered. The filtered global mask and the built-in global verification data are used to solidify cross-vehicle perception knowledge into the core tensor through constrained optimization to update the holographic base and broadcast it to each in-vehicle intelligent terminal. Repeat steps A3 to A5 until the perception and decision model converges. Each in-vehicle intelligent terminal then executes the driving perception and decision task through the converged perception and decision model.
[0009] Furthermore, the elements of the fourth-order parametric tensor in the i-th row and j-th column of the weight matrix of the m-th submodule in the l-th layer... Defined as: ; In the formula, For submodule indexing, , The total number of submodules is represented by the submodules in the multi-head self-attention module and the feedforward network module. The multi-head self-attention module includes a query projection matrix, a key projection matrix, a value projection matrix, and an output projection matrix. The feedforward network module includes an up projection matrix and a down projection matrix. This is the weight matrix of the m-th submodule in the l-th layer; The four modal dimensions of a fourth-order parametric tensor are defined as follows: ; In the formula, As a layer dimension, From a module perspective, For input dimensions, Where L is the output dimension and L is the total number of layers. This represents the total number of submodules. To hide dimensions in the model, This is the intermediate dimension of the feedforward network; To address the differences in output dimensions between different submodules, a zero-padding strategy is used to align the output dimensions of all submodules to the specified values. .
[0010] Furthermore, in step A2, a higher-order orthogonal iterative algorithm is used to solve the Tucker decomposition. The specific steps are as follows: For fourth-order parameter tensors Matrix expansion along the nth mode , before The nth mode factor matrix is initialized by the first left singular vector. ; During the iterative optimization phase, for each mode, the auxiliary tensor is calculated and its mode-n expansion matrix is subjected to singular value decomposition. The first few values are then selected. The left singular vector is updated. Modal factor matrix Continue until all factor matrices converge; Core Tensor Calculated by the following formula: ; In the formula, This represents the modal-n product operation. T denotes matrix transpose; The selection of the multilinear rank employs an adaptive strategy based on energy retention rate; for the nth mode, the rank with the minimum energy retention rate is selected. This ensures that the cumulative energy ratio is not less than the preset energy retention threshold.
[0011] Furthermore, in step A2, each in-vehicle intelligent terminal reconstructs the executable perception decision model through one of the following two methods: Complete reconstruction method: When the video memory of the vehicle intelligent terminal is sufficient to accommodate the parameters of complete reconstruction, the complete parameter tensor is directly calculated and converted back to the weight matrix format of each layer for perception inference. Delayed reconstruction method: When the video memory of the in-vehicle intelligent terminal is limited, the weights of each layer are calculated on demand during the perception inference process. When calculating the output of the l-th layer and the m-th sub-module, the corresponding slices of the core tensor are extracted. The weights of this layer are calculated in real time using the following formula. : ; In the formula, This is the third modal factor matrix. This is the fourth mode factor matrix.
[0012] Furthermore, in step A3, the binary mask The following is obtained by applying a Heaviside step function to the mask fraction: ; ; In the formula, For vehicle-mounted intelligent terminals The mask score, It is the Heaviside step function.
[0013] Furthermore, in step A4, the composite regularization strategy is to establish a composite regularization function. The composite regularization function includes a sparsity penalty term and a polarization regularization term, and its expression is as follows: ; In the formula, All are regularization coefficients. , For sparse penalty terms, This is a polarization canonical term; Introducing target sparsity During training, dynamically adjust according to the following formula. : ; In the formula, For the first The current sparsity rate in the next iteration. For adjustment coefficients, is the initial regularization coefficient.
[0014] Furthermore, in step A4, the uplink communication volume for a single vehicle-mounted intelligent terminal to upload the activation index set to the cloud server is: ; In the formula, For vehicle-mounted intelligent terminals Uplink traffic. For vehicle-mounted intelligent terminals mask sparsity, To activate the number of indexes, For the core tensor The total number of elements, This is the floor function.
[0015] Furthermore, in step A5, the global mask is obtained by performing a logical OR operation on the binary mask corresponding to the activation index set through the cloud server, and the neuron importance frequency index is introduced to count the frequency of activation at each position. Given an activation frequency threshold, the global mask with a neuron importance frequency index greater than the activation frequency threshold is selected as the global mask.
[0016] Furthermore, in the cross-vehicle perception knowledge solidification process of step A5, only the filtered global mask is updated. Activated parameter location: ; In the formula, For the i-th element of the updated core tensor, The first core tensor before the update Each element represents a globally fine-tuned learning rate. To verify the loss regarding the first The gradient of each element, It is the i-th element of the filtered global mask.
[0017] The principle and effect of this basic scheme are as follows: 1. Compared with existing technologies, this invention maps all parameters of the pre-trained vehicle-mounted large model to a fourth-order tensor space along the layer and module dimensions. It extracts the low-rank core tensor containing cross-layer semantic relationships through Tucker decomposition as a holographic base, breaking the paradigm of traditional methods that process weight matrices in isolation layer by layer. This effectively captures the structured semantic relationships across layers and modules, improves model compression efficiency, and enables large models to be deployed on resource-constrained vehicle-mounted intelligent terminals.
[0018] 2. Compared with the prior art, the present invention achieves sublinear communication through the high sparsity of binary masks. The vehicle-mounted intelligent terminal only needs to upload the set of activation indexes instead of the complete parameters. The communication volume is proportional to the sparsity of the mask rather than linearly related to the model size, which reduces the communication overhead of federated learning. Moreover, the mask mechanism is naturally compatible with the heterogeneous computing power of vehicle-mounted intelligent terminals deployed by each vehicle node.
[0019] 3. Compared with the prior art, the present invention selects different neural pathways using different masks on a unified holographic base, enabling vehicle nodes to share and transfer cross-node driving perception knowledge in a unified representation space. Through cloud-based logical superposition and aggregation and cross-vehicle perception knowledge solidification mechanism, the driving perception knowledge scattered in various vehicle intelligent terminals is persisted to the holographic base, realizing the global propagation of cross-vehicle node driving perception knowledge and improving the driving perception decision quality of each vehicle node. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart of the edge deployment and federated mask learning method for large vehicle models proposed in this application is shown. Detailed Implementation
[0022] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0023] Edge deployment and federated mask learning methods for large-scale vehicle models, implementing, for example Figure 1 The steps shown are as follows: Step A1: Obtain the pre-trained vehicle-mounted large model for the driving perception and decision-making task, and stack the weight matrices of the multi-head self-attention modules and feedforward network modules of all its layers along the layer dimension and module dimension respectively to construct a fourth-order parameter tensor.
[0024] This method is applied to a distributed in-vehicle intelligent driving system, which is used in distributed intelligent driving scenarios and consists of a cloud server and multiple in-vehicle intelligent terminals deployed at various vehicle nodes. In this embodiment, vehicle nodes include traffic participants equipped with in-vehicle intelligent terminals, such as passenger vehicles, freight vehicles, and buses equipped with autonomous driving systems, used to perform environmental understanding and driving decision reasoning based on multimodal perception data collected during driving.
[0025] The cloud server possesses ample computing power and storage resources for performing model compression, global aggregation, and knowledge solidification operations. Each in-vehicle intelligent terminal is a resource-constrained terminal device, including, but not limited to, in this embodiment, an in-vehicle domain controller equipped with an embedded GPU and an intelligent driving computing platform with an edge computing chip. Each in-vehicle intelligent terminal locally stores the driving perception dataset corresponding to the vehicle node. The driving perception dataset includes multimodal perception data collected by in-vehicle sensors, such as LiDAR point cloud data, camera image data, and millimeter-wave radar data, as well as corresponding scene annotation data, such as object detection annotations, semantic segmentation annotations, lane line recognition annotations, and driving decision labels. Specifically, LiDAR point cloud data supports object detection annotation, semantic segmentation annotation, and lane line recognition annotation. Camera image data supports object detection annotation, semantic segmentation annotation, lane line recognition annotation, and driving decision annotation. Millimeter-wave radar data supports object detection annotation. During the federated learning process, the original driving perception dataset of each in-vehicle intelligent terminal remains locally and is not uploaded to protect user driving privacy and automotive company trade secrets.
[0026] In this embodiment, a pre-trained in-vehicle large model based on the Transformer architecture is used as the target model for the driving perception and decision-making task. This pre-trained in-vehicle large model contains L stacked Transformer layers. For the l-th layer, its core parameters include a multi-head self-attention module and a feedforward network module. The multi-head self-attention module includes a query projection matrix: Key projection matrix: Value projection matrix: Output projection matrix: The feedforward network module includes an up-projection matrix: Lower projection matrix: .
[0027] in, This represents the total number of layers in the Transformer. For layer index, , To hide dimensions in the model, This represents the intermediate dimension of the feedforward network.
[0028] Specifically, the weight matrices of all L layers are stacked along the layer and module dimensions to construct a fourth-order parameter tensor. , The elements of the fourth-order parametric tensor in the i-th row and j-th column of the weight matrix of the m-th submodule in the l-th layer. Defined as: ; In the formula, For submodule indexing, , This represents the total number of submodules. The sub-modules are the various sub-modules within the multi-head self-attention module and the feedforward network module, including Q, K, V, O, up, and down. Let be the weight matrix of the m-th submodule in the l-th layer.
[0029] The four modal dimensions of a fourth-order parametric tensor are defined as follows: ; In the formula, As a layer dimension, From a module perspective, For input dimensions, For output dimensions.
[0030] In this embodiment, to address the differences in output dimensions between different sub-modules, a zero-padding strategy is used to align the output dimensions of all sub-modules to the specified values. .
[0031] Step A2: Perform Tucker decomposition on the fourth-order parameter tensor, extract the low-rank core tensor and modal factor matrices, and jointly construct the holographic base. The cloud server broadcasts the holographic base to each in-vehicle intelligent terminal. Each in-vehicle intelligent terminal performs model reconstruction based on the holographic base to deploy an executable perception and decision model.
[0032] Specifically, the fourth-order parameter tensor is decomposed using Tucker decomposition, and its decomposition form is as follows: ; In the formula, For the core tensor, , , , , Let be the multilinear ranks corresponding to the first, second, third, and fourth modes, respectively, corresponding to the layer dimension, module dimension, input feature dimension, and output feature dimension, satisfying . , , Indicates the first Modal factor matrix, , Representing modes Product operation, It indicates a low-rank approximation relationship.
[0033] Modal The product is defined as: ; In the formula, For the core tensor The corresponding first mode index ..., nth modal index ..., 4th Modal Index elements, The nth modality factor matrix is the jth row of the nth modality factor matrix. The elements of the column.
[0034] The scalar expansion of the Tucker decomposition is as follows: ; In the formula, The index of the first mode in the fourth-order parameter tensor Second mode index 3rd Mode Index 4th Modal Index elements, The index corresponding to the first mode in the core tensor Second mode index 3rd Mode Index 4th Modal Index elements, The first modality factor matrix is the first... Line number Column elements, The second modality factor matrix is the first... Line number Column elements, The third modality factor matrix is the first... Line number Column elements, The fourth modality factor matrix is the first... Line number The elements of the column.
[0035] In this embodiment, the extraction of the holographic base is formalized as a least-squares approximation problem with orthogonal constraints: ; In the formula, For Frobenius norm, for An identity matrix of order 1.
[0036] In this embodiment, a high-order orthogonal iterative algorithm is used for solving the problem. The specific steps are as follows: First, for the fourth-order parameter tensor Along the first Matrix expansion of modes , , before The nth mode factor matrix is initialized by the first left singular vector. ; Then, the iterative optimization phase begins. For each mode, the auxiliary tensor is calculated and its mode is adjusted. Expand the matrix and perform singular value decomposition, then take the first few elements. Update factor matrix of left singular vectors ; When all factor matrices converge, the core tensor Calculated by the following formula: ; In the formula, T represents the matrix transpose.
[0037] Regarding the selection of the multilinear rank, this embodiment employs an adaptive strategy based on energy retention rate. For the ... Modality, choose the smallest one. This ensures that the cumulative energy ratio satisfies: ; In the formula, For the expanded matrix The A singular value, For the preset energy retention threshold, .
[0038] After completing the Tucker decomposition, the core tensor is... With four factor matrices jointly build a holographic base : .
[0039] The holographic base serves as the unified initialization state for all in-vehicle intelligent terminals, broadcast once by the cloud server to each terminal. At the edge, each in-vehicle intelligent terminal can reconstruct an executable perception and decision-making model in two ways: (1) Complete reconstruction method: If the video memory of the vehicle intelligent terminal is sufficient to accommodate the parameters of complete reconstruction, the complete parameter tensor is directly calculated according to the following formula. And convert it back to the weight matrix format of each layer for inference: .
[0040] (2) Delayed Reconstruction Method: For in-vehicle intelligent terminals with limited video memory, a delayed reconstruction strategy is adopted, calculating the weights of each layer on demand during the perception and reasoning process. When calculating the weights of the first layer... Layer, First When outputting from each submodule, first extract the corresponding slice of the core tensor. , Then, the weights of this layer are calculated in real time using the following formula. : .
[0041] In this example, for in-vehicle intelligent terminals equipped with large amounts of video memory, such as in-vehicle domain controllers with 16GB of video memory, a full reconfiguration approach is adopted. For in-vehicle intelligent terminals with smaller amounts of video memory, such as embedded edge computing devices with 4GB to 8GB of video memory, a delayed reconfiguration approach is adopted to enable the deployment of perception and decision-making models under resource-constrained conditions.
[0042] Step A3: Each vehicle-mounted intelligent terminal introduces a mask fraction with the same dimension as the core tensor, and obtains a binary mask through Heaviside step function mapping. The core tensor is selectively activated by performing Hadamard product between the binary mask and the core tensor.
[0043] Specifically, each in-vehicle intelligent terminal introduces a binary mask while freezing the values of the holographic base. Achieve task adaptation. ,in The total number of elements in the core tensor. , For in-vehicle intelligent terminal indexes.
[0044] In this embodiment, the onboard intelligent terminals of different vehicle nodes are designed for different driving scenarios: vehicle nodes traveling on highways focus on perception tasks such as high-speed following, lane keeping, and lane changing decisions; vehicle nodes traveling on urban roads focus on decision-making tasks such as target detection at complex intersections and perception of pedestrians and non-motorized vehicles; and vehicle nodes in low-speed parking scenarios focus on perception tasks such as obstacle perception and precise positioning in narrow spaces. Each onboard intelligent terminal learns its own binary mask and selectively activates neural pathways in the core tensor related to its own driving scenario, thereby achieving differentiated driving scenario perception adaptation on a unified holographic base.
[0045] The task adaptation model is defined as follows: ; In the formula, For Hadamard product, The input data is multimodal sensing data. The forward propagation function is adapted to the task model.
[0046] Due to binary mask Due to the discrete nature of the problem, direct optimization faces two fundamental challenges: the exponential growth of the feasible solution space and the failure of gradient optimization. To overcome these problems, a continuous relaxation strategy is introduced in this embodiment.
[0047] Introduce a real-valued tensor with the same dimension as the core tensor. As a mask score Each of its elements It encodes the degree of activation at the corresponding position. Binary mask. The following is obtained by applying a Heaviside step function to the mask fraction: ; ; In the formula, For vehicle-mounted intelligent terminals The mask score, For the first The mask score at each position, It is the Heaviside step function.
[0048] Under this parameterization, the in-vehicle intelligent terminal The optimization objective is transformed into learning the optimal mask score. : .
[0049] In this embodiment, mask fraction Use offset Gaussian initialization: ; In the formula, To initialize the mean, Standard deviation, To satisfy the mean is Variance is The Gaussian distribution. In this embodiment, Set to a positive value to ensure that most positions are activated initially.
[0050] Step A4: Based on the locally stored driving perception dataset, each vehicle-mounted intelligent terminal uses a pass-through estimator to calculate the gradient of the mask score, while freezing the holographic base. It then controls the mask sparsity based on a composite regularization strategy, optimizes the mask score to obtain a trained binary mask, and extracts the position indices of the value 1 in the trained binary mask to form an activation index set. The activation index set is then uploaded to the cloud server.
[0051] Due to the Heaviside function exist Discontinuity at, Since the derivative is always zero, directly applying the chain rule would completely block the gradient signal. Therefore, this embodiment uses a pass-through estimator technique to solve this problem.
[0052] The pass-through estimator specifies different functions for forward and backward propagation in the computation graph: Forward propagation strictly uses the Heaviside function to generate a binary mask: .
[0053] During backpropagation, The derivative is approximately equal to the identity mapping: ; In the formula, It is an identity matrix.
[0054] Based on the pass-through estimator, the mask score is derived. The gradient expression. Let... For the effective parameter tensor after masking, apply the chain rule: ; In the formula, The effective parameter tensor after masking. Let be the gradient of the loss function with respect to the mask score.
[0055] Substituting the pass-through estimator into the approximate value, we obtain the gradient of the mask fraction: ; In the formula, For the gradient based on the pass-through estimator, This is the gradient of the loss function with respect to the effective parameters.
[0056] Furthermore, the composite regularization strategy involves establishing a composite regularization function. This guides the masked scores to evolve towards sparser and binarized values. ; In the formula, All are regularization coefficients. , For sparse penalty terms, This is the polarization regularization term.
[0057] The sparsity penalty term is defined as: ; In the formula, For the Sigmoid function, .
[0058] This sparse penalty term encourages the mask scores to shift toward negative values by imposing an L1 norm constraint on the mask scores after Sigmoid mapping, thereby reducing the proportion of activated positions.
[0059] The polarization regularization term is defined as: .
[0060] Polarization regularization term with respect to mask fraction The gradient is: ; In the formula, The derivative of the Sigmoid function. .
[0061] The function of this polarization regularization term is to cause the mask score to move away from the decision boundary, i.e. The nearby region causes the mask scores to accelerate towards a specific 0 or 1 direction during training.
[0062] Furthermore, a target sparsity rate is introduced. , And dynamically adjust during training. : ; In the formula, For the first The current sparsity rate in the next iteration. , For adjustment coefficients, , The initial regularization coefficients are... for Norm. When the current sparsity exceeds the target sparsity, Adaptively increase to enhance the sparsity penalty.
[0063] Integrating task loss and regularization term, in-vehicle intelligent terminal The complete training objective function is: ; In the formula, For vehicle-mounted intelligent terminals The number of samples of driving perception data stored locally. For the loss function related to the perception task, Multimodal sensing data collected by vehicle-mounted sensors. For the corresponding scene-labeled data, we obtain the i-th input data and its label, and use the input data as training samples. For mission losses, Let be the overall objective function.
[0064] In this embodiment, the learning rate for the mask score uses a cosine annealing strategy: ; In the formula, For the first The learning rate for the next iteration. and These are the maximum and minimum learning rates, respectively. This represents the total number of iterations during local training.
[0065] In this embodiment, Set as , Set as Target sparsity Set it to between 0.05 and 0.20. Set as to between.
[0066] In traditional federated learning, edge nodes need to upload the full gradients or model updates, resulting in a communication overhead of [missing information]. Where N is the total number of model parameters. In this embodiment, sublinear communication is achieved by utilizing the high sparsity of the mask.
[0067] Specifically, for in-vehicle intelligent terminals Binary mask after training Define its active index set as: ; In the formula, For edge nodes The set of active indexes For the core tensor The total number of elements.
[0068] In this embodiment, the vehicle-mounted intelligent terminal Upload only the collection of active indexes Instead of a complete mask vector. Let the mask sparsity be... The uplink communication volume for a single vehicle-mounted intelligent terminal to upload the activation index set to the cloud server is: ; In the formula, For vehicle-mounted intelligent terminals Uplink traffic. For vehicle-mounted intelligent terminals mask sparsity, To activate the number of indexes, This is the floor function.
[0069] In this embodiment, when the target sparsity When the value is 0.10 (i.e. only 10% of the core tensor elements are activated), compared to traditional federated learning which uploads all model parameters, the uplink communication volume of this invention is significantly reduced, and the communication volume is independent of the model parameter size, but only related to the mask sparsity and the core tensor size.
[0070] Step A5: The cloud server performs logical superposition and aggregation on the binary masks corresponding to the activation index set to obtain a global mask. Based on the activation frequency threshold, the global mask is filtered for paths. Based on the filtered global mask and the built-in global verification data, cross-vehicle perception knowledge is solidified into the core tensor through constrained optimization to update the holographic base and broadcast to each in-vehicle intelligent terminal.
[0071] Repeat steps A3 to A5 for iterative training until the perception and decision model converges. Each in-vehicle intelligent terminal then executes the driving perception and decision task through the converged perception and decision model.
[0072] Specifically, the cloud server receives a collection of active in-vehicle intelligent terminals participating in the aggregation. After activating the index set, , For all in-vehicle intelligent terminals, a global mask will be applied. Defined as the logical union of all local binary masks: .
[0073] In the formula, This is the local binary mask uploaded by the k-th in-vehicle intelligent terminal.
[0074] The global mask is obtained by performing a logical OR operation on the masks of each in-vehicle intelligent terminal: ; In the formula, The first global mask One element, It is the i-th element of the local binary mask.
[0075] Furthermore, simply stacking logic might lead to an overly dense global mask, weakening generalization ability. Therefore, a neuron importance frequency index is introduced. , Count the frequency of activation at each location: ; In the formula, This is the frequency vector of neuron importance. For the first Activation frequency at each location, It is a set of non-negative integers. .
[0076] Given activation frequency threshold , The i-th element of the filtered global mask Defined as: ; In the formula, For indicator functions, It is a set of positive integers.
[0077] In this embodiment, the activation frequency threshold Control the following trade-offs: when It is equivalent to pure logical superposition, retaining all paths activated by any vehicle node's perception task, and has the densest global mask. when Only paths that are jointly activated by all vehicle node perception tasks are retained, resulting in the sparsest global mask. when It is necessary to strike a balance between the specificity and shared nature of driving scenarios at each vehicle node.
[0078] After multiple rounds of federated iterations, the filtered global mask The cross-vehicle perception knowledge contained within needs to be solidified into the core tensor. This enables the persistence and dissemination of perceived knowledge across vehicle nodes.
[0079] Assume the cloud server has a small amount of global verification data. The global validation data is a general perception test set covering various driving scenarios. Cross-vehicle perception knowledge is solidified through the following constrained optimizations: ; In the formula, For the updated core tensor, For global validation dataset, To verify the size of the dataset.
[0080] In this embodiment, only the filtered global mask is updated. Activated parameter location: ; In the formula, The first core tensor before the update One element, For the i-th element of the updated core tensor, To fine-tune the learning rate globally, To verify the loss regarding the first The gradient of each element, The loss function is defined on the validation set.
[0081] In this embodiment, the aforementioned constrained optimization strategy ensures that the positions of parameters not activated by the global mask remain unchanged during the cross-vehicle perception knowledge solidification process, thereby preserving the existing semantic structure in the holographic base. During the cross-vehicle perception knowledge solidification process, the factor matrices U(1), U(2), U(3), and U(4) remain unchanged, while only the core tensor is affected. The activated location is updated.
[0082] After the cross-vehicle perception knowledge is solidified, the cloud server broadcasts the updated holographic base to each in-vehicle intelligent terminal. Each in-vehicle intelligent terminal then re-executes steps A3 to A5 based on the updated holographic base to conduct the next round of federated training.
[0083] Repeat steps A3 to S5 for iterative training until the perception and decision-making model converges. After convergence, the onboard intelligent terminal of each vehicle node performs the following driving perception and decision-making tasks through the converged perception and decision-making model: While the vehicle is in motion, the onboard intelligent terminal receives multimodal perception data collected by the onboard sensors in real time. The perception decision model performs environmental understanding and scenario reasoning on the input multimodal perception data and outputs driving decision results, including but not limited to target detection results, semantic segmentation results, lane line recognition results, driving path planning suggestions, collision risk warnings, etc.
[0084] In this embodiment, the driving perception datasets of each in-vehicle intelligent terminal are always stored locally and are not uploaded. During the federated iteration process, each in-vehicle intelligent terminal only uploads a sparse set of activation indices, resulting in an extremely low risk of information leakage and effectively protecting the driving privacy of users on each vehicle node and the trade secrets of the automaker. Furthermore, in-vehicle intelligent terminals on different vehicle nodes can have masks with different sparsity rates, naturally compatible with heterogeneous computing power, and do not require all vehicle nodes to deploy terminal devices with the same configuration.
[0085] This invention maps all parameters of a pre-trained large vehicle model to a fourth-order tensor space along the layer and module dimensions. By extracting a low-rank core tensor containing cross-layer semantic relationships through Tucker decomposition, it serves as a holographic base, breaking the paradigm of traditional methods that process weight matrices in isolation layer by layer. This effectively captures structured semantic relationships across layers and modules, improves model compression efficiency, and enables large models to be deployed on resource-constrained vehicle intelligent terminals.
[0086] This invention achieves sublinear communication through the high sparsity of binary masks. The vehicle-mounted intelligent terminal only needs to upload the set of activation indexes instead of the complete parameters. The communication volume is proportional to the sparsity of the mask rather than linearly related to the model size, thus reducing the communication overhead of federated learning.
[0087] This invention enables the sharing and transfer of cross-vehicle perception knowledge between vehicle nodes by selecting different neural pathways using different masks on a unified holographic base. Through cloud-based logical overlay aggregation and cross-vehicle perception knowledge solidification mechanism, the driving perception knowledge scattered across various vehicle intelligent terminals is persistently transferred to the holographic base, realizing the global propagation of cross-vehicle perception knowledge and improving the perception decision-making quality of each vehicle node.
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. An edge deployment and federated mask learning method for large-scale vehicle models, characterized in that, Includes the following steps: Step A1: Obtain the pre-trained vehicle-mounted large model for the driving perception and decision-making task, and stack the weight matrices of the multi-head self-attention modules and feedforward network modules of all its layers along the layer dimension and module dimension respectively to construct a fourth-order parameter tensor. Step A2: Perform Tucker decomposition on the fourth-order parameter tensor, extract the low-rank core tensor and modal factor matrices, and jointly construct the holographic base. The cloud server broadcasts the holographic base to each vehicle intelligent terminal. Each vehicle intelligent terminal performs model reconstruction based on the holographic base to deploy an executable perception and decision model. Step A3: Each vehicle-mounted intelligent terminal introduces a mask fraction with the same dimension as the core tensor, and obtains a binary mask through Heaviside step function mapping. The core tensor is selectively activated by performing Hadamard product between the binary mask and the core tensor. Step A4: Based on the locally stored driving perception dataset, each vehicle-mounted intelligent terminal uses a pass-through estimator to calculate the gradient of the mask score, and controls the mask sparsity based on a composite regularization strategy. The mask score is then optimized and trained to obtain the trained binary mask. The positions with a value of 1 in the trained binary mask are extracted to form an activation index set, which is then uploaded to the cloud server. Step A5: The cloud server performs logical superposition and aggregation on the binary masks corresponding to the activation index set to obtain a global mask. Based on the activation frequency threshold, the global mask is path-filtered. The filtered global mask and the built-in global verification data are used to solidify cross-vehicle perception knowledge into the core tensor through constrained optimization to update the holographic base and broadcast it to each in-vehicle intelligent terminal. Repeat steps A3 to A5 until the perception and decision model converges. Each in-vehicle intelligent terminal then executes the driving perception and decision task through the converged perception and decision model.
2. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 1, characterized in that, Elements of the fourth-order parametric tensor in the i-th row and j-th column of the weight matrix of the m-th submodule in layer l. Defined as: ； In the formula, For submodule indexing, , The total number of submodules is represented by the submodules in the multi-head self-attention module and the feedforward network module. The multi-head self-attention module includes a query projection matrix, a key projection matrix, a value projection matrix, and an output projection matrix. The feedforward network module includes an up projection matrix and a down projection matrix. This is the weight matrix of the m-th submodule in the l-th layer; The four modal dimensions of a fourth-order parametric tensor are defined as follows: ； In the formula, As a layer dimension, From a module perspective, For input dimensions, Where L is the output dimension and L is the total number of layers. This represents the total number of submodules. To hide dimensions in the model, This is the intermediate dimension of the feedforward network; To address the differences in output dimensions between different submodules, a zero-padding strategy is used to align the output dimensions of all submodules to the specified values. .
3. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 2, characterized in that, In step A2, a higher-order orthogonal iterative algorithm is used to solve the Tucker decomposition. The specific steps are as follows: For fourth-order parameter tensors Matrix expansion along the nth mode , before The nth mode factor matrix is initialized by the first left singular vector. ; During the iterative optimization phase, for each mode, the auxiliary tensor is calculated and its mode-n expansion matrix is subjected to singular value decomposition. The first few values are then selected. The left singular vector is updated. Modal factor matrix Continue until all factor matrices converge; Core Tensor Calculated by the following formula: ； In the formula, This represents the modal-n product operation. T denotes matrix transpose; The selection of the multilinear rank employs an adaptive strategy based on energy retention rate; for the nth mode, the rank with the minimum energy retention rate is selected. This ensures that the cumulative energy ratio is not less than the preset energy retention threshold.
4. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 1, characterized in that, In step A2, each in-vehicle intelligent terminal reconstructs the executable perception decision model through one of the following two methods: Complete reconstruction method: When the video memory of the vehicle intelligent terminal is sufficient to accommodate the parameters of complete reconstruction, the complete parameter tensor is directly calculated and converted back to the weight matrix format of each layer for perception inference. Delayed reconstruction method: When the video memory of the in-vehicle intelligent terminal is limited, the weights of each layer are calculated on demand during the perception inference process. When calculating the output of the l-th layer and the m-th sub-module, the corresponding slices of the core tensor are extracted. The weights of this layer are calculated in real time using the following formula. : ； In the formula, This is the third mode factor matrix. This is the fourth mode factor matrix.
5. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 1, characterized in that, In step A3, the binary mask The following is obtained by applying a Heaviside step function to the mask fraction: ； ； In the formula, For vehicle-mounted intelligent terminals The mask score, It is the Heaviside step function.
6. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 5, characterized in that, In step A4, the composite regularization strategy is to establish a composite regularization function. The composite regularization function includes a sparsity penalty term and a polarization regularization term, and its expression is as follows: ； In the formula, All are regularization coefficients. , For sparse penalty terms, This is a polarization canonical term; Introducing target sparsity During training, dynamically adjust according to the following formula. : ； In the formula, For the first The current sparsity rate in the next iteration. For adjustment coefficients, is the initial regularization coefficient.
7. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 1, characterized in that, In step A4, the uplink communication volume for a single vehicle-mounted intelligent terminal to upload the activation index set to the cloud server is: ； In the formula, For vehicle-mounted intelligent terminals Uplink traffic. For vehicle-mounted intelligent terminals mask sparsity, To activate the number of indexes, For the core tensor The total number of elements, This is the floor function.
8. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 1, characterized in that, In step A5, the global mask is obtained by performing a logical OR operation on the binary mask corresponding to the activation index set through the cloud server, and the neuron importance frequency index is introduced to count the frequency of activation at each position. Given an activation frequency threshold, the global mask with a neuron importance frequency index greater than the activation frequency threshold is selected as the global mask.
9. The edge deployment and federated mask learning method for large vehicle-mounted models according to claim 8, characterized in that, In the cross-vehicle perception knowledge solidification process in step A5, only the filtered global mask is updated. Activated parameter location: ； In the formula, For the i-th element of the updated core tensor, The first core tensor before the update One element, To fine-tune the learning rate globally, To verify the loss regarding the first The gradient of each element, It is the i-th element of the filtered global mask.