A collaborative training method for heterogeneous edge large models based on nested manifold alignment

Abstract

This invention relates to the fields of edge computing and distributed machine learning technology, specifically disclosing a heterogeneous edge large-scale model collaborative training method based on nested manifold alignment. By introducing a nested heterogeneous compatible architecture guided by singular value decomposition, an ordered spectral basis is constructed in the parameter space, enabling in-vehicle intelligent terminals with different computing power to train different spectral slices of the large model separately. This achieves elastic scaling for inference across the entire spectrum of in-vehicle intelligent terminals with a single global collaborative training. Secondly, to address gradient conflicts in multiple tasks, an energy and density-aware parameter-semantic dual-stream spectral aggregation mechanism is designed. This mechanism uses orthogonal spectral projection to decouple complex mixed gradients into independent feature components, eliminating destructive interference at the microscopic level. Simultaneously, it uses singular value energy density as a physical prior to robustly calibrate long-tail noise, breaking down the dual barriers of heterogeneity and task conflicts in in-vehicle intelligent terminals.

Description

Technical Field

[0001] This application relates to the fields of edge computing and distributed machine learning technology, and specifically discloses a collaborative training method for heterogeneous edge large models based on nested manifold alignment. Background Technology

[0002] With the rapid development of large model technology, the demand for large model inference capabilities in edge terminals is becoming increasingly urgent. However, the parameter scale of large models usually ranges from billions to trillions, far exceeding the computing power, storage, and energy consumption limits of ordinary edge terminals. Moreover, there are huge differences in computing power among different edge terminals, and traditional federated learning combined with low-rank adaptation schemes cannot adapt to this heterogeneous computing power distribution.

[0003] In existing technologies, federated learning enables a large model to be trained collaboratively by a massive number of edge terminals under the premise of privacy protection by preserving local data and aggregating parameters across devices. Meanwhile, with the application of Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), the barrier to fine-tuning large models on the edge terminal side has been further lowered. Its paradigm of freezing the pre-trained backbone and updating only a small number of adaptation parameters has achieved significant results in many perception tasks.

[0004] Taking heterogeneous in-vehicle intelligent terminals as an example, when this paradigm is extended to the multi-task scenario of heterogeneous in-vehicle intelligent terminals with vehicle-road cooperative perception, its core premise quickly becomes invalid.

[0005] On the one hand, regarding the heterogeneity of in-vehicle intelligent terminals, the large parameter scale of the models stretches the capabilities of these terminals from an approximately continuous distribution into a discrete fragment: high-end in-vehicle domain controllers can run models with billions of parameters under extreme optimization, mid-range embedded edge computing devices often require aggressive quantization to barely load them, while a large number of low-end in-vehicle intelligent terminals cannot accommodate any complete model structure. This capability fragmentation means that the traditional federated learning approach of using a unified model structure and simple parameter averaging, even with PEFT mechanisms like LoRA, still lacks a systematic solution for allocating matching and adaptable capacity to in-vehicle intelligent terminals with different computing power on the same pre-trained backbone.

[0006] On the other hand, regarding the heterogeneity of perception tasks, even if large models possess general capabilities, the specialized needs of different vehicle nodes in driving perception scenarios remain highly diverse: highway scenarios emphasize high-speed following and lane-changing decisions, urban road scenarios emphasize complex intersection perception and pedestrian detection, and low-speed parking scenarios rely on obstacle perception and precise localization in narrow spaces. These perception tasks differ fundamentally in input distribution, label space, loss function, and even decision objectives, making it increasingly difficult to effectively represent them using a single global model or a single task head.

[0007] More importantly, in a shared adaptation subspace represented by LoRA, the gradient conflict problem between multiple sensing tasks is further amplified in federated scenarios. Specifically, the update directions of different sensing tasks in the shared LoRA subspace are often inconsistent, and even nearly opposite in some spectral directions: when the cloud server performs a simple weighted average of the LoRA increments from different sensing tasks and different in-vehicle intelligent terminals, the updates in these directions will significantly cancel each other out, resulting in extremely slow effective progress of the global large model along these high-value spectral directions.

[0008] Meanwhile, limited onboard computing resources restrict each onboard intelligent terminal to training only within a low-dimensional subspace of finite rank. This makes deciding which spectral directions to share and which to decouple in a multi-sensory task collaboration a core design problem. Therefore, the goal of a unified model structure plus global single-point convergence, which is relied upon by traditional federated learning, is no longer feasible.

[0009] Meanwhile, the aforementioned problems are also reflected in typical edge computing scenarios such as industrial internet equipment inspection and smart security video analysis.

[0010] This invention provides a method for collaborative training of heterogeneous edge large models based on nested manifold alignment, in order to solve the above-mentioned problems. Summary of the Invention

[0011] The purpose of this invention is to solve the problems of poor computing power adaptation and lack of efficient collaborative training when deploying large models on heterogeneous edge terminals.

[0012] To achieve the above objectives, the basic solution of this invention provides a method for collaborative training of heterogeneous edge large models based on nested manifold alignment, comprising the following steps: Step A1: Perform low-rank incremental modeling on the pre-trained weights of the pre-trained large model for vehicle-road cooperative perception tasks carried by each vehicle-mounted intelligent terminal, and perform singular value decomposition on the pre-trained weight matrix. Step A2: Establish a super adapter on the cloud server to be compatible with different rank configurations, calculate the optimal rank of each vehicle intelligent terminal, and obtain sub-adaptors from the super adapter according to the optimal rank of the vehicle intelligent terminal. After constructing a global perception semantic prototype set based on the perception category division of the vehicle-road cooperative perception task, the paired sub-adaptors and the global perception semantic prototype set are sent as data packets to the corresponding vehicle intelligent terminals. Step A3: Each in-vehicle intelligent terminal updates the sub-adapter parameters and local perception semantic prototype set based on the issued sub-adapter and global perception semantic prototype set with the locally stored driving perception dataset, calculates the parameter increment of the sub-adapter, and uploads the parameter increment, local perception semantic prototype set, and the number of local driving perception samples corresponding to each perception category to the cloud server. Step A4: The cloud server receives the uploaded information, completes the aggregation of parameter increments, and aggregates the local perception semantic prototype set based on the number of local driving perception samples corresponding to each perception category, and updates the super adapter and the global perception semantic prototype set. Step A5: Based on the updated super adapter and the global perception semantic prototype set, the cloud server iterates from step A2 to step A4 until the large model converges. Each vehicle-mounted intelligent terminal then performs vehicle-road cooperative perception tasks through the converged large model.

[0013] Furthermore, in step A1, the following trainable low-rank increment matrix is ​​constructed. : ; In the formula, The dimension is The input rank matrix, , Indicates the input feature dimension. The dimension is The output rank matrix, , Indicates the output feature dimension. Indicates rank. .

[0014] Furthermore, in step A2, the parameters of the constructed super adapter are as follows: ; ; In the formula, This represents the global input rank matrix, corresponding to the mapping from the input feature dimension to the rank dimension. This represents the global output rank matrix, corresponding to the mapping from the rank dimension to the output feature dimension. This indicates the maximum rank supported by the highest-performance in-vehicle intelligent terminal.

[0015] Furthermore, in step A2, the optimal rank of each vehicle-mounted intelligent terminal is calculated on a cloud server based on the singular value energy retention rate and latency budget constraints.

[0016] Furthermore, the singular value energy retention rate is measured by rank. The degree of preservation of the principal energy of the pre-trained weights by the low-rank fit subspace is calculated.

[0017] Furthermore, the delay budget constraint includes computation delay constraint and communication delay constraint.

[0018] Furthermore, in step A2, a nested truncation operator is also constructed. Based on the super adapter, the cloud server generates sub-adaptors with different ranks but nested column spaces for in-vehicle intelligent terminals with different computing power through the constructed nested truncation operator. as follows: In the t-th round of training, for the t-th Each in-vehicle intelligent terminal, based on its supported optimal rank , Extract the global input rank matrix for the current round. The former The column yields the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal. The expression is as follows: ; In the formula, , This indicates extracting the global input rank matrix. The former The input truncation matrix obtained after the column; Extract the global output rank matrix of the current round forward The output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal is obtained by row. The expression is as follows: ; In the formula, , This indicates extracting the global output rank matrix. forward The output truncation matrix obtained from the row; For the first The vehicle-mounted intelligent terminal and the first Each in-vehicle intelligent terminal, if its order satisfies Then the column space of the corresponding adapter satisfies the following nesting relationship: ; In the formula, This represents the input low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. This represents the output low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. Let represent the input low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal. Let represent the output low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal.

[0019] Furthermore, in step A3, the calculation formulas for the parameter increments of the input low-rank sub-adapter and the output low-rank sub-adapter are as follows: ; ; In the formula, This represents the initial value of the input low-rank sub-adapter matrix of the k-th vehicle-mounted intelligent terminal in the t-th round of training. Let represent the initial value of the output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal during the t-th round of training. Let represent the updated value of the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal after the t-th round of training. This represents the updated value of the low-rank sub-adapter matrix output by the k-th in-vehicle intelligent terminal after the t-th round of training. , It is the rank assigned to the k-th in-vehicle intelligent terminal in this round, and is much smaller than .

[0020] Furthermore, the input low-rank sub-adapter update value and output low-rank sub-adaptor update value The update is performed using a stochastic gradient descent optimizer, as shown in the following expression: ; ; In the formula, The learning rate is represented by the learning rate, which is locally scheduled by the in-vehicle intelligent terminal. Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix of the input of the k-th vehicle-mounted intelligent terminal in the t-th round of training. gradient, Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix output by the k-th vehicle-mounted intelligent terminal in the t-th round of training. The gradient.

[0021] Furthermore, in step A4, after the cloud server receives the information uploaded by each vehicle-mounted smart terminal, it unifies the parameter increment dimension through the zero-fill projection operator and performs weighted aggregation of parameter increments in combination with the coverage vector to update the super adapter.

[0022] The principle and effect of this basic scheme are as follows: 1. Compared with the prior art, the present invention introduces a nested heterogeneous compatible architecture guided by singular value decomposition, and constructs a set of ordered spectral bases in the parameter space, so that edge devices with different computing power, such as vehicle-mounted intelligent terminals, can train different spectral slices of large models respectively, realizing a global collaborative training and elastic scaling of inference by vehicle-mounted intelligent terminals of the entire spectrum everywhere.

[0023] Secondly, to address gradient conflicts in multi-task applications, an energy- and density-aware parameter-semantic dual-stream spectral aggregation mechanism was designed. This mechanism utilizes orthogonal spectral projection to decouple complex mixed gradients into independent feature components, eliminating destructive interference at the microscopic level. Simultaneously, it uses singular value energy density as a physical prior to robustly calibrate long-tail noise, breaking down the dual barriers of heterogeneity and task conflicts in in-vehicle intelligent terminals.

[0024] 2. Compared with existing technologies, this invention, when applied to the collaborative training of large-scale pre-trained models by multiple types of vehicle-mounted intelligent terminals, simultaneously satisfies constraints on computational latency, communication latency, and task semantic consistency. This method achieves collaborative alignment of large models in the parameter space and feature space by constructing a nested low-rank adapter parametric manifold, a dynamic rank configuration model based on singular value energy, a prototype-guided semantic manifold anchoring mechanism, a spectral energy-driven rank orchestration protocol, and a cover-compensated heterogeneous subspace fusion strategy. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of the heterogeneous edge large model collaborative training method based on nested manifold alignment proposed in this application is shown. Detailed Implementation

[0027] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0028] Based on the method of heterogeneous edge large model collaborative training based on nested manifold alignment, this embodiment takes the vehicle-road cooperative perception scenario of the Internet of Vehicles as an example to describe the method of the present invention in detail. The method of the present invention is also applicable to other heterogeneous edge large model collaborative training scenarios such as industrial Internet and smart security.

[0029] Specifically, implementation for example Figure 1 As shown, it includes the following steps: Step A1: Perform low-rank incremental modeling on the pre-trained weights of the pre-trained large model for vehicle-road cooperative perception tasks carried by each vehicle-mounted intelligent terminal, and perform singular value decomposition on the pre-trained weight matrix.

[0030] This method is applied to a distributed vehicle-road cooperative perception system, which consists of a cloud server and multiple in-vehicle intelligent terminals deployed at various vehicle nodes. In this embodiment, vehicle nodes include traffic participants equipped with in-vehicle intelligent terminals, such as passenger vehicles, freight vehicles, and buses equipped with autonomous driving systems. Each vehicle node is equipped with an in-vehicle intelligent terminal for environmental understanding and driving decision reasoning based on multimodal perception data collected during driving.

[0031] The cloud server possesses ample computing power and storage resources to perform super adapter management, global aggregation, and global perception semantic prototype maintenance operations. Each in-vehicle intelligent terminal is a resource-constrained terminal device, including, but not limited to, an in-vehicle domain controller equipped with an embedded GPU and an intelligent driving computing platform equipped with an edge computing chip. Each in-vehicle intelligent terminal locally stores the driving perception dataset corresponding to the vehicle node. The driving perception dataset includes multimodal environmental data collected by in-vehicle sensors, such as LiDAR point cloud data, camera image data, and millimeter-wave radar data, as well as corresponding scene annotation data, such as object detection annotations, semantic segmentation annotations, lane line recognition annotations, and driving decision annotations. Specifically, LiDAR point cloud data supports object detection annotation, semantic segmentation annotation, and lane line recognition annotation. Camera image data supports object detection annotation, semantic segmentation annotation, lane line recognition annotation, and driving decision annotation. Millimeter-wave radar data supports object detection annotation. During the federated learning process, the driving perception dataset of each in-vehicle intelligent terminal remains locally and is not uploaded to protect user driving privacy and automotive company trade secrets.

[0032] In this embodiment, low-rank incremental modeling is performed on the pre-trained weights in the Transformer layer of the pre-trained large model mounted on each in-vehicle intelligent terminal. Specifically, low-rank incremental modeling of the pre-trained weights is performed using LoRA, and the pre-trained weight matrix is ​​frozen during subsequent iterative training, with only the low-rank increment matrix being updated. By constructing a trainable low-rank increment matrix, trainable parameters are concentrated on the rank matrices corresponding to the input and output dimensions, respectively, thereby achieving fine-grained adjustment of the behavior of the large model without modifying the pre-trained backbone weights.

[0033] Specifically, the following trainable low-rank increment matrix is ​​constructed. : ; In the formula, The dimension is The input rank matrix, , Indicates the input feature dimension. The dimension is The output rank matrix, , Indicates the output feature dimension. Indicates rank. .

[0034] For any Transformer layer, given an input feature vector , The forward computation process of this Transformer layer is represented as follows: ; In the formula, To output the feature vector, , This represents the pre-trained weight matrix of the Transformer layer, and the pre-trained weight matrix remains unchanged during the calculation process. .

[0035] Subsequently, singular value decomposition is performed on the pre-trained weight matrix to construct a nested fitting subspace with geometric structure constraints in the parameter space. This obtains the energy distribution of the pre-trained weight matrix in different spectral directions, allowing the subsequently constructed low-rank fitting subspace to be aligned with the important spectral directions of the pre-trained model. This significantly reduces the number of trainable parameters while maintaining the expressive power of the pre-trained large model.

[0036] In this embodiment, the singular value decomposition of the pre-trained weight matrix is ​​performed according to the following formula: ; In the formula, and All are orthogonal matrices. It is a left singular vector matrix. It is a right singular vector matrix. , , This represents the singular value diagonal matrix arranged in descending order. And satisfy , , This represents the i-th singular value. Represents a left singular vector. This represents the transpose of a right singular vector. This represents the component of the pre-trained weight in the i-th spectral direction.

[0037] Furthermore, a sequence of singular values ​​arranged in descending order is constructed. ,in, The primary singular value represents the direction of maximum energy.

[0038] Step A2: Establish a super adapter on the cloud server to be compatible with different rank configurations, calculate the optimal rank of each vehicle intelligent terminal, and obtain sub-adaptors from the super adapter according to the optimal rank of each vehicle intelligent terminal. After constructing a global perception semantic prototype set based on the perception category division of the vehicle-road cooperative perception task, the paired sub-adaptors and the global perception semantic prototype set are sent as data packets to the corresponding vehicle intelligent terminals.

[0039] Specifically, in this embodiment, the cloud server refers to a central parameter server with high-performance computing and communication capabilities, and the vehicle-mounted intelligent terminal refers to heterogeneous terminal computing devices distributed and deployed on various vehicle nodes, including but not limited to vehicle domain controllers, intelligent driving computing platforms, vehicle gateways equipped with edge computing chips, and roadside perception devices. The vehicle-mounted intelligent terminal and the cloud server participate in the distributed collaborative fine-tuning of large-scale pre-trained models in the field of vehicle-road cooperative perception.

[0040] A super adapter is built on a cloud server to ensure compatibility with in-vehicle smart terminals of different rank configurations. In this embodiment, the parameters of the super adapter are as follows: ; ; In the formula, This represents the global input rank matrix, corresponding to the mapping from the input feature dimension to the rank dimension. This represents the global output rank matrix, corresponding to the mapping from the rank dimension to the output feature dimension. This indicates the maximum rank supported by the highest-performance in-vehicle intelligent terminal.

[0041] In this embodiment, the optimal rank of each vehicle-mounted intelligent terminal is calculated on a cloud server based on the singular value energy retention rate and latency budget constraints. The calculated optimal rank enables precise matching between the expressive capabilities of the large model carried by the heterogeneous vehicle-mounted intelligent terminal and the terminal resources, so that the large model carried by the heterogeneous vehicle-mounted intelligent terminal maximizes the coverage of the main spectrum energy while satisfying the latency budget of the vehicle-mounted intelligent terminal.

[0042] Specifically, by calculating the singular value energy retention rate Based on the rank of the measurement The degree to which the low-rank fit subspace preserves the principal energy of the pre-trained weights is calculated as follows: ; In the formula, Indicates rank as The main spectrum energy that can be covered by the low-rank fit subspace. The optimal rank is The total energy corresponding to the super adapter, ,and The value of is kept constant to prevent the denominator from approaching zero, where the energy retention rate is... Following the order Monotonically increasing is the theoretical basis for measuring that "the higher the rank, the stronger the expressive power." Indicates the first The energy contribution of each spectral direction in the sense of the Frobenius norm.

[0043] Subsequently, the cloud server establishes a computational latency model and a communication latency model for each in-vehicle intelligent terminal. The computational latency model addresses the local training computational latency of the in-vehicle intelligent terminal, while the communication latency model addresses the communication latency of the in-vehicle intelligent terminal during data transmission.

[0044] In this embodiment, the k-th in-vehicle intelligent terminal is trained globally. The available resources of a wheel are abstracted into a vector representation, resulting in a resource state vector. The expression is as follows: ; In the formula, This represents the computational throughput of the k-th in-vehicle intelligent terminal, expressed in FLOPs / s. This represents the effective transmission bandwidth of the k-th in-vehicle intelligent terminal, in bytes per second. This represents the total training latency that the k-th in-vehicle intelligent terminal can tolerate in a vehicle-road cooperative perception scenario.

[0045] For rank The The computational latency model constructed by an in-vehicle intelligent terminal As shown below: ; In the formula, Indicates the number of epochs. Indicates the first The number of local driving perception samples for each in-vehicle intelligent terminal. Represents the feature dimension scale. , This represents the floating-point operation constant for each sample and each dimension. Indicates the first The computing throughput of an in-vehicle intelligent terminal, measured in FLOPs / s. Indicates in-vehicle intelligent terminal The calculation of the delay coefficient, .

[0046] For rank The Communication latency model constructed by an in-vehicle intelligent terminal As shown below: ; In the formula, Indicates the first The effective transmission bandwidth of each vehicle-mounted intelligent terminal Indicates the first The communication latency coefficient of an in-vehicle intelligent terminal .

[0047] The total delay is obtained by adding the computation delay to the communication delay. , means as follows: ; Furthermore, by calculating the ratio of the maximum acceptable training latency budget to the sum of the corresponding coefficients of the total latency for the in-vehicle intelligent terminal in a vehicle-road cooperative perception scenario, the feasible rank range of the k-th in-vehicle intelligent terminal is obtained. The expression is as follows: ; In the formula, Indicates the first The maximum acceptable training latency budget for an in-vehicle intelligent terminal in a vehicle-road cooperative perception scenario. .

[0048] To maximize energy retention while satisfying edge latency, based on the maximum training latency budget of the in-vehicle intelligent terminal, the optimal rank that the k-th in-vehicle intelligent terminal can withstand under latency constraints is calculated using the following formula. : ; in, This indicates rounding down, ensuring that the vehicle's intelligent terminal does not experience timeouts due to excessively high rank, nor waste available resources due to excessively low rank.

[0049] In this embodiment, due to the energy retention rate Following the order Since it is monotonically increasing, maximizing the energy retention rate under time delay constraints is equivalent to selecting the maximum feasible rank that satisfies the constraints. This is the optimal rank, which also corresponds to the maximum energy retention rate.

[0050] Furthermore, the cloud server calculates the optimal rank that each in-vehicle intelligent terminal can withstand, thus obtaining the set of in-vehicle intelligent terminals participating in this round. The optimal rank configuration is given by the following expression: .

[0051] In this embodiment, a nested truncation operator is also established. Based on the super adapter, the cloud server generates sub-adaptors with different ranks but nested column spaces for vehicle intelligent terminals with different computing power through the established nested truncation operator, so as to realize the adaptation of heterogeneous vehicle intelligent terminals in the flow of unified parameters, so as to be compatible with vehicle intelligent terminals with different computing power levels.

[0052] Specifically, the constructed nested truncation operator as follows: In the t-th round of training, for the t-th Each in-vehicle intelligent terminal, based on its supported optimal rank , Extract the global input rank matrix for the current round. The former The column yields the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal. The expression is as follows: ; In the formula, , Represents the first step of global training. wheel, This indicates extracting the global input rank matrix. The former The input truncation matrix obtained after the column.

[0053] Extract the global output rank matrix of the current round forward The output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal is obtained by row. The expression is as follows: ; In the formula, , This indicates extracting the global output rank matrix. forward The output truncation matrix obtained from the row.

[0054] Finally, based on the combination of the input low-rank sub-adaptor matrix and the output low-rank sub-adaptor matrix, a sub-adaptor that matches the resources of the vehicle-mounted intelligent terminal is obtained.

[0055] Furthermore, regarding the first The vehicle-mounted intelligent terminal and the first Each in-vehicle intelligent terminal, if its order satisfies Then the column space of the corresponding adapter satisfies the following nesting relationship: ; In the formula, This represents the input low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. This represents the output low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. Let represent the input low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal. Let represent the output low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal.

[0056] This ensures that the low-rank model is a consistent projection of the high-rank model onto a low-dimensional subspace, rather than independent subnetworks. Through this nested structure, this invention enables collaborative training of in-vehicle intelligent terminals with various computing power levels on a unified parametric manifold, laying the foundation for subsequent dynamic rank configuration and heterogeneous subspace fusion.

[0057] In this embodiment, a semantic manifold anchoring mechanism based on prototype vectors is also proposed. This mechanism constructs a stable global perception semantic reference frame and guides local features to align with this manifold, achieving feature learning for consistent perception semantics across vehicle nodes. Specifically, based on the perception category classification of the vehicle-road cooperative perception task, the following global perception semantic prototype set is constructed on the cloud server: ; In the formula, Indicates the perception category as Globally perceptive semantic prototype vectors. And it satisfies the following normalization conditions , , This represents the overall perception category in the vehicle-road cooperative perception task. The perception category includes target types in traffic scenarios such as vehicles, pedestrians, non-motorized vehicles, traffic signs, and lane lines. This indicates the dimension of the output feature, such as the dimension of the hidden vector in the last layer of a Transformer.

[0058] Finally, the cloud server packages the sub-adapter and the global awareness semantic prototype set into a data packet and sends it to the corresponding in-vehicle intelligent terminal. Each in-vehicle intelligent terminal receives the same global awareness semantic prototype set. (Regarding the first...) Data packets sent by an in-vehicle intelligent terminal The expression is as follows: ; In the formula, Represents the set of globally perceptual semantic prototypes for the previous iteration round: round t-1, when At that time, it represents the initial set of globally aware semantic prototypes constructed.

[0059] Step A3: Each in-vehicle intelligent terminal updates the sub-adapter parameters and the local perception semantic prototype set based on the issued sub-adapter and global perception semantic prototype set, using the locally stored driving perception dataset, calculates the parameter increment of the sub-adapter, and uploads the parameter increment, the local perception semantic prototype set, and the number of local driving perception samples corresponding to each perception category to the cloud server.

[0060] Specifically, the large model carried by each in-vehicle intelligent terminal includes a pre-trained backbone model, sub-adaptors, and a set of local perception semantic prototypes.

[0061] Among them, the pre-trained weight matrix of the pre-trained backbone model Freeze it from training to maintain consistency with the large model.

[0062] The trainable rank in the sub-adapter is issued by the cloud server. Sub-adapter parameters: , This process is updated.

[0063] Locally-aware semantic prototype set: It is used to capture the local perceptual semantic distribution and is updated using the EMA method.

[0064] In this embodiment, a dual-track joint optimization objective of task loss and semantic loss is constructed. After updating the adapter parameters and the local perception semantic prototype, the parameter increment is uploaded. Through the dual-track joint optimization of task loss and semantic loss, the vehicle intelligent terminal can simultaneously optimize task performance and semantic consistency on its local driving perception data distribution, so that different vehicle intelligent terminals can maintain consistency with the large model in both feature space and parameter space.

[0065] Specifically, regarding the first For a vehicle-mounted intelligent terminal, the following joint loss function is constructed during the dual-track joint optimization process: ; In the formula, Represents the total loss function. This represents the standard cross-entropy loss. Indicates semantic loss, The weights represent the semantic alignment loss, used to balance task performance and semantic consistency. .

[0066] The standard cross-entropy loss is used to represent the loss of the perceptual task, and its expression is as follows: ; In the formula, , Indicates the first Local driving perception dataset of each in-vehicle intelligent terminal ,in, This represents the i-th local driving perception sample, i.e., the multimodal perception data collected by the vehicle's sensors. Indicates and The corresponding scene annotation data, i.e., the true category label of the perceived sample, The classification probability is expressed as follows: .

[0067] The semantic loss is established to prevent perceptual feature space drift, and its expression is as follows: ; In the formula, Represents normalized perceptual features. , This represents the globally perceptual semantic prototype vector from the previous iteration. This is the initial global-aware semantic prototype vector. This indicates the temperature hyperparameter.

[0068] In this example, a stochastic gradient descent optimizer is used to update the sub-adaptor parameters, including taking the gradient of the joint loss: ; In the formula, Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix of the input of the k-th vehicle-mounted intelligent terminal in the t-th round of training. gradient, Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix output by the k-th vehicle-mounted intelligent terminal in the t-th round of training. The gradient.

[0069] The sub-adaptor parameters are updated using a stochastic gradient descent (SGD) optimizer, including updating the input low-rank sub-adaptor values. and output low-rank sub-adaptor update value : ; ; In the formula, This represents the initial value of the input low-rank sub-adapter matrix of the k-th vehicle-mounted intelligent terminal in the t-th round of training. Let represent the initial value of the output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal during the t-th round of training. Let represent the updated value of the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal after the t-th round of training. This represents the updated value of the low-rank sub-adapter matrix output by the k-th in-vehicle intelligent terminal after the t-th round of training. The learning rate is represented by the learning rate, which is scheduled locally by the vehicle's intelligent terminal.

[0070] Specifically, the formulas for calculating the parameter increments of the input low-rank sub-adaptor and the output low-rank sub-adaptor are as follows: ; ; In the formula, , It is the rank assigned to the k-th in-vehicle intelligent terminal in this round, and is much smaller than .

[0071] Each in-vehicle intelligent terminal extracts the normalized perception feature vector of the local driving perception sample and constrains its convergence to align the embedding spaces of different in-vehicle intelligent terminals. Specifically, the normalized perception feature vector of the local driving perception sample extracted by the k-th in-vehicle intelligent terminal... The expression is as follows: ; In the formula, Represents local driving perception samples The original feature vector, , This represents the large model feature extractor for the k-th in-vehicle intelligent terminal. .

[0072] In this embodiment, a semantic contrastive loss function based on InfoNCE is used to make the normalized perceptual feature vector Towards the global perceptual semantic prototype vector corresponding to its perceptual category Convergence, expressed as follows: ; In the formula, Represents local driving perception samples The corresponding real category label, Indicates temperature parameter, .

[0073] In this embodiment, each in-vehicle intelligent terminal maintains its local perception semantic prototype locally and updates it using an exponential moving average (EMA) mechanism to capture the perception categories at the in-vehicle intelligent terminal side. The local semantic distribution structure, taking the k-th in-vehicle intelligent terminal as an example, is expressed as follows: ; In the formula, Indicates the first The vehicle-mounted intelligent terminal in the first After the round of global collaborative training is completed, the updated local perceptual semantic prototype vector corresponding to the perceptual category c is obtained. , Indicates the first The vehicle-mounted intelligent terminal in the first After -1 rounds of global collaborative training, the historical local perceptual semantic prototype vector corresponding to perceptual category c is obtained. Indicates the perception category as The local driving perception sample set, , Indicates local training batch. This represents the momentum smoothing coefficient of the EMA.

[0074] Subsequently, the in-vehicle intelligent terminal uploads parameter increments, the local perception semantic prototype set, and the number of local driving perception samples corresponding to each perception category to the cloud server. Taking the k-th in-vehicle intelligent terminal as an example, the expression is as follows: ; In the formula, This represents the total number of perception samples corresponding to category c for the k-th in-vehicle intelligent terminal during the t-th round of local training.

[0075] Step A4: The cloud server receives the uploaded information, completes the aggregation of parameter increments, and aggregates the local perception semantic prototype set based on the number of local driving perception samples corresponding to each perception category, and updates the super adapter and the global perception semantic prototype set.

[0076] Specifically, the cloud server unifies the parameter increment dimension by constructing a zero-fill projection operator, performs weighted aggregation of parameter increments in combination with the coverage vector to update the super adapter, and aggregates the local perception semantic prototype sets based on the number of local driving perception samples corresponding to each perception category to update the global perception semantic prototype set.

[0077] Specifically, the established zero-filling projection operator This includes zero-padding the column direction of the parameter increments for the input low-rank sub-adapter, as shown in the following expression: .

[0078] In addition, the parameter increments of the output low-rank sub-adapter are zero-padded in the row direction, as shown in the following expression: .

[0079] Since in-vehicle intelligent terminals of different ranks are used to update the spectral coverage of the large model, each spectral direction needs to be normalized according to how many in-vehicle intelligent terminals have effectively updated it. Based on this, a coverage vector is established: ; Among them, the Dimension is defined as: ; In the formula, Indicates the first The number of all local driving perception samples for each in-vehicle intelligent terminal, indicating function Indicates the first Does the sub-adapter of the vehicle-mounted intelligent terminal cover the first... In each spectral direction, if If so, the vehicle-mounted intelligent terminal makes no update contribution in that direction.

[0080] Subsequently, the cloud server independently performs weighted aggregation on each spectral direction based on the calculation results, including updating the global input low-rank sub-adaptor matrix and updating the global output low-rank sub-adaptor matrix.

[0081] Specifically, the input low-rank sub-adaptor matrix corresponding to the j-th spectral direction The update expression is as follows: ; In the formula, Represents the first sub-adapter matrix of the input low-rank sub-adapter matrix. List, As a stability constant, when the maximum rank of all in-vehicle intelligent terminals is the same, .

[0082] The output low-rank adapter matrix corresponding to the j-th spectral direction The update expression is as follows: .

[0083] In the formula, The first sub-adapter matrix represents the output low-rank sub-adapter matrix. OK.

[0084] Furthermore, the cloud server needs to integrate the local perception semantic prototype sets uploaded by each in-vehicle intelligent terminal. The cloud server performs global perception semantic prototype vector aggregation, weighted by the number of local driving perception samples corresponding to each perception category, as shown in the following expression: .

[0085] In the formula, Let c represent the intermediate vector of the global perceptual semantic prototype after the sample size weighted average in round t, without normalization. This represents the local perception semantic prototype vector of perception category c uploaded by the k-th in-vehicle intelligent terminal in round t, which is consistent with the updated local prototype definition of EMA.

[0086] And perform unitization as follows: .

[0087] In the formula, This represents the final global perceptual semantic prototype vector corresponding to the perceptual category c after the t-th round of updates.

[0088] If a certain perception category is not covered by any in-vehicle intelligent terminal in this round, then remain: .

[0089] Step A5: Based on the updated super adapter and the global perception semantic prototype set, the cloud server iterates from step A2 to step A4 until the large model converges. Each vehicle-mounted intelligent terminal then performs vehicle-road cooperative perception tasks through the converged large model.

[0090] In the first iteration, the parameter initialization process of the super adapter is as follows: Select an orthogonal matrix. The former The columns form a submatrix, and the first column of the singular value diagonal matrix is ​​selected. The singular values ​​form a diagonal submatrix, expressed as follows: ; ; In the formula, Represents a submatrix. , Represents a diagonal submatrix. , This represents a diagonal matrix whose diagonal elements are the square roots of their respective elements. This represents a matrix consisting entirely of zeros.

[0091] By outputting the global rank matrix Initialize it as a zero matrix, such that the initial increment... This ensures that the output of the pre-trained large model remains unchanged during initialization, avoiding performance degradation in the early stages of fine-tuning and achieving lossless initialization. Furthermore, during initialization, the global input rank matrix... It is composed of weighted principal singular vectors and encodes the pre-trained weight matrix. The energy distribution along the main spectrum direction allows the fitting subspace to naturally align with the intrinsic geometric structure of the pre-trained weights, accelerating the convergence of subsequent collaborative training and completing the spectral alignment.

[0092] Furthermore, during the first iteration, any one of the following three global-aware semantic prototype initialization schemes can be chosen to initialize the global-aware semantic prototype set. The three global-aware semantic prototype initialization schemes include: Initialization is driven by a pre-trained large model, which utilizes the embedded features of the pre-trained large model on a public driving dataset and uses the mean of features of each perception category as the initial prototype.

[0093] Random unit hypersphere initialization, in Uniform sampling Initialize in a random direction, as shown in the expression: .

[0094] In the first round of data-driven initialization, if the training system allows the collection of local perception semantic prototypes from all in-vehicle intelligent terminals in round 0, then the initial global perception semantic prototype can be directly aggregated, and the expression is: .

[0095] After the large model converges, the onboard intelligent terminals of each vehicle node perform the following vehicle-road cooperative perception tasks through the converged large model: While the vehicle is in motion, the onboard intelligent terminal receives multimodal perception data collected in real time by onboard sensors, including LiDAR point cloud data, camera image data, millimeter-wave radar data, etc. The large model performs environmental understanding and scene reasoning on the input multimodal perception data and outputs perception decision results, including but not limited to target detection results, semantic segmentation results, lane line recognition results, driving path planning suggestions, collision risk warnings, etc.

[0096] This invention is applied to the collaborative training of large-scale pre-trained models by multiple types of vehicle-mounted intelligent terminals, simultaneously satisfying constraints on computational latency, communication latency, and task semantic consistency. The method achieves collaborative alignment of large models in the parameter space and feature space by constructing a nested low-rank adapter parametric manifold, a dynamic rank configuration model based on singular value energy, a prototype-guided semantic manifold anchoring mechanism, a spectral energy-driven rank orchestration protocol, and a heterogeneous subspace fusion strategy with coverage compensation.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for collaborative training of heterogeneous edge large models based on nested manifold alignment, characterized in that, Includes the following steps: Step A1: Perform low-rank incremental modeling on the pre-trained weights of the pre-trained large model for vehicle-road cooperative perception tasks carried by each vehicle-mounted intelligent terminal, and perform singular value decomposition on the pre-trained weight matrix. Step A2: Establish a super adapter on the cloud server to be compatible with different rank configurations, calculate the optimal rank of each vehicle intelligent terminal, and obtain sub-adaptors from the super adapter according to the optimal rank of the vehicle intelligent terminal. After constructing a global perception semantic prototype set based on the perception category division of the vehicle-road cooperative perception task, the paired sub-adaptors and the global perception semantic prototype set are sent as data packets to the corresponding vehicle intelligent terminals. Step A3: Each in-vehicle intelligent terminal updates the sub-adapter parameters and local perception semantic prototype set based on the issued sub-adapter and global perception semantic prototype set with the locally stored driving perception dataset, calculates the parameter increment of the sub-adapter, and uploads the parameter increment, local perception semantic prototype set, and the number of local driving perception samples corresponding to each perception category to the cloud server. Step A4: The cloud server receives the uploaded information, completes the aggregation of parameter increments, and aggregates the local perception semantic prototype set based on the number of local driving perception samples corresponding to each perception category, and updates the super adapter and the global perception semantic prototype set. Step A5: Based on the updated super adapter and the global perception semantic prototype set, the cloud server iterates from step A2 to step A4 until the large model converges. Each vehicle-mounted intelligent terminal then performs vehicle-road cooperative perception tasks through the converged large model.

2. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 1, characterized in that, In step A1, the following trainable low-rank increment matrix is ​​constructed. : ； In the formula, The dimension is The input rank matrix, , Indicates the input feature dimension. The dimension is The output rank matrix, , Indicates the output feature dimension. Indicates rank. .

3. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 2, characterized in that, In step A2, the parameters of the constructed super adapter are as follows: ； ； In the formula, This represents the global input rank matrix, corresponding to the mapping from the input feature dimension to the rank dimension. This represents the global output rank matrix, corresponding to the mapping from the rank dimension to the output feature dimension. This indicates the maximum rank supported by the highest-performance in-vehicle intelligent terminal.

4. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 1, characterized in that, In step A2, the optimal rank of each vehicle-mounted intelligent terminal is calculated on a cloud server based on singular value energy retention rate and time delay budget constraints.

5. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 4, characterized in that, The singular value energy retention rate is measured in terms of rank. The degree of preservation of the principal energy of the pre-trained weights by the low-rank fit subspace is calculated.

6. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 4, characterized in that, The delay budget constraints include computation delay constraints and communication delay constraints.

7. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 3, characterized in that, In step A2, a nested truncation operator is also constructed. Based on the super adapter, the cloud server generates sub-adaptors with different ranks but nested column spaces for in-vehicle intelligent terminals with different computing power through the constructed nested truncation operator. as follows: In the t-th round of training, for the t-th Each in-vehicle intelligent terminal, based on its supported optimal rank , Extract the global input rank matrix for the current round. The former The column yields the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal. The expression is as follows: ； In the formula, , This indicates extracting the global input rank matrix. The former The input truncation matrix obtained after the column; Extract the global output rank matrix of the current round forward The output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal is obtained by row. The expression is as follows: ； In the formula, , This indicates extracting the global output rank matrix. forward The output truncation matrix obtained from the row; For the The vehicle-mounted intelligent terminal and the first A vehicle-mounted intelligent terminal, if its order satisfies Then the column space of the corresponding adapter satisfies the following nesting relationship: ； In the formula, This represents the input low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. This represents the output low-rank sub-adapter matrix corresponding to the k-th in-vehicle intelligent terminal. Let represent the input low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal. Let represent the output low-rank sub-adapter matrix corresponding to the j-th in-vehicle intelligent terminal.

8. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 7, characterized in that, In step A3, the formulas for calculating the parameter increments of the input low-rank sub-adaptor and the output low-rank sub-adaptor are as follows: ； ； In the formula, This represents the initial value of the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal during the t-th round of training. Let represent the initial value of the output low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal during the t-th round of training. Let represent the updated value of the input low-rank sub-adapter matrix of the k-th in-vehicle intelligent terminal after the t-th round of training. This represents the updated value of the low-rank sub-adapter matrix output by the k-th in-vehicle intelligent terminal after the t-th round of training. , It is the rank assigned to the k-th in-vehicle intelligent terminal in this round, and is much smaller than .

9. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 8, characterized in that, The input low-rank sub-adapter update value and output low-rank sub-adaptor update value The update is performed using a stochastic gradient descent optimizer, as shown in the following expression: ； ； In the formula, The learning rate is represented by the learning rate, which is locally scheduled by the in-vehicle intelligent terminal. Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix of the input of the k-th vehicle-mounted intelligent terminal in the t-th round of training. gradient, Represents the total loss function Initial values ​​are distributed to the low-rank sub-adapter matrix output by the k-th vehicle-mounted intelligent terminal in the t-th round of training. The gradient.

10. The heterogeneous edge large model collaborative training method based on nested manifold alignment according to claim 1, characterized in that, In step A4, after receiving the information uploaded by each vehicle-mounted smart terminal, the cloud server unifies the parameter increment dimension through the zero-fill projection operator and performs weighted aggregation of parameter increments in combination with the coverage vector to update the super adapter.

Images