Fedfish federated learning method and system for vehicle networking automatic driving long tail scene

Abstract

This invention discloses a FedFish federated learning method and system for long-tail scenarios in connected vehicle autonomous driving. It addresses the technical bottlenecks of existing dynamic sparse federated learning techniques in connected vehicle non-independent and identically distributed data scenarios, such as the mispruning of rare feature parameters and global catastrophic forgetting caused by over-reliance on amplitude pruning, and the dual memory and computing power overflow faced by traditional second-order optimization algorithms on in-vehicle computing devices. This invention introduces the Fisher information matrix from information geometry as a novel second-order evaluation criterion for network topology evolution at the in-vehicle micro-nodes. Simultaneously, through fundamental innovations in engineering implementation, it achieves high-quality second-order curvature approximation at extremely low cost with zero additional GPU memory usage and zero additional backpropagation computing power. This completely eradicates the forgetting phenomenon in long-tail scenarios under the stringent condition of extremely compressed vehicle-to-cloud communication bandwidth, enabling the autonomous driving model to achieve high-precision, highly robust, and smooth convergence in heterogeneous networks.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence, autonomous driving and distributed privacy computing, and specifically relates to the FedFish federated learning method and system for long-tail scenarios of vehicle-to-everything (V2X) autonomous driving. Background Technology

[0002] In recent years, deep learning has made groundbreaking progress in the fields of environmental perception and object detection for autonomous driving. Traditional autonomous driving model training heavily relies on the centralized collection of massive road test data to the cloud (automotive company data centers). However, with the tightening of global data privacy and security regulations, and the extremely high cost of transmitting massive high-definition video streams via in-vehicle wide area networks (4G / 5G), a large amount of real-world edge road condition data cannot be fully transmitted back, resulting in widespread data silos.

[0003] To break this deadlock, Federated Learning (FL) has been introduced into the Internet of Vehicles (IoV). Its core paradigm is "data remains stationary, model moves": a massive number of vehicles on the road update their models using only local computing power, then upload encrypted parameters to the cloud for global aggregation. However, deploying it in a real-world mass-production automotive environment faces two extremely severe engineering bottlenecks: First, there are extremely limited communication and in-vehicle computing bottlenecks: the parameters of large-scale vision models for autonomous driving can reach tens of millions to hundreds of millions. In the unstable and bandwidth-limited environment of in-vehicle wide area networks, frequent uploading of dense network parameters not only consumes a huge amount of bandwidth but also easily leads to communication crashes. In addition, the computing power of in-vehicle computing platforms (such as NVIDIA Orin) must prioritize real-time safe driving perception tasks, and can only allocate a very small portion of free GPU memory (usually less than 2GB) for background model fine-tuning. Traditional quadratic derivative algorithms can cause on-the-spot memory overflow (OOM).

[0004] Second, the catastrophic amnesia caused by extreme long-tail scenarios (corner cases): Data collected from vehicles in different regions exhibits extreme non-independent identically distributed (Non-IID) characteristics. For example, some vehicles may have only encountered a fallen cardboard box or a wild animal crossing the road for a very short period of time. If existing dynamic sparsity reduction techniques (FedDST) are used, the system will mechanically remove the small weights responsible for identifying these rare features as redundant noise, causing the global autonomous driving model aggregated in the cloud to permanently lose its ability to identify these rare and dangerous scenarios, leading to fatal driving hazards.

[0005] While existing technologies have alleviated the pain points of federated learning in vehicle-to-everything (V2X) systems to some extent, they still suffer from the following serious underlying theoretical flaws and engineering limitations when dealing with complex driving environments that contain a large number of long-tail scenarios (corner cases): The full federation framework (FedAvg / FedProx) cannot overcome the physical communication barrier, static sparse federation loses its long-tail adaptive capability, dynamic sparse federation based on amplitude pruning (FedDST) has a fatal importance assessment misalignment trap, and traditional second-order theoretical solutions face the engineering fatal flaw of double overflow of memory and computing power. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a FedFish federated learning method and system for long-tail scenarios in vehicle-to-everything (V2X) autonomous driving. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles, including: S1 utilizes the vehicle manufacturer's cloud central server to build a deep visual neural network model for autonomous driving perception, generates the initial global sparse parameters of the model according to the preset initialization strategy, and distributes the initial global sparse parameters of the model to all vehicle edge nodes participating in this round of training. S2, based on the initial global sparse parameters of the model received by the vehicle edge node, uses the heterogeneous road condition privacy dataset collected by the vehicle edge node during local driving to perform multiple rounds of local iterative training, and obtains the unbiased continuous approximation of the local diagonal Fisher information matrix of the vehicle edge node through a pre-built state parasitic second-order information extraction mechanism. S3, when the local iterative training reaches the preset topology evolution frequency, calculate the second-order sensitivity survival score of each active connection in the current network based on the unbiased continuous approximation and weight value; based on the second-order sensitivity survival score, set the mask corresponding to the active connection with the preset dormant ratio to zero, and treat it as a dormant connection; S4, calculate the absolute value of the instantaneous loss gradient of the currently masked dormant connection with respect to the current batch of input data; based on the absolute value of the instantaneous loss gradient, reactivate the mask of the dormant connection with the preset connection ratio and initialize the weights to obtain the updated local mask; S5, encrypt and upload the updated local mask and the corresponding non-zero parameter weights to the vehicle manufacturer's cloud center server; use the vehicle manufacturer's cloud center server to execute the sparse aggregation protocol, generate new global sparse parameters and distribute them, iteratively execute steps S2~S5 until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network. The number of active connections with a preset sleep ratio is the same as the number of sleep connections with a preset connection ratio.

[0007] In one embodiment of the present invention, the initial global sparse parameters include: The global initial model parameters and their mask topology after sparsification.

[0008] In one embodiment of the present invention, in the second-order information extraction mechanism, when the car edge node performs local mini-batch backpropagation update based on the adaptive optimizer, the gradient squared exponential moving average vector maintained by the optimizer in the internal working area is directly intercepted and extracted as an unbiased continuous approximation of the local diagonal Fisher information matrix F of the car edge node.

[0009] In one embodiment of the present invention, the expression for the unbiased persistence approximation is as follows: ; in, For the first t Unbiased continuity approximation for each round, The second-order momentum decay rate of the optimizer. for t The squared exponential moving average vector of the gradient after -1 rounds. The first-order cross-entropy loss gradient is calculated for the current batch of road condition data.

[0010] In one embodiment of the present invention, the second-order sensitivity survival score of each active connection is the product of the unbiased persistence approximation corresponding to each active connection and the square of the absolute value of its weight.

[0011] In one embodiment of the present invention, the step of setting the mask corresponding to the active connection with a preset dormant ratio to zero, based on the second-order sensitivity survival score, to serve as the dormant connection includes: All active connections are sorted in ascending order based on their second-order sensitivity survival scores. The masks of active connections with low scores are set to zero according to a preset ratio, and these connections are then designated as dormant connections.

[0012] In one embodiment of the present invention, based on the absolute value of the instantaneous loss gradient, the mask of dormant connections with a preset connection ratio is reactivated and the weights are initialized to obtain an updated local mask, including: Dormant connections are sorted from largest to smallest based on the absolute value of the instantaneous loss gradient. The mask corresponding to the dormant connection with the largest absolute gradient value is set to 1 according to the preset connection ratio, and the weights are initialized and activated with zero value or preset Gaussian noise.

[0013] Secondly, this invention provides a FedFish federated learning system for long-tail scenarios of autonomous driving in the Internet of Vehicles, comprising: The car manufacturer's cloud center server is used to generate the initial global sparse parameters of the model according to the preset initialization strategy, and then distribute the initial global sparse parameters of the model to all car edge nodes participating in this round of training. The vehicle edge node is used to perform multiple rounds of local iterative training on a heterogeneous road condition privacy dataset collected during local driving, based on the initial global sparse parameters it receives. Through a pre-built state-parasitic second-order information extraction mechanism, it obtains an unbiased, persistent approximation of the local diagonal Fisher information matrix of the vehicle edge node. When the local iterative training reaches a preset topology evolution frequency, it calculates the second-order sensitivity survival score of each active connection in the current network based on the unbiased, persistent approximation and weight values. Based on the second-order sensitivity survival score, the masks corresponding to a preset dormant proportion of active connections are set to zero, becoming dormant connections. The absolute value of the instantaneous loss gradient of the currently masked dormant connection with respect to the current batch of input data is calculated. Based on the absolute value of the instantaneous loss gradient, the masks of the preset proportion of dormant connections are reactivated and their weights are initialized to obtain an updated local mask. The updated local mask and its corresponding non-zero parameter weights are encrypted and uploaded to the vehicle manufacturer's cloud central server. The cloud-based central server of the car manufacturer is also used to execute a sparse aggregation protocol, generate new global sparse parameters and distribute them for the next round of iterative training until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network.

[0014] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided by the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided in the embodiments of the present invention.

[0016] The beneficial effects of this invention are: The solution provided by this invention breaks through the traditional low-order numerical heuristic criteria and designs a state parasitic second-order information extraction mechanism based on the Fisher information matrix. This mechanism provides a strict theoretical protection boundary for weight parameters that are in a local dormant state but carry global key features (such as extreme long-tail road conditions and Corner Cases), effectively blocking the path of catastrophic forgetting, thereby significantly improving the generalization ability and safety of the autonomous driving perception model in harsh long-tail environments. By seamlessly reusing the acquisition of second-order information into the exponential moving average state variables maintained inside the adaptive optimizer, unbiased estimation of the local true curvature of parameters is achieved under the stringent conditions of not increasing the memory usage of the on-board chip and not introducing any additional backpropagation floating-point operations. This gives physical feasibility to execute complex second-order optimization algorithms at the vehicle edge nodes and ensures that the background federated fine-tuning task never interferes with the real-time autonomous driving main perception process, completely avoiding the risk of memory overflow. This invention uses second-order historical Fisher information for robust pruning of active connections and first-order instantaneous gradients for aggressive regeneration of dormant connections, endowing sparse topologies with powerful dynamic optimization and adaptive capabilities, accelerating the overall convergence speed and stability of federated networks, and compressing the amount of vehicle-to-cloud bidirectional communication data by more than 80% during the global federated training lifecycle, greatly reducing the traffic bandwidth load and overall operating cost of large-scale distributed collaborative training based on vehicle networks (4G / 5G). Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the steps of a FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram showing the performance comparison between the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided by an embodiment of the present invention and existing algorithms in a 2-class scenario with extremely fragmented data. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] Existing dynamic sparse federated learning techniques face several challenges in the context of non-independent and identically distributed (Non-IID) data in connected vehicles. These challenges stem from over-reliance on magnitude pruning, leading to the mispruning of rare feature parameters and catastrophic global forgetting. Furthermore, traditional second-order optimization algorithms (Hessian matrix or full Fisher matrix) suffer from memory and computational power overflows on in-vehicle computing devices. The core objective of this invention is to break away from the traditional heuristic blind spot of judging importance solely by the absolute value of parameters, and to introduce the Fisher Information Matrix (FIM) from information geometry as a novel second-order evaluation criterion for network topology evolution at the vehicle-mounted micro-nodes. Simultaneously, through fundamental innovations in engineering implementation, it achieves high-quality second-order curvature approximations at extremely low cost with zero additional GPU memory usage and zero additional backpropagation computation. This completely eradicates the long-tail scene forgetting phenomenon under the stringent conditions of extreme compression of vehicle-to-cloud communication bandwidth (e.g., sparsity exceeding 80%), enabling deep visual neural network models for autonomous driving perception to achieve high-precision, highly robust, and smooth convergence in heterogeneous networks.

[0020] To address the shortcomings of existing dynamic sparse federated learning techniques in extreme non-independent and identically distributed (Non-IID) data scenarios, such as the inability of the full federated framework (FedAvg / FedProx) to overcome physical communication barriers, the loss of long-tail adaptive capabilities in static sparse federation, the fatal importance assessment misalignment trap in amplitude-pruning-based dynamic sparse federation (FedDST), and the dual problems of memory and computing power overflow faced by traditional second-order theoretical solutions, this invention provides a FedFish federated learning method, system, electronic device, and storage medium for long-tail scenarios in connected vehicle autonomous driving.

[0021] It should be noted that the execution entity of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided in this embodiment of the invention can be a system, which can run in an electronic device. This electronic device can be a server or a terminal device, but is not limited to these.

[0022] Below, we will first introduce a FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles, provided by an embodiment of the present invention.

[0023] This invention provides a FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles, as shown in the embodiments of the present invention. Figure 1 As shown, it may include the following steps: S1 utilizes the vehicle manufacturer's cloud-based central server to construct a deep visual neural network model for autonomous driving perception. It generates the initial global sparse parameters of the model according to a preset initialization strategy and distributes the initial global sparse parameters of the model to all vehicle edge nodes participating in this round of training.

[0024] In the In the initial stage of federated training, the automaker's cloud-based central server constructs a deep visual neural network model for autonomous driving perception, whose full parameter vector is denoted as... Based on the preset wide area network communication bandwidth limitations of the vehicle system, a target sparsity is set. (For example This means that 80% of the parameters will be set to zero. The automaker's cloud-based central server utilizes Erd... The s-Rényi (ERK) topology initialization strategy or the uniform random distribution strategy generates a topology that satisfies sparsity. Initial binary indicator mask , making The initial global sparse parameters may include: the sparsified global initial model parameters and their mask topology. The vehicle manufacturer's cloud-based central server will then process the sparsified global initial model parameters. and its mask topology The data is distributed to all vehicle edge nodes participating in this round of training via the vehicle-to-everything (V2X) network.

[0025] S2, through the vehicle edge node based on the initial global sparse parameters of the model it receives, uses the heterogeneous road condition privacy dataset collected by the vehicle edge node during local driving to perform multiple rounds of local iterative training, and obtains the unbiased continuous approximation of the local diagonal Fisher information matrix of the vehicle edge node through a pre-built state parasitic second-order information extraction mechanism.

[0026] Specifically, vehicle edge nodes After receiving the global sparse parameters, it utilizes the heterogeneous road condition privacy dataset collected during its local driving. Local training is performed over multiple epochs. To overcome the physical computing power and memory limitations of calculating the second-order Hessian matrix in in-vehicle devices, this invention constructs a state-parasitic second-order information extraction mechanism. In this mechanism, at the in-vehicle end, when the vehicle edge node performs local mini-batch backpropagation updates based on an adaptive optimizer (such as Adam or AdamW), the gradient squared exponential moving average vector (i.e., the second-order moment state variable) maintained by the optimizer in its internal working area is directly intercepted and extracted. ), which is used as an unbiased continuous approximation of the local diagonal Fisher information matrix F of the car edge node.

[0027] The expression for the unbiased persistence approximation is as follows: ; in, For the first t Unbiased continuity approximation for each round, This is the second-order momentum decay rate of the optimizer (usually set to 0.999 by default). for t The squared exponential moving average vector of the gradient after -1 rounds. This is the first-order cross-entropy loss gradient calculated from the current batch of road condition data. This step cleverly integrates the acquisition of second-order information entirely into the standard lifecycle of forward / backward propagation, achieving strictly zero additional memory allocation and zero additional FLOPs overhead, ensuring that it does not interfere with the autonomous driving main perception program.

[0028] S3, when the local iterative training reaches the preset topology evolution frequency, calculate the second-order sensitivity survival score of each active connection in the current network based on the unbiased continuous approximation and weight value; based on the second-order sensitivity survival score, set the mask corresponding to the active connection with the preset dormant ratio to zero, and treat it as a dormant connection.

[0029] Local training iterations reach the preset topology evolution frequency. At that time, the vehicle client triggers a sparse topology update of the micro-nodes.

[0030] The embodiments of the present invention abandon the traditional absolute value of parameters The sorting method, for all surviving active connections (i.e., masks) in the current network. (Parameters), calculate its second-order sensitivity survival score. According to the second-order Taylor expansion principle, the second-order sensitivity survival score of each active connection is the product of the unbiased persistence approximation of each active connection and the square of the absolute value of its weight.

[0031] ; in, For the first i The second-order sensitivity survival score of each active connection For the first i The unbiased persistence approximation corresponding to each active connection. For the first i The absolute value of the weight corresponding to each active connection.

[0032] Based on the second-order sensitivity survival score, the masks corresponding to active connections with a preset dormant ratio are set to zero and used as dormant connections. These can include: All active connections are sorted in ascending order based on their second-order sensitivity survival scores. The masks of active connections with low scores are set to zero according to a preset ratio, and these connections are then designated as dormant connections.

[0033] According to the embodiments of the present invention, the calculated score array Sort all active connections in ascending order. The proportion of redundant parameters with the lowest scores (whose impact on the global loss surface is minimal even if removed) is [not specified]. The connection is hard pruned, that is, its corresponding mask is... By forcibly setting the weight to 0, the weight is released. This mechanism provides an insurmountable theoretical protection barrier for rare feature parameters that are in a local data dormant state (with minimal absolute weight decay) but are responsible for identifying rare and dangerous long-tail scenarios (with extremely high local Fisher information).

[0034] S4, calculate the absolute value of the instantaneous loss gradient of the currently masked dormant connection with respect to the current batch of input data; based on the absolute value of the instantaneous loss gradient, reactivate the masks of dormant connections with a preset connection ratio and initialize the weights to obtain the updated local mask. The number of active connections with a preset dormant ratio is the same as the number of dormant connections with a preset connection ratio.

[0035] Based on the absolute value of the instantaneous loss gradient, the mask of dormant connections with a preset connection ratio is reactivated and the weights are initialized to obtain the updated local mask, which may include: Dormant connections are sorted from largest to smallest based on the absolute value of the instantaneous loss gradient. The mask corresponding to the dormant connection with the largest absolute gradient value is set to 1 according to the preset connection ratio, and the weights are initialized and activated with zero value or preset Gaussian noise.

[0036] To maintain the overall sparsity of the model S Strictly constant to meet communication bandwidth limitations, in a sleep connection pool (mask) composed of sleep connections. In the parameter space, a new number of connections are reawakened that are absolutely equal to the number pruned in step S3. Considering that dormant connections are blocked during historical forward propagation and do not contain accumulated second-order Fisher information, this embodiment of the invention uses the instantaneous first-order gradient magnitude as a probe to explore the unknown road condition structure. Specifically, the absolute value of the instantaneous loss gradient of all currently dormant connections with respect to the current batch of input data is calculated. Select the first gradient with the largest absolute value. A certain proportion of dormant connections are masked to 1 and activated with weights initialized with zero or extremely small Gaussian noise. This step achieves a perfect logical decoupling between second-order conservation (preventing forgetting) and first-order expansion (adapting to new scenarios) in topology evolution.

[0037] S5, encrypt and upload the updated local mask and the corresponding non-zero parameter weights to the vehicle manufacturer's cloud center server; use the vehicle manufacturer's cloud center server to execute the sparse aggregation protocol, generate new global sparse parameters and distribute them, iteratively execute steps S2~S5 until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network. After each vehicle edge node completes a specified number of training rounds and dynamic topology evolution, it will use the newly generated local highly personalized sparse mask. The corresponding non-zero parameter weights are encrypted and uploaded to the car manufacturer's cloud center server.

[0038] The process by which the cloud-based central server of an automotive manufacturer executes the sparse aggregation protocol is as follows: First, perform a logical OR (or union) operation on all collected vehicle heterogeneous masks, i.e. This generates a superset mask for the next round of globally effective parameters; subsequently, a weighted average operation based on the number of vehicles (or uniform weights) is performed only on the parameters within the union overlap space to generate entirely new globally sparse parameters. Then, it enters the next round of global communication cycle until the model meets the early stopping condition or reaches the maximum number of communication rounds on the retained validation set.

[0039] To verify the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles proposed in this embodiment of the invention, and the breakthrough effect of the second-order sensitivity dynamic evolution mechanism in a real harsh federated environment, this embodiment of the invention can construct a federated simulation testbed containing one central aggregation server and ten simulated heterogeneous edge nodes to verify the corresponding beneficial effects.

[0040] Experimental environment and pathological heterogeneous dataset setup: The hardware platform employs a computing cluster capable of containing multiple NVIDIA 4090 GPUs, with simulated bandwidth limiting between nodes to realistically reflect the communication bottlenecks of a Wide Area Network (WAN). To rigorously evaluate the algorithm's ability to combat "catastrophic forgetting," this embodiment uses the CIFAR-10 standard complex dataset for computer vision as the proxy verification environment and meticulously constructs an ill-conditioned 2-Class Non-IID scenario: a forced splitting of the dataset ensures that each edge client can only obtain skewed samples from 2 out of 10 categories (e.g., node A only has cats and dogs, node B only has airplanes and cars). The global communication rounds are set to 100, and the system's target sparsity is strictly locked to a specific value. .

[0041] Simulation Experiment 1: Simulation Experiment 1 aims to verify the upper limit of the generalization accuracy of the global model on the final test set. The baseline for comparison can include: the classic federated algorithm (FedAvg) with full parameter transfer, and the cutting-edge dynamic sparse federated algorithm (FedDST) based on amplitude pruning. A performance comparison diagram of different algorithms in extremely fragmented 2-class Non-IID scenarios and 80% sparsity scenarios is shown below. Figure 2As shown, in the scenario of extremely fragmented 2-class data, the traditional FedDST algorithm, due to the inherent defects of its magnitude criterion, causes each node to frantically prune key feature neurons of unseen classes, resulting in extremely severe catastrophic forgetting. The accuracy plummets to 66.15%, and the model falls into a breakdown oscillation. In contrast, the embodiment of this invention (FedFish), with the second-order protection of Fisher information, achieves a test set accuracy of 91.65% under the stringent condition of maintaining 80% extreme communication compression. This almost completely matches the full-parameter FedAvg model (83.92%), which consumes 5 times the communication bandwidth, fully demonstrating the absolute protection capability of the embodiment of this invention for key topological structures at the micro-node level.

[0042] Simulation Experiment 2: To demonstrate the engineering superiority of the zero-additional-overhead embodiments of the present invention, the embodiments of the present invention used PyTorchProfiler to accurately capture and analyze the memory usage and computation time of a single edge device during the local training phase. The comparison included methods that directly calculate the Hessian approximation matrix.

[0043] In-depth data analysis: Experimental monitoring revealed that using the traditional second-order Hessian matrix calculation method, the peak memory usage per node surged from the original 1.8 GB to 5.6 GB (an increase of >200%), and the calculation time for a single epoch increased by nearly 4.5 times. However, by employing the state parasitic mechanism proposed in this invention (directly reusing the Adam second-order matrix...)... The peak memory usage of the local node is precisely maintained at 1.81GB, and the single-step FLOPs overhead of backpropagation is completely consistent with that of standard first-order network training, with additional resource increases strictly controlled to 0%. This embodiment of the invention achieves a perfect balance between mathematical approximation and underlying engineering architecture, making it possible to run second-order optimization algorithms on micro IoT devices.

[0044] Compared to traditional full-parameter federated learning and existing amplitude-based sparse federated learning architectures (such as FedDST), this invention, starting from underlying mathematical derivation and information geometry theory, has carried out in-depth algorithm customization and system-level architecture improvements for the highly heterogeneous (Non-IID) long-tail data distribution of the Internet of Vehicles, bringing the following significant technical advantages and beneficial effects: To enhance robustness in long-tail scenarios, this invention theoretically suppresses catastrophic forgetting. Existing dynamic sparse federated technologies rely excessively on parameter magnitude priors, leading to irreversible mispruning of key parameters in extreme non-independent and identically distributed (Non-IID) environments in connected vehicles. This invention breaks through traditional low-order numerical heuristics by pioneering a topology evolution metric based on the second-order sensitivity of the Fisher Information Matrix (FIM). This mechanism provides a strict theoretical protection boundary for weight parameters that are in a "locally dormant" state but carry global key features (such as extreme long-tail road conditions and Corner Cases), effectively blocking the path to catastrophic forgetting. This significantly improves the generalization ability and safety of deep visual neural network models used for autonomous driving perception in harsh long-tail environments.

[0045] Low-level resource scheduling optimization: Zero-overhead second-order information extraction based on state reuse. In the traditional theoretical framework, the calculation of the second-order curvature matrix is ​​usually accompanied by extremely high space and time complexity, forming a core barrier to its deployment in vehicle edge computing platforms with extremely limited computing power. This invention seamlessly reuses the acquired second-order curvature information in the exponential moving average state variables maintained internally by the adaptive optimizer. Under the stringent conditions of not increasing the onboard chip's memory usage and not introducing any additional backpropagation floating-point operations, it achieves unbiased estimation of the local true curvature of the parameters. This provides physical feasibility for vehicle edge nodes to execute complex second-order optimization algorithms and ensures that background federated fine-tuning tasks never interfere with the real-time autonomous driving main perception process, completely avoiding the risk of memory overflow.

[0046] This invention achieves theoretical decoupling between preserving known knowledge and exploring unknown structures. Regarding the micro-network topology evolution mechanism, this embodiment applies differentiated mathematical guidance criteria through a decoupling strategy: for surviving active connections, robust and precise pruning is performed using historical second-order Fisher information to ensure the lower bound of model performance; for dormant connections, exploratory connection regeneration is performed using first-order instantaneous gradient magnitude as a probe to quickly fit the new road condition data distribution faced by vehicles. This decoupled evolutionary architecture not only endows sparse networks with excellent dynamic optimization and adaptive feature extraction capabilities, but also stably compresses the amount of vehicle-to-cloud bidirectional communication data by more than 80% throughout the global federated training lifecycle, greatly reducing the traffic bandwidth load and overall operating cost of large-scale distributed collaborative training based on vehicle networks (4G / 5G).

[0047] This invention breaks through the traditional low-order numerical heuristic criteria and designs a state parasitic second-order information extraction mechanism based on the Fisher information matrix. This mechanism provides a strict theoretical protection boundary for weight parameters that are in a local dormant state but carry global key features (such as extreme long-tail road conditions and Corner Cases), effectively blocking the path of catastrophic forgetting, thereby significantly improving the generalization ability and safety of the autonomous driving perception model in harsh long-tail environments. By seamlessly reusing the acquisition of second-order information into the exponential moving average state variables maintained inside the adaptive optimizer, unbiased estimation of the local true curvature of parameters is achieved under the stringent conditions of not increasing the memory usage of the on-board chip and not introducing any additional backpropagation floating-point operations. This gives physical feasibility to execute complex second-order optimization algorithms at the vehicle edge nodes and ensures that the background federated fine-tuning task never interferes with the real-time autonomous driving main perception process, completely avoiding the risk of memory overflow. This invention uses second-order historical Fisher information for robust pruning of active connections and first-order instantaneous gradients for aggressive regeneration of dormant connections, endowing sparse topologies with powerful dynamic optimization and adaptive capabilities, accelerating the overall convergence speed and stability of federated networks, and compressing the amount of vehicle-to-cloud bidirectional communication data by more than 80% during the global federated training lifecycle, greatly reducing the traffic bandwidth load and overall operating cost of large-scale distributed collaborative training based on vehicle networks (4G / 5G).

[0048] Secondly, embodiments of the present invention also provide a FedDST federated learning optimization system for heterogeneous data environments, which may include: The car manufacturer's cloud center server is used to generate the initial global sparse parameters of the model according to the preset initialization strategy, and then distribute the initial global sparse parameters of the model to all car edge nodes participating in this round of training. The vehicle edge node is used to perform multiple rounds of local iterative training on a heterogeneous road condition privacy dataset collected during local driving, based on the initial global sparse parameters it receives. Through a pre-built state-parasitic second-order information extraction mechanism, it obtains an unbiased, persistent approximation of the local diagonal Fisher information matrix of the vehicle edge node. When the local iterative training reaches a preset topology evolution frequency, it calculates the second-order sensitivity survival score of each active connection in the current network based on the unbiased, persistent approximation and weight values. Based on the second-order sensitivity survival score, the masks corresponding to a preset dormant proportion of active connections are set to zero, becoming dormant connections. The absolute value of the instantaneous loss gradient of the currently masked dormant connection with respect to the current batch of input data is calculated. Based on the absolute value of the instantaneous loss gradient, the masks of the preset proportion of dormant connections are reactivated and their weights are initialized to obtain an updated local mask. The updated local mask and its corresponding non-zero parameter weights are encrypted and uploaded to the vehicle manufacturer's cloud central server. The cloud-based central server of the car manufacturer is also used to execute the sparse aggregation protocol, generate new global sparse parameters and distribute them for the next round of iterative training until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network.

[0049] Thirdly, embodiments of the present invention also provide an electronic device, such as... Figure 3 As shown, it includes a processor 001, a communication interface 002, a memory 003, and a communication bus 004, wherein the processor 001, the communication interface 002, and the memory 003 communicate with each other through the communication bus 004. The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements any of the FedFish federated learning methods for long-tail scenarios of autonomous driving in the Internet of Vehicles provided in the first aspect of the present invention.

[0050] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0051] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0052] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0053] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0054] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0055] Fourthly, corresponding to the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided in the first aspect, this embodiment of the invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements any of the steps of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles provided in the first aspect of this invention.

[0056] For system / electronic device / storage medium embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0057] It should be noted that the system, electronic device and storage medium in the embodiments of the present invention are respectively the system, electronic device and storage medium applying the above method. Therefore, all embodiments of the above method are applicable to the system, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0058] It should be noted that, in the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0059] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A FedFish federated learning method for long-tail scenarios of vehicle-to-everything (V2X) autonomous driving, characterized in that, include: S1 utilizes the vehicle manufacturer's cloud central server to build a deep visual neural network model for autonomous driving perception, generates the initial global sparse parameters of the model according to the preset initialization strategy, and distributes the initial global sparse parameters of the model to all vehicle edge nodes participating in this round of training. S2, based on the initial global sparse parameters of the model received by the vehicle edge node, uses the heterogeneous road condition privacy dataset collected by the vehicle edge node during local driving to perform multiple rounds of local iterative training, and obtains the unbiased continuous approximation of the local diagonal Fisher information matrix of the vehicle edge node through a pre-built state parasitic second-order information extraction mechanism. S3, when the local iterative training reaches the preset topology evolution frequency, calculate the second-order sensitivity survival score of each active connection in the current network based on the unbiased continuous approximation and weight value. Based on the second-order sensitivity survival score, the mask corresponding to the active connection with the preset dormancy ratio is set to zero and used as the dormant connection; S4, calculate the absolute value of the instantaneous loss gradient of the currently masked dormant connection with respect to the current batch of input data; based on the absolute value of the instantaneous loss gradient, reactivate the mask of the dormant connection with the preset connection ratio and initialize the weights to obtain the updated local mask; S5, encrypt and upload the updated local mask and the corresponding non-zero parameter weights to the vehicle manufacturer's cloud center server; use the vehicle manufacturer's cloud center server to execute the sparse aggregation protocol, generate new global sparse parameters and distribute them, iteratively execute steps S2~S5 until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network. The number of active connections with a preset sleep ratio is the same as the number of sleep connections with a preset connection ratio.

2. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... The initial global sparse parameters include: The global initial model parameters and their mask topology after sparsification.

3. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... In the second-order information extraction mechanism, when the vehicle edge node performs local mini-batch backpropagation update based on the adaptive optimizer, it directly intercepts and extracts the gradient squared exponential moving average vector maintained by the optimizer in the internal working area, as an unbiased continuous approximation of the local diagonal Fisher information matrix F of the vehicle edge node.

4. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... The expression for the unbiased persistence approximation is as follows: ； in, For the first t Unbiased continuity approximation for each round, The second-order momentum decay rate of the optimizer. for t The squared exponential moving average vector of the gradient after -1 rounds. This is the first-order cross-entropy loss gradient calculated from the current batch of road condition data.

5. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... The second-order sensitivity survival score of each active connection is the product of the unbiased persistence approximation corresponding to each active connection and the square of the absolute value of its weight.

6. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... The step of setting the mask corresponding to the active connections with a preset dormant ratio to zero based on the second-order sensitivity survival score, and using these as dormant connections, includes: All active connections are sorted in ascending order based on their second-order sensitivity survival scores. The masks of active connections with low scores are set to zero according to a preset ratio, and these connections are then designated as dormant connections.

7. The FedFish federated learning method for long-tail scenarios of autonomous driving in vehicle-to-everything (V2X) networks, as described in claim 1, is characterized in that... The step of reactivating the mask of dormant connections with a preset connection ratio and initializing the weights based on the absolute value of the instantaneous loss gradient to obtain the updated local mask includes: Dormant connections are sorted from largest to smallest based on the absolute value of the instantaneous loss gradient. The mask corresponding to the dormant connection with the largest absolute gradient value is set to 1 according to the preset connection ratio, and the weights are initialized and activated with zero value or preset Gaussian noise.

8. A FedFish federated learning system for long-tail scenarios of autonomous driving in the Internet of Vehicles, characterized in that, include: The car manufacturer's cloud center server is used to generate the initial global sparse parameters of the model according to the preset initialization strategy, and then distribute the initial global sparse parameters of the model to all car edge nodes participating in this round of training. The vehicle edge node is used to perform multiple rounds of local iterative training on the heterogeneous road condition privacy dataset collected during local driving based on the initial global sparse parameters it receives. Through a pre-built state parasitic second-order information extraction mechanism, it obtains the unbiased continuous approximation of the local diagonal Fisher information matrix of the vehicle edge node. When the local iterative training reaches the preset topology evolution frequency, the second-order sensitivity survival score of each active connection in the current network is calculated based on the unbiased persistence approximation and weight values. Based on the second-order sensitivity survival score, the masks corresponding to the active connections with a preset dormant ratio are set to zero and used as dormant connections. The absolute value of the instantaneous loss gradient of the dormant connection with the current input data batch is calculated. Based on the absolute value of the instantaneous loss gradient, the masks of the dormant connections with a preset connection ratio are reactivated and the weights are initialized to obtain the updated local mask. The updated local mask and the corresponding non-zero parameter weights are encrypted and uploaded to the vehicle manufacturer's cloud center server. The cloud-based central server of the car manufacturer is also used to execute a sparse aggregation protocol, generate new global sparse parameters and distribute them for the next round of iterative training until the model meets the preset iteration termination condition, so that the model can achieve smooth convergence in the heterogeneous network.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the FedFish federated learning method for long-tail scenarios of autonomous driving in the Internet of Vehicles as described in any one of claims 1-7.

Images