A Personalized Traffic Sign Recognition Method Based on Federated Learning

Abstract

This invention discloses a personalized traffic sign discrimination method based on federated learning, comprising: a server initializing model parameters and sending the model to a client; the client receiving the initialized model parameters from the server; preprocessing the data; training the model on the client; the server aggregating the model parameters from the client; the client receiving the aggregated model parameters from the server and loading the aggregated model into its local model; and continuing to train the model until the respective iteration count is reached and the model is output. This invention is a method for traffic sign discrimination in intelligent vehicle networks based on a hybrid model of modified terms of distributed methods such as AIDE and DANE, personalized federated learning, and dynamic calibration terms. This model can consider the computing power of each user and its data distribution, and performs traffic sign discrimination with very low performance while ensuring data privacy and security.

Description

Technical Field

[0001] This invention belongs to the field of image recognition, and in particular relates to a personalized traffic sign discrimination method based on federated learning. Background Technology

[0002] With the advent of the internet boom and the widespread adoption of wireless networks, the number of network edge devices and the amount of data they generate are growing rapidly. Vehicle-to-everything (IoV), as an important scenario within the Internet of Things, is gradually coming into focus. Intel CEO Brian Krzanich once stated, "Data is the new driving force for the future of autonomous driving." Traditional machine learning techniques primarily involve uploading all data to a central server for unified processing. However, with the rapid increase in the amount of data generated by vehicles in connected vehicle scenarios, this traditional machine learning approach has three drawbacks: First, user privacy data leakage: Traditional machine learning uploads data to a central server for unified processing, greatly increasing the risk of vehicle privacy data leakage and posing unpredictable security risks to vehicle owners. Second, uneven computing power across vehicles: Vehicles from different manufacturers are equipped with chips from different manufacturers, such as NVIDIA Drive Orin, Mobileye Eye Q5, and Snapdragon Ride. The different maximum computing power of these chips leads to uneven computing speeds between vehicles. This means that vehicles with sufficient computing power must wait for vehicles with insufficient computing power to finish their calculations before aggregation, severely impacting communication efficiency. Third, high latency transmission: As the amount of local data in vehicles increases, the transmission latency inevitably increases, failing to meet the low latency requirements of connected vehicle scenarios. Therefore, compared to traditional learning methods, federated learning is more suitable for connected vehicle environments with large data volumes.

[0003] Federated learning, a concept first proposed by Google in 2016, achieves data sharing and collaborative modeling while ensuring data privacy, security, and legal compliance. Its core idea is that when multiple clients participate in model training, the model can be jointly trained solely through interaction with the server using intermediate parameters, without requiring client-side data transfer. Client data can remain locally on their machines. This approach achieves a balance between data privacy protection and data sharing analysis, a "data usable but not visible" data application model. McMahan designed the FedAvg framework and demonstrated the feasibility of federated learning through experiments. FedAvg effectively prevents data leakage and protects data privacy and security. While FedAvg is the original framework for federated learning and can handle some mildly non-independent and identically distributed data, communication overhead and heterogeneity issues still need improvement. This is especially evident when dealing with clients whose data is not independently and identically distributed and who have different computing power systems. Various solutions have been proposed to address these issues, such as secure data coordination and joint computation procedures based on Alternating Direction Multipliers (ADMM). However, these methods mostly only consider one aspect and have high computational and communication costs. To address this challenge, a feasible approach is to design a personalized federated learning algorithm. This offers a significant lightweight advantage compared to other methods. Currently, personalized federated learning methods mainly fall into two categories: (1) optimizing the local model to more closely approximate the global model; and (2) each client autonomously learns a personalized model locally and retains its unique model structure. However, both of these approaches focus on unidirectional optimization and fail to optimize bidirectionally, which often leads to poor performance in certain specific scenarios.

[0004] Meanwhile, existing network models are highly complex. Because the performance of in-vehicle computing systems is typically low, applying complex network models to traffic sign recognition on these systems consumes significant time and resources, impacting other functions. Furthermore, the data distribution and volume vary considerably among users. The previous assumption that data from each client is independent and identically distributed is unrealistic. Differences in data distribution and volume negatively affect model aggregation and severely waste the computing power of in-vehicle systems. The computing power of different in-vehicle systems also varies. Therefore, uniformly calculating rounds leads to high communication latency, which is inconsistent with the dynamic nature of vehicle-to-everything (V2X) scenarios. Thus, it is necessary to consider how to balance the computational and model performance of clients with varying computing power. Summary of the Invention

[0005] Purpose of the Invention: To address the problems existing in the background technology, the purpose of this invention is to provide a personalized traffic sign discrimination method based on federated learning. This method can reduce the computational load on the client side, lower communication pressure, and improve communication efficiency while ensuring client data privacy and security.

[0006] Technical solution: The present invention provides a personalized traffic sign discrimination method based on federated learning, characterized by the following steps:

[0007] Step 1: Set the initial parameters of the pre-trained model on the server and send the initial parameters to each client;

[0008] Step 2: Receive initial model parameters from the server on each client, build the model using the initial model parameters, and optimize the model based on the federated learning algorithm;

[0009] Step 3: Obtain personalized traffic sign images and preprocess them on each client.

[0010] Step 4: Using the preprocessed personalized traffic sign image as input and the image category as output, train the model on the client side. Set the calculation rounds based on the client's computing power level dynamic calibration item, add correction terms during calculation, update the correction terms using gradient information, and obtain local model parameters.

[0011] Step 5: The server aggregates the local model parameters from each client and then sends them to each client.

[0012] Step 6: Each client receives the aggregated local model parameters from the server and loads them into its local model;

[0013] Step 7: Update the local model parameters based on the gradient information and correction term, and determine whether the training has reached the expected goal or the maximum number of rounds. If not, return to step 4 to continue training the model until the number of iterations is reached and the model is output to obtain the trained model. Apply the trained model to identify personalized traffic signs.

[0014] Furthermore, step 2 specifically includes the following steps:

[0015] Step 2.1: Build the model using the LeNet5 convolutional neural network;

[0016] Step 2.2: Initialize the architecture of the LeNet5 convolutional neural network model, set the loss function and hyperparameters, select the personalized federated learning algorithm as the optimizer, and set the learning rate to 0.005.

[0017] Furthermore, the hyperparameter settings are as follows: the local batch size is set to 32, the maximum number of local iterations is 20, the local model relaxation is 0.1, the global iteration count is 10, the number of clients is 100, and the client sampling ratio is 0.2.

[0018] Furthermore, the goal of the personalized federated learning algorithm is to obtain a model that minimizes the loss, with the mathematical form shown in equations (1) and (2):

[0019]

[0020]

[0021] Where F(w) represents the loss of the server model, f i (w) is the client-side local loss function. In the server model, N is the number of clients, and D... n The server aggregates models based on the proportion of data from each client, taking into account the current data volume of the client. On the client's local machine, a correction term is added when calculating the loss function. Its specific description is shown in formula (3):

[0022]

[0023] The correction term uses iterative recording of past gradient information to ensure that the model does not deviate from the initial value during updates. This provides the current gradient information for the current client. This refers to the gradient information from the previous round for the current client. This refers to information from previous batches.

[0024] Furthermore, step 3 specifically includes the following steps:

[0025] Step 3.1: The client scales the width and height of traffic sign images of different sizes to 64×64, and then crops the 64×64 images from the center to a size of 32×32, ultimately ensuring that all images are 32×32 in size;

[0026] Step 3.2: The client performs standard normalization on the image. The standard normalization is shown in formula (4):

[0027]

[0028] The input sample data is subtracted from the mean data, and the result is divided by the standard deviation to obtain the standardized result.

[0029] The image is normalized channel by channel, so that the mean becomes 0 and the standard deviation becomes 1, so that each element of the image is in the range of [-1, 1].

[0030] Furthermore, in step 3, the personalized traffic sign images are obtained by collecting data from the BelgiumTS and GTSRB datasets.

[0031] Furthermore, step 4 specifically includes the following steps:

[0032] Step 4.1: Divide each client into four levels based on its local computing power, with four computing power levels from l0 to l3, decreasing in power progressively. The client dynamically adjusts its local iteration count according to its computing power level, with a maximum of 20 iterations. Then, each client receives a model from the server, pre-trains it locally, and initializes the correction terms using the training results.

[0033] Step 4.2: Following the FedAvg algorithm, during local loop training, the maximum number of local training rounds is set by the computing power level l. When the expected result or the maximum number of rounds is reached on the client side, the training will automatically end. In each round of training, samples are randomly selected and the gradient information of the samples is calculated. After obtaining the gradient information, the gradient is pruned and the correction term is updated using the gradient information.

[0034] Step 4.3: The client sends local model parameters to the server.

[0035] Furthermore, in step 5, the server aggregates the local model parameters of each client, and the aggregation method is as shown in formula (1):

[0036]

[0037] Among them, D n D is the total number of datasets held by each client, and N is the total number of datasets held by all clients. After aggregation, the datasets are distributed to the clients participating in the next round of training.

[0038] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0039] (1) The present invention uses a modified term combining the two distributed methods AIDE and DANE with the LENET5 convolutional neural network.

[0040] Due to the low computing power of current in-vehicle systems and the high complexity of existing network models, complex network models and a large amount of computation consume a lot of the performance and time of the in-vehicle system. However, by using correction terms and LENET5 convolutional neural networks, better accuracy can be achieved in the same number of computation rounds using a relatively simple seven-layer network structure, thereby reducing the computation of each client and saving model update time.

[0041] (2) The present invention uses personalized federated learning with dynamic adjustment items.

[0042] The varying onboard computing power systems of different users lead to performance differences. Most existing algorithms fail to account for this, meaning that for the same computational needs, clients with lower performance will require more time to update, increasing overall time consumption. Therefore, incorporating dynamic adjustment into personalized federated learning allows for dynamic setting of local computation rounds based on each client's computing power. This reduces the computational burden on clients with insufficient computing power with minimal impact on accuracy, while improving communication efficiency between clients and the server.

[0043] (3) This invention adds a new correction term to the update process of each client in federated learning. This correction term combines the distributed methods DANE and AIDE. By using this correction term, clients can refer to the global model's update direction and learn from the global model's experience during local updates, ensuring that each client's local updates do not deviate from the global model's direction. This reduces the disturbance caused by different data distributions of clients to model aggregation. In addition, a dynamic adjustment term is needed to allow participating clients to dynamically adjust their local computation rounds based on their own computing power, enabling each client model to perform corresponding local computations according to its local computing power. Through these two methods, the computational load on clients can be reduced, communication pressure lowered, and communication efficiency improved while ensuring client data privacy and security. Finally, a traffic sign discrimination model adapted to the computing power and data distribution of each user is obtained. Attached Figure Description

[0044] Figure 1 Overall structure of traffic sign identification methods.

[0045] Figure 2 Local model architecture.

[0046] Figure 3 Client drift diagram.

[0047] Figure 4 Diagram of the correction items.

[0048] Figure 5 Client-side personalized federated learning algorithm. Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0050] This section demonstrates the entire process of a personalized traffic sign discrimination system based on federated learning.

[0051] Step 1. The server initializes and sets the parameters of a pre-trained initial model according to the task requirements, without any personalized effects.

[0052] Step 2. The client receives the initialization model parameters from the server.

[0053] 2.1 See the local model architecture. Figure 2 The local model built uses the LeNet-5 convolutional neural network. The basic structure of LeNet-5 consists of 7 layers (excluding the input layer): 2 convolutional layers, 2 downsampling layers (pooling layers), 2 fully connected layers, and an output layer. 1. The input layer receives a 32*32 image of a handwritten digit, including grayscale values ​​(0-255). 2. The convolutional layer C1 consists of 6 convolutional kernels, each 5*5 in size, with a stride of 1 and zero padding. Therefore, each kernel produces a 28*28 feature map (6 output channels). 3. The subsampling layer S2 uses max-pooling, with each window being 2*2 in size and a stride of 2. Therefore, each pooling operation selects the maximum value from four adjacent feature maps, producing a 14*14 feature map (6 output channels). This reduces the size of the feature maps, improves computational efficiency, and maintains some invariance to slight positional changes. 4. Convolutional layer C3 consists of 16 convolutional kernels, each 5x5 in size, with a stride of 1 and zero padding. Therefore, each kernel produces a 10x10 feature map (16 output channels). 5. Subsampling layer S4 uses max pooling, with each window being 2x2 in size and a stride of 2. Therefore, each pooling operation selects the maximum value from four adjacent feature maps, producing a 5x5 feature map (16 output channels). 6. Fully connected layer C5 stretches each 5x5 feature map into a vector of length 400 and connects them through a fully connected layer with 120 neurons. 7. Fully connected layer F6 connects the 120 neurons to 84 neurons. 8. The output layer consists of 10 neurons, each corresponding to a digit from 0 to 9, and outputs the final classification result.

[0054] 2.2 First, the initial model parameters received from the server are loaded into the local model. Then, the main hyperparameter settings are configured. We set the local batch size to 32, the maximum number of local iterations to 20, the local model relaxation to 0.1, the global iteration count to 10, the number of clients to 100, the client sampling ratio to 0.2, the optimizer to the personalized federated learning algorithm, and the learning rate to 0.005.

[0055] The goal of federated learning is to obtain a model that minimizes the loss, as shown in Equations 1 and 2, where F(w) represents the loss of the server model, f i (w) is the client-side local loss function. In the server model, N is the number of clients, and D... n This refers to the current amount of data on each client. The server aggregates models based on the proportion of data from each client. On the client's local machine, a correction term is added when calculating the loss function. Its specific description is given in Formula 3. The correction term ensures that the model update does not deviate from the initial value by iteratively recording past gradient information. This provides the current gradient information for the current client. This refers to the gradient information from the previous round for the current client. It records information about previous batches. (Through...) Learning from past gradient information ensures that the overall performance is not reduced due to individual data, and reduces the impact of non-independent and identically distributed systems while tolerating system heterogeneity.

[0056]

[0057]

[0058]

[0059] Step 3. Data Preprocessing

[0060] 3.1 The client first needs to scale the width and height of traffic sign images of different sizes to 64×64, and then crop the 64×64 images from the center to a size of 32×32, ultimately ensuring that all images are 32×32 in size.

[0061] 3.2 The client performs standard normalization on the image. Standard normalization ensures good network convergence. Before the relative importance of each dimension is known, standard normalization makes the distribution of each dimension of the input similar, allowing us to treat each dimension "equally" during network training (i.e., setting the same learning rate, regularization coefficient, weight initialization, and activation function). Here, standard normalization is performed as shown in Equation 4, normalizing the image channel by channel (mean becomes 0, standard deviation becomes 1). This makes each element of the image fall within the range [-1, 1].

[0062]

[0063] Step 4. Train the model on the client side.

[0064] 4.1 First, each client is divided into four levels based on its local computing power, ranging from l0 to l3, with the computing power decreasing in each level. The client dynamically adjusts its local iteration count according to its computing power level, with a maximum of 20 iterations. Then, each client receives a model from the server, pre-trains it locally, and initializes the correction terms using the training results.

[0065] 4.3 Following the FedAvg algorithm steps, during local loop training, the maximum number of local training rounds is set by the computing power level l. Training automatically ends when the expected result or the maximum number of rounds is reached on the client side. In each training round, samples are randomly selected and their gradient information is calculated. After obtaining the gradient information, the gradient is pruned, and the correction term is updated using the gradient information. Then, the client will update the correction term based on the gradient information and the correction term. This updates the client-side model and determines whether the expected goal or maximum number of rounds has been achieved.

[0066] 4.4 Client i sends local model parameters Give it to the server.

[0067] Step 5. The server aggregates the local model parameters from client i. The aggregation method is shown in Formula 1. Where D... n This represents the total number of datasets held by each client, where D is the sum of the datasets held by all clients. N is the total number of clients currently participating in model aggregation. After aggregation, the datasets are distributed to the clients participating in the next round of training.

[0068] Step 6. Each participating client receives the aggregated model from the server, loads it into its local model, and prepares it for training.

[0069] Step 7. Return to step 4 and continue training the model until the required number of iterations is reached and the model is output.

[0070] After completing model training, each vehicle client calls the trained model to perform a discrimination operation when encountering road sections with traffic sign images. Based on the model output, it provides the user with relevant traffic sign information, mainly the category and meaning of the traffic signs.

Claims

1. A personalized traffic sign discrimination method based on federated learning, characterized in that, Includes the following steps: Step 1: Set the initial parameters of the pre-trained model on the server and send the initial parameters to each client; Step 2: Receive initial model parameters from the server on each client, build the model using the initial model parameters, and optimize the model based on the federated learning algorithm; Step 2 specifically includes the following steps: Step 2.1: Build the model using the LeNet5 convolutional neural network; Step 2.2: Initialize the architecture of the LeNet5 convolutional neural network model, set the loss function and hyperparameters, select the personalized federated learning algorithm as the optimizer, and set the learning rate to 0.005; The goal of the personalized federated learning algorithm is to obtain a model that minimizes the loss, and its mathematical form is as shown in formulas (1) and (2): （1） （2） Among them, This represents the loss of the server model. It is the client-side loss function. In the server model, N is the number of clients. The server aggregates models based on the proportion of data from each client, taking into account the current data volume of the client. On the client's local machine, a correction term is added when calculating the loss function. For a detailed description, see formula (3): （3） The correction term ensures that the model does not deviate from the initial value during updates by iteratively recording past gradient information; where... This provides the current gradient information for the current client. This refers to the gradient information from the previous round for the current client. This refers to information from previous batches; Step 3: Obtain personalized traffic sign images and preprocess them on each client. Step 4: Using the preprocessed personalized traffic sign image as input and the image category as output, train the model on the client side. Set the calculation rounds based on the client's computing power level dynamic calibration item, add correction terms during calculation, update the correction terms using gradient information, and obtain local model parameters. Step 5: The server aggregates the local model parameters from each client and then sends them to each client. Step 6: Each client receives the aggregated local model parameters from the server and loads them into its local model; Step 7: Update the local model parameters based on the gradient information and correction term, and determine whether the training has reached the expected goal or the maximum number of rounds. If not, return to step 4 to continue training the model until the number of iterations is reached and the model is output to obtain the trained model. Apply the trained model to identify personalized traffic signs.

2. The personalized traffic sign discrimination method based on federated learning according to claim 1, characterized in that, The hyperparameter settings are as follows: local batch size is set to 32, maximum local iterations are set to 20, local model relaxation is set to 0.1, global iterations are set to 10, number of clients is set to 100, and client sampling ratio is set to 0.

2.

3. The personalized traffic sign discrimination method based on federated learning according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: The client scales the width and height of traffic sign images of different sizes to 64×64, and then crops the 64×64 images from the center to a size of 32×32, ultimately ensuring that all images are 32×32 in size; Step 3.2: The client performs standard normalization on the image. The standard normalization is shown in formula (4): （4） The input sample data is subtracted from the mean data, and the result is divided by the standard deviation to obtain the standard normalized result. The image is normalized channel by channel, so that the mean becomes 0 and the standard deviation becomes 1, so that each element of the image is in the range of [-1, 1].

4. The personalized traffic sign discrimination method based on federated learning according to claim 1, characterized in that, In step 3, the personalized traffic sign images are obtained by collecting data from the BelgiumTS and GTSRB datasets.

5. The personalized traffic sign discrimination method based on federated learning according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Divide each client into four levels according to its local computing power, with four computing power levels from l0 to l3, and the computing power decreases in that order. The client dynamically adjusts its local iteration number according to the computing power level, with a maximum of no more than 20 iterations. Then, each client receives a model from the server, performs pre-training locally, and initializes the correction term using the training results. Step 4.2: Following the FedAvg algorithm, during local loop training, the maximum number of local training rounds is set by the computing power level l. When the expected result or the maximum number of rounds is reached on the client side, the training will automatically end. In each round of training, samples are randomly selected and the gradient information of the samples is calculated. After obtaining the gradient information, the gradient is pruned and the correction term is updated using the gradient information. Step 4.3: The client sends local model parameters to the server.

6. The personalized traffic sign discrimination method based on federated learning according to claim 1, characterized in that, In step 5, the server aggregates the local model parameters of each client, and the aggregation method is as shown in formula (1): （1） in, It is the total number of datasets held by each client. The sum of the datasets held by all clients. This represents the total number of clients currently participating in model aggregation, which will then be distributed to clients participating in the next round of training.

Images