Distributed Model Training Method and Vehicle Control Method Based on Exponential Moving Average

Abstract

The present application provides a distributed model training method and a vehicle control method based on exponential moving average, belonging to the field of artificial intelligence technology. Each vehicle terminal obtains the global street scene understanding model obtained by the server in the previous training round, updates the global street scene understanding model based on the local street scene image dataset to obtain a local street scene understanding model, and sends the local street scene understanding model and the number of subsamples of the street scene image dataset to the server. The server determines the total number of samples according to the number of subsamples sent by each vehicle terminal, calculates the aggregation weight corresponding to each vehicle terminal according to the number of subsamples and the total number of samples, and aggregates the local street scene understanding models corresponding to each vehicle terminal according to each aggregation weight to obtain an initial aggregation model. The server fuses the previous global street scene understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street scene understanding model, which can improve the accuracy of street scene semantic understanding.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a distributed model training method and a vehicle control method based on exponential moving average. Background Art

[0002] Under the federated learning framework, in each training round, the vehicle terminal uses the locally collected street view images to train the local partial model, and sends the local partial model to the server to aggregate the global model. The server will distribute the aggregated global model to all vehicle terminals participating in the training to overwrite the local partial models of the vehicle terminals. However, the street view images collected by the vehicle during driving will change dynamically with driving environments such as weather patterns, traffic rules, road infrastructure, etc. This training method enables the local model to only learn the semantics of the local street view images collected in the current training round, and the local model cannot accurately understand the semantics of various street view images collected during the entire dynamic driving process of the vehicle, resulting in a low accuracy of street view semantic understanding. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a distributed model training method and a vehicle control method based on exponential moving average, aiming to improve the accuracy of street view semantic understanding.

[0004] To achieve the above object, the first aspect of the embodiments of this application proposes a distributed model training method based on exponential moving average, and the method includes:

[0005] Each vehicle terminal obtains the global street view understanding model obtained by the server in the previous training round, updates the model parameters of the global street view understanding model based on the locally collected street view image dataset to obtain a local street view understanding model, and sends the local street view understanding model and the number of subsamples of the street view image dataset to the server; wherein, the global street view understanding model is obtained by the server based on the historical global street view understanding model trained locally and the historical local street view understanding models sent by each vehicle terminal in the previous training round;

[0006] The server determines the total number of samples according to the number of subsamples sent by each vehicle terminal, calculates the aggregation weight corresponding to each vehicle terminal according to the number of subsamples and the total number of samples, and aggregates the local street view understanding models corresponding to each vehicle terminal according to each aggregation weight to obtain an initial aggregation model;

[0007] The server fuses the previous global street view understanding model and the initial aggregation model based on an exponential moving average factor to obtain a target global street view understanding model; wherein, the exponential moving average factor is used to control the fusion proportion of the previous global street view understanding model during fusion, the exponential moving average factor includes a dynamic exponential factor, and the dynamic exponential factor is inversely proportional to the fusion weight.

[0008] In some embodiments, the street view image dataset includes multiple street view images and the reference street view semantics of each street view image. Updating the model parameters of the global street view understanding model based on the locally collected street view image dataset to obtain a local street view understanding model includes:

[0009] Performing street view semantic understanding on each street view image through the global street view understanding model to obtain the predicted street view semantics of each street view image;

[0010] Calculating a target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image;

[0011] Performing iterative update of the model parameters of the global street view understanding model a preset number of times according to the target semantic loss to obtain the local street view understanding model.

[0012] In some embodiments, calculating the target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image includes:

[0013] Calculating a negative entropy regularization loss according to the predicted street view semantics of each street view image;

[0014] Calculating a cross-entropy loss according to the reference street view semantics and the corresponding predicted street view semantics of each street view image;

[0015] Performing loss summation on the negative entropy regularization loss and the cross-entropy loss to obtain the target semantic loss.

[0016] In some embodiments, before the server fuses the previous global street view understanding model and the initial aggregation model based on an exponential moving average factor to obtain a target global street view understanding model, the distributed model training method based on exponential moving average further includes:

[0017] Obtaining a dynamic exponential factor;

[0018] Calculating the ratio between a preset constant and the dynamic exponential factor to obtain the exponential moving average factor.

[0019] In some embodiments, the server fuses the previous global street view understanding model and the initial aggregation model based on an exponential moving average factor to obtain a target global street view understanding model, including:

[0020] Calculate the aggregation factor of the initial aggregation model according to the exponential moving average factor; wherein, the sum of the exponential moving average factor and the aggregation factor is 1;

[0021] Multiply the exponential moving average factor by the model parameters of the previous global street view understanding model to obtain a first candidate model;

[0022] Multiply the aggregation factor by the model parameters of the initial aggregation model to obtain a second candidate model;

[0023] Add the model parameters of the first candidate model and the model parameters of the second candidate model to obtain the target global street view understanding model.

[0024] In some embodiments, after the server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain the target global street view understanding model, the method further includes:

[0025] Obtain the communication rate of each vehicle terminal, and determine the communication weight of each vehicle terminal according to the communication rate of each vehicle terminal;

[0026] Calculate the model similarity between the target global street view understanding model and the current local street view understanding model of each vehicle terminal;

[0027] Determine the vehicle weight of each vehicle terminal according to the communication weight, the aggregation weight, and the model similarity;

[0028] Screen each vehicle terminal according to the vehicle weight to obtain a target vehicle terminal;

[0029] Send the target global street view understanding model to the target vehicle terminal.

[0030] To achieve the above object, a second aspect of the embodiments of the present application proposes a vehicle control method, the method includes:

[0031] Obtain the target global street view understanding model sent by the server; wherein, the target global street view understanding model is trained according to the distributed model training method based on exponential moving average described in the first aspect above;

[0032] Obtain the target street view image collected locally by the target vehicle;

[0033] Perform street view semantic understanding on the target street view image through the target global street view understanding model to obtain the target street view semantics;

[0034] Control the driving of the target vehicle according to the target street view semantics.

[0035] To achieve the above object, a third aspect of the embodiments of the present application provides a vehicle control device, the device includes:

[0036] A model acquisition module, configured to acquire a target global street scene understanding model sent by a server; wherein, the target global street scene understanding model is trained according to the distributed model training method based on exponential moving average described in the first aspect above;

[0037] An image acquisition module, configured to acquire a target street scene image collected locally by a target vehicle;

[0038] A street scene understanding module, configured to perform street scene semantic understanding on the target street scene image through the target global street scene understanding model to obtain a target street scene semantics;

[0039] A control module, configured to control the driving of the target vehicle according to the target street scene semantics.

[0040] To achieve the above object, a fourth aspect of the embodiments of the present application provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the distributed model training method based on exponential moving average in the first aspect above or the vehicle control method in the second aspect.

[0041] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the distributed model training method based on exponential moving average in the first aspect above or the vehicle control method in the second aspect.

[0042] The distributed model training method, vehicle control method, vehicle control device, electronic device and computer-readable storage medium based on exponential moving average according to the embodiments of the present application. Each vehicle terminal obtains the global street scene understanding model obtained by the server in the previous training round to retain the learned historical basic street scene knowledge based on the previous global street scene understanding model. The local street scene images collected by different vehicle terminals have different feature distributions. To enable the global street scene understanding model to better adapt to the personalized street scene feature distributions of different vehicle terminals and quickly adapt to new scene patterns, the model parameters of the global street scene understanding model are updated based on the locally collected street scene image dataset to obtain a local street scene understanding model. Each vehicle terminal trains based on its local dataset and only learns local street scene features. To enable the model to learn different street scene feature distributions in the new scene pattern, the local street scene understanding model and the number of subsamples of the street scene image dataset are sent to the server to integrate the global street scene knowledge based on the server. The server determines the total number of samples according to the number of subsamples sent by each vehicle terminal, calculates the aggregation weight corresponding to each vehicle terminal according to the number of subsamples and the total number of samples, and aggregates the local street scene understanding models corresponding to each vehicle terminal according to each aggregation weight to obtain an initial aggregation model. The data distributions of different vehicle terminals vary greatly. By adopting a weighted aggregation method, the global model can better adapt to this data heterogeneity. The server fuses the previous global street scene understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street scene understanding model. The exponential moving average factor is used to control the fusion ratio of the previous global street scene understanding model during fusion. The exponential moving average factor includes a dynamic exponential factor, and the dynamic exponential factor is inversely proportional to the fusion weight. The dynamic exponential factor can be adjusted according to actual needs to form the exponential moving average factor, so as to flexibly control the fusion ratio of the previous global street scene understanding model, and avoid the problem that overly rigidly retaining historical basic knowledge will hinder the model's adaptation to new scenes, and overly focusing on the current short-term knowledge in the new scene will erase the learned historical basic knowledge, so as to balance the contributions of the historical model and the current model to street scene semantic understanding, enabling the global model to adapt to the new mode while retaining the fitting ability of the historical model, thereby overcoming the catastrophic forgetting problem and improving the accuracy of street scene semantic understanding. Description of the Drawings

[0043] Figure 1 is a flowchart of the distributed model training method based on exponential moving average provided by the embodiments of the present application;

[0044] Figure 2 is Figure 1 a flowchart of step S110 in

[0045] Figure 3 is Figure 2 a flowchart of step S220 in

[0046] Figure 4 is another flowchart of the distributed model training method based on exponential moving average provided by the embodiments of the present application;

[0047] Figure 5 is Figure 1 the flowchart of step S130 in

[0048] Figure 6 is another flowchart of the distributed model training method based on exponential moving average provided by the embodiments of the present application;

[0049] Figure 7 is the flowchart of the vehicle control method provided by the embodiments of the present application;

[0050] Figure 8 is the structural schematic diagram of the vehicle control device provided by the embodiments of the present application;

[0051] Figure 9 is the hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application.

[0053] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different module division in the device or a different order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Under the federated learning framework, in each training round, the vehicle terminal uses the locally collected street view images to train the local partial model and sends the local partial model to the server for aggregation to obtain the global model. The server will distribute the aggregated global model to all vehicle terminals participating in the training to overwrite the local partial models of the vehicle terminals. However, the street view images collected by the vehicle during driving will change dynamically with the driving environment such as weather patterns, traffic rules, road infrastructure, etc. This training method enables the partial model to only learn the semantics of the locally collected street view images in the current training round. During the entire dynamic driving process of the vehicle, the partial model cannot accurately understand the semantics of the various street view images collected, resulting in a low accuracy of street view semantic understanding.

[0056] Based on this, the embodiments of the present application provide a distributed model training method, a vehicle control method, a vehicle control device, an electronic device, and a computer-readable storage medium based on exponential moving average, aiming to improve the accuracy of street view semantic understanding. The distributed model training method, vehicle control method, vehicle control device, electronic device, and computer-readable storage medium provided by the embodiments of the present application are specifically described through the following embodiments. First, the distributed model training method based on exponential moving average in the embodiments of the present application is described.

[0057] The distributed model training method based on exponential moving average provided by the embodiments of the present application relates to the field of artificial intelligence technology. The distributed model training method based on exponential moving average provided by the embodiments of the present application can be applied to the terminal, can also be applied to the server side, or can be software running on the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the distributed model training method based on exponential moving average, etc., but is not limited to the above forms.

[0058] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0059] Figure 1 FIG. is an alternative flowchart of a distributed model training method based on exponential moving average provided by an embodiment of this application. Figure 1 The method in may include but is not limited to steps S110 to S130.

[0060] Step S110, each vehicle terminal obtains the global street view understanding model obtained by the server in the previous training round, updates the model parameters of the global street view understanding model based on the locally collected street view image dataset to obtain a local street view understanding model, and sends the local street view understanding model and the number of subsamples of the street view image dataset to the server; wherein, the global street view understanding model is obtained by the server fusing the historical global street view understanding model locally trained and the historical local street view understanding models sent by each vehicle terminal in the previous training round;

[0061] Step S120, the server determines the total number of samples according to the number of subsamples sent by each vehicle terminal, calculates the aggregation weight corresponding to each vehicle terminal according to the number of subsamples and the total number of samples, and aggregates the local street view understanding models corresponding to each vehicle terminal according to each aggregation weight to obtain an initial aggregation model;

[0062] Step S130, the server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street view understanding model; wherein, the exponential moving average factor is used to control the fusion proportion of the previous global street view understanding model during fusion, and the exponential moving average factor includes a dynamic exponential factor, and the dynamic exponential factor is inversely proportional to the fusion weight.

[0063] Please refer to Figure 2 , in some embodiments, step S110 may include but is not limited to steps S210 to S230:

[0064] Step S210: Perform street view semantic understanding on each street view image through the global street view understanding model to obtain the predicted street view semantics of each street view image.

[0065] Step S220: Calculate the target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image.

[0066] Step S230: Iteratively update the model parameters of the global street view understanding model a preset number of times according to the target semantic loss to obtain the local street view understanding model.

[0067] In step S210 of some embodiments, the server locally stores the global street view understanding model obtained in the previous training round, determines multiple vehicle terminals participating in model training, and sends the previous global street view understanding model to the multiple vehicle terminals. Each vehicle terminal obtains the previous global street view understanding model sent by the server and updates the model parameters of the global street view understanding model respectively using the street view image dataset collected locally by each vehicle terminal. The street view image dataset includes multiple street view images and the reference street view semantics of each street view image. The street view image is the surrounding environment image collected by the vehicle terminal, and the reference street view semantics are used to identify and classify various elements in the street view image, such as elements like lanes, pedestrians, traffic signs, buildings, etc. The previous global street view understanding model is obtained by fusing the historical global street view understanding model locally trained by the server and the historical local street view understanding models sent by each vehicle terminal in the previous training round. The historical global street view understanding model is the global street view understanding model obtained by the server before the previous training round, and the historical local street view understanding model is the local street view understanding model obtained by the vehicle terminal updating the model parameters of the historical global street view understanding model based on the street view image dataset collected locally in the previous training round. It should be noted that in the first training round, the previous global street view understanding model is obtained by the server randomly initializing the model parameters.

[0068] For each vehicle terminal, each street view image in the street view image dataset collected by it is sequentially input into the global street view understanding model obtained in the previous training round for street view semantic understanding to obtain the predicted street view semantics of each street view image. The predicted street view semantics are the semantic categories predicted by the global street view understanding model for various elements in the street view image, such as lanes, traffic lights, etc.

[0069] In step S220 of some embodiments, in order to evaluate the difference between the predicted value and the true value, calculate the target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image, so as to measure the street view semantic recognition performance of the global street view understanding model on the vehicle terminal's local street view image dataset based on the target semantic loss.

[0070] In step S230 of some embodiments, the vehicle terminal calculates the gradient of the target semantic loss with respect to the global street scene understanding model, determines the model update step size, multiplies the gradient by the model update step size to obtain the parameter update value. The model parameters of the global street scene understanding model are updated according to the parameter update value, and the number of times of model parameter updates is recorded until the number reaches the preset number, thereby obtaining the local street scene understanding model. The formula for model update is expressed as:

[0071] ,

[0072] where, are the model parameters of the global street scene understanding model; is the model update step size; represents the gradient of the target semantic loss L with respect to the model parameters of the global street scene understanding model.

[0073] Due to the differences in driving environments and driving habits, the semantic features of the street scene image datasets collected by each vehicle terminal are also different. Through iterative updates for a preset number of times, the global street scene understanding model can continuously learn the street scene semantic features of different vehicle terminals, thereby improving the semantic understanding accuracy of the global model and realizing the personalization of the local models of each vehicle terminal, and improving the accuracy of autonomous driving for each vehicle terminal. At the same time, each vehicle terminal sends the local model after multiple iterative updates to the server instead of one iterative update, reducing the communication overhead between the vehicle terminal and the server.

[0074] Through the above steps S210 to S230, it can help the global model better learn the unique street scene semantic features of each vehicle terminal, so that each vehicle terminal can achieve personalized autonomous driving.

[0075] Please refer to Figure 3 , in some embodiments, step S220 may include but is not limited to steps S310 to S330:

[0076] Step S310, calculating the negative entropy regularization loss according to the predicted street scene semantics of each street scene image;

[0077] Step S320, calculating the cross-entropy loss according to the reference street scene semantics and the corresponding predicted street scene semantics of each street scene image;

[0078] Step S330, performing loss summation on the negative entropy regularization loss and the cross-entropy loss to obtain the target semantic loss.

[0079] In step S310 of some embodiments, the negative entropy of the street view image is calculated according to the predicted street view semantics of the street view image. The negative entropy is used to measure the uncertainty of the predicted street view semantics. The larger the value of the negative entropy, the smaller the uncertainty of the predicted street view semantics. The negative entropies of each street view image are summed to obtain the negative entropy sum value. The number of street view images in the street view image dataset is determined to obtain the subsample number. The mean value of the negative entropy sum value is calculated according to the subsample number to obtain the negative entropy regularization loss. The calculation formula of the negative entropy regularization loss is expressed as:

[0080] ,

[0081] where H is the negative entropy regularization loss; c represents the vehicle terminal; E represents the mean calculation; K is the number of categories of semantic classes; is the model parameter of the local street view understanding model; x is the street view image; is the street view image dataset; is the probability that the street view image represented by the predicted street view semantics belongs to the k-th semantic class.

[0082] By maximizing the negative entropy regularization loss, the uncertainty of the model prediction result can be reduced, and the overconfident prediction result can be prevented, thereby avoiding model overfitting and improving the generalization ability of the model.

[0083] In step S320 of some embodiments, according to the reference street view semantics of the street view image and the predicted street view semantics of the street view image, the cross entropy of the street view image is calculated. The cross entropies of each street view image are summed to obtain the cross entropy sum value. The mean value of the cross entropy sum value is calculated according to the subsample number to obtain the cross entropy loss. The calculation formula of the cross entropy loss is expressed as:

[0084] ,

[0085] where, is the cross entropy loss; is the street view image dataset; is the subsample number of the street view image dataset; x is the street view image; y is the reference street view semantics of the street view image; is the model parameter of the local street view understanding model; is the predicted probability that the street view image represented by the predicted street view semantics belongs to the k-th semantic class, and the predicted probability is the softmax probability; is the true probability that the street view image represented by the reference street view semantics belongs to the k-th semantic class.

[0086] In step S330 of some embodiments, determine the weight of the negative entropy regularization loss, multiply the weight by the negative entropy regularization loss, and sum the negative of the calculation result of the multiplication operation with the cross-entropy loss to obtain the target semantic loss. The target semantic loss of vehicle terminal c is expressed as:

[0087] ,

[0088] where, is the cross-entropy loss; is the weight parameter; H is the negative entropy regularization loss; represents the multiplication operation.

[0089] In the above steps S310 to S330, the negative entropy regularization loss is used as a regularization term to penalize the complexity of the local model, avoiding temporal overfitting caused by considering historical models, thereby improving the semantic understanding performance of the local model.

[0090] In step S120 of some embodiments, each vehicle terminal participating in the training sends the corresponding local street scene understanding model and the number of subsamples of the street scene image dataset to the server, and the server sums the number of subsamples sent by each vehicle terminal to obtain the total number of samples. The ratio between the number of subsamples corresponding to each vehicle terminal and the total number of samples is used as the aggregation weight corresponding to each vehicle terminal. To enable the model to adapt to the data distributions of different vehicle terminals, the local street scene understanding models corresponding to each vehicle terminal are aggregated according to each aggregation weight to obtain an initial aggregated model. The aggregation process is expressed as:

[0091] ,

[0092] where, are the model parameters of the initial aggregated model; c represents the vehicle terminal; C is the set composed of all vehicle terminals participating in the training; |D| is the total number of samples; is the number of subsamples of the street scene image dataset of vehicle terminal c; are the model parameters of the local street scene understanding model of vehicle terminal c.

[0093] Please refer to Figure 4 , in some embodiments, before step S130, the distributed model training method based on exponential moving average may further include but is not limited to steps S410 to S420:

[0094] Step S410, obtain the dynamic exponential factor;

[0095] Step S420, calculate the ratio between the preset constant and the dynamic exponential factor to obtain the exponential moving average factor.

[0096] In step S410 of some embodiments, the exponential moving average factor is used to control the fusion ratio of the previous global street view understanding model during fusion. The value of the exponential moving average factor ranges from 0 to 1, and there is an inverse relationship between the dynamic exponential factor and the fusion weight. Obtain the dynamic exponential factor, which can be set according to the actual situation. If the fusion ratio of the historical model during model fusion is large, set a smaller dynamic exponential factor; if the fusion ratio of the historical model during model fusion is small, set a larger dynamic exponential factor. The historical model is the previous global street view understanding model.

[0097] The dynamic exponential factor can be set according to the number of training rounds. The dynamic exponential factor is a real number greater than or equal to 1. For convenience of calculation, the dynamic factor can be an integer. As the number of training rounds increases, gradually decrease the dynamic exponential factor according to the decay rate to increase the dependence on the historical model, so that the vehicle terminal can enhance the understanding of various street view semantics by leveraging the learned long-term environmental patterns during dynamic driving. The calculation formula of the dynamic exponential factor is expressed as:

[0098] ,

[0099] where, is the dynamic exponential factor at training round t; is the initial dynamic exponential factor; * represents the multiplication operation; is the decay rate, and its value ranges from (0, 1); represents the t-th power of the decay rate; is the minimum limit value of the dynamic exponential factor.

[0100] The dynamic exponential factor can be set according to the loss. Obtain the target semantic loss of the current training round and the target semantic loss of the previous training round, calculate the ratio of the current target semantic loss and the previous target semantic loss to obtain the loss change rate. The preset change rate threshold ranges from (0, 1), such as 0.8. If the loss change rate is greater than the preset change rate threshold, it indicates that the street view semantics have changed greatly. To avoid over-adapting to the new street view semantics, reduce the dynamic exponential factor. The calculation formula of the dynamic exponential factor is expressed as:

[0101] ,

[0102] where, is the minimum limit value of the dynamic exponential factor; is the exponential factor increment, which is greater than 0; is the dynamic exponential factor at training round t; is the dynamic exponential factor at the previous training round t - 1.

[0103] If the loss change rate is less than or equal to the preset change rate threshold, it indicates that the street view semantic change is small. To adapt to the new street view semantics more quickly, the dynamic exponential factor is increased. The calculation formula of the dynamic exponential factor is expressed as:

[0104] ,

[0105] wherein, is the maximum limit value of the dynamic exponential factor; is the exponential factor increment, which is greater than 0; is the dynamic exponential factor of the training round t; is the dynamic exponential factor of the previous training round t - 1.

[0106] In step S420 of some embodiments, the preset constant is a preset constant, which can be 2. The ratio between the preset constant and the dynamic exponential factor is used as the exponential moving average factor.

[0107] Through the above steps S410 to S420, the exponential moving average factor can be obtained to control the fusion weight of the historical model through the exponential moving average factor.

[0108] Please refer to Figure 5 , in some embodiments, step S130 may include but is not limited to steps S510 to S540:

[0109] Step S510, calculating the aggregation factor of the initial aggregation model according to the exponential moving average factor; wherein, the sum of the exponential moving average factor and the aggregation factor is 1;

[0110] Step S520, multiplying the exponential moving average factor by the model parameters of the previous global street view understanding model to obtain the first candidate model;

[0111] Step S530, multiplying the aggregation factor by the model parameters of the initial aggregation model to obtain the second candidate model;

[0112] Step S540, adding the model parameters of the first candidate model and the model parameters of the second candidate model to obtain the target global street view understanding model.

[0113] In step S510 of some embodiments, too rigidly retaining historical parameters will hinder the model from adapting to new scenarios, and too much focus on recent street view data will erase the historical street view knowledge learned by the model. The embodiments of the present application weigh the contributions of the historical model and the current aggregation model to the understanding of street view semantics through the exponential moving average factor and the aggregation factor. Specifically, subtracting the exponential moving average factor from 1 to obtain the aggregation factor. The aggregation factor is a number greater than 0 and less than 1, and is used to control the fusion weight of the initial aggregation model during fusion.

[0114] In step S520 of some embodiments, multiply the exponential moving average factor β by the model parameters of the previous global street view understanding model to obtain a first candidate model.

[0115] In step S530 of some embodiments, multiply the aggregation factor (1 - β) by the model parameters of the initial aggregation model to obtain a second candidate model.

[0116] In step S540 of some embodiments, add the model parameters of the first candidate model and the model parameters of the second candidate model to obtain a target global street view understanding model. Steps S110 to S130 can be repeatedly executed until the training round reaches a preset training round limit value. If the preset training round limit value is R, the model convergence rate of the target global street view understanding model is , which speeds up the convergence speed of the model.

[0117] Through the above steps S510 to S540, the global model can not only include the semantic understanding ability of the historical model, but also adapt to the recently collected street view images, thereby improving the accuracy of street view semantic understanding.

[0118] Please refer to Figure 6 , in some embodiments, after step S130, the distributed model training method based on exponential moving average may further include but is not limited to steps S610 to S650:

[0119] Step S610, obtain the communication rate of each vehicle terminal, and determine the communication weight of each vehicle terminal according to the communication rate of each vehicle terminal;

[0120] Step S620, calculate the model similarity between the target global street view understanding model and the current local street view understanding model of each vehicle terminal;

[0121] Step S630, determine the vehicle weight of each vehicle terminal according to the communication weight, aggregation weight and model similarity;

[0122] Step S640, screen each vehicle terminal according to the vehicle weight to obtain a target vehicle terminal;

[0123] Step S650, send the target global street view understanding model to the target vehicle terminal.

[0124] ​​In step S610 of some embodiments, the server obtains the communication rate of each vehicle terminal. The communication rate is the rate at which the vehicle terminal transmits data, measured in bits per second. The communication rates of each vehicle terminal are summed to obtain the total communication rate. The ratio of the communication rate corresponding to each vehicle terminal to the total communication rate is calculated to obtain the communication weight of each vehicle terminal. The greater the communication weight, the faster the vehicle terminal transmits data, and the vehicle terminal can update the model quickly. To reduce the communication overhead between the server and the vehicle terminal, the model update frequency of the vehicle terminal can be reduced. The smaller the communication weight, the slower the vehicle terminal transmits data. To ensure that the local model of the vehicle terminal can be updated in a timely manner, the server needs to send the global model to the vehicle terminal.

[0125] In step S620 of some embodiments, by using a unified similarity calculation metric, the similarity between the model parameters of the target global street scene understanding model and the model parameters of the current local street scene understanding model of each vehicle terminal is calculated respectively to obtain the model similarity of each vehicle terminal. The similarity calculation metric can be selected according to the actual situation, such as cosine similarity. The greater the model similarity, the more similar the target global street scene understanding model and the local street scene understanding model are. To reduce the communication overhead between the server and the vehicle terminal, the model update frequency of the vehicle terminal can be reduced.

[0126] In step S630 of some embodiments, the greater the aggregation weight, the greater the role played by the local model of the vehicle terminal in model aggregation. For each vehicle terminal, the communication weight, aggregation weight, and model similarity of the vehicle terminal are added together to obtain the vehicle weight of the vehicle terminal. The greater the vehicle weight, the greater the impact of the vehicle terminal on model training. To reduce the model instability and communication overhead caused by frequent model updates, the model update frequency of the vehicle terminal can be reduced.

[0127] In step S640 of some embodiments, to reduce the communication overhead of the server, the embodiments of the present application comprehensively consider the communication weight, aggregation weight, and model similarity to strictly limit the screening conditions of the vehicle terminal. The vehicle terminals are sorted in ascending order of vehicle weight, and the first K vehicle terminals after sorting are used as the target vehicle terminals, where K is an integer greater than 1 and less than the total number of vehicle terminals.

[0128] In step S650 of some embodiments, the target global street scene understanding model is sent to the target vehicle terminal so that the target vehicle terminal can update the target global street scene understanding model based on the street scene image dataset collected locally with reference to steps S110 to S130.

[0129] Through the above steps S610 to S650, the number of vehicle terminals that the server needs to send the target global street scene understanding model to can be reduced, thereby reducing the communication overhead of the server.

[0130] After the server sends the target global street scene understanding model to the vehicle terminal, in order to make the target global street scene understanding model better adapt to the local driving environment of the vehicle terminal, the distributed model training method based on exponential moving average may further include:

[0131] The vehicle terminal collects a street scene image dataset. To improve the efficiency of street scene semantic understanding, for each street scene image in the recently collected street scene image dataset, calculate the similarity between this street scene image and each historical street scene image in the historical street scene image dataset used in all previous training rounds, and obtain the first image similarity. Select the maximum first image similarity of this street scene image as the second image similarity. If the second image similarity of each street scene image is greater than or equal to the first similarity threshold, it means that the target global street scene understanding model can accurately understand the street scene semantics of the vehicle terminal locally, and the target global street scene understanding model can be directly used to perform street scene semantic understanding on the recently collected street scene image dataset.

[0132] If there is a street scene image whose second image similarity is less than the first similarity threshold, it means that the target global street scene understanding model may not be able to fit the recently collected street scene image dataset. To improve the accuracy of street scene semantic understanding, calculate the similarity between every two street scene images in the street scene image dataset, cluster each street scene image in the street scene image dataset according to the similarity, and obtain multiple clustering clusters. The street scene image dataset contains street scene images of the vehicle terminal in different driving environments such as rainy and foggy weather conditions, different driving sections, etc. The clustering process is used to separate street scene images with large data heterogeneity. To adapt to different driving environments, referring to step S110, the target global street scene understanding model is updated separately through each clustering cluster to obtain the cluster model of each clustering cluster.

[0133] Due to the influence of adverse weather conditions such as rain and fog, the similarity between street view images in the same driving area under different weather conditions is small, resulting in the street view images being divided into different clustering clusters. However, the street view semantics in the same driving area under different weather conditions should be the same, so that the cluster models trained based on the street view images in the same driving area under different weather conditions should be similar. To improve the semantic recognition accuracy of street view images in the same driving area under different weather conditions, calculate the similarity between every two cluster models to obtain the first model similarity, and select the cluster models with the first model similarity greater than or equal to the second similarity threshold as the first cluster models, and the first cluster models have the first cluster identifiers. Calculate the second model similarity between each cluster model and the target global street view understanding model, and select the cluster models with the second model similarity greater than or equal to the second similarity threshold as the second cluster models, and the second cluster models have the second cluster identifiers. Obtain the first cluster models and the second cluster models with the same first cluster identifier and second cluster identifier. Calculate the mean value of the model parameters of the first cluster models, the second cluster models, and the target global street view understanding model. If there are no cluster models with the second model similarity greater than or equal to the second similarity threshold, calculate the mean value of the model parameters of the first cluster models.

[0134] If there are no cluster models with the first model similarity greater than or equal to the second similarity threshold, to improve the adaptability to various driving environments, calculate the number of street view images in each clustering cluster to obtain the first quantity. Sum the first quantities of each clustering cluster to obtain the second quantity. Use the ratio between the first quantity and the second quantity corresponding to each clustering cluster as the weight parameter. According to the weight parameter of each clustering cluster, perform a weighted sum of the model parameters of the cluster models corresponding to the clustering clusters.

[0135] Figure 7 It is an optional flowchart of a vehicle control method provided by an embodiment of the present application. Figure 7 The method in may include but is not limited to steps S710 to S740.

[0136] Step S710, obtain the target global street view understanding model sent by the server; wherein, the target global street view understanding model is trained according to the above-mentioned distributed model training method based on exponential moving average.

[0137] Step S720, obtain the target street view image collected locally by the target vehicle.

[0138] Step S730, perform street view semantic understanding on the target street view image through the target global street view understanding model to obtain the target street view semantics.

[0139] Step S740, control the target vehicle to travel according to the target street view semantics.

[0140] In step S710 of some embodiments, the server sends the target global street view understanding model to the target vehicle.

[0141] In step S720 of some embodiments, an image of the surrounding environment is collected by a shooting device mounted on the target vehicle to obtain a target street view image.

[0142] In step S730 of some embodiments, the target street view image is input into the target global street view understanding model for street view semantic understanding to obtain a target street view semantics. The target street view semantics is used to indicate the semantic categories of various elements in the target street view image.

[0143] In step S740 of some embodiments, the target vehicle outputs a vehicle control instruction according to the target street view semantics and controls the target vehicle to travel according to the vehicle control instruction. The vehicle control instruction can be a vehicle steering instruction, a vehicle acceleration instruction, etc.

[0144] Through the above steps S710 to S740, the target global street view understanding model has the semantic understanding capabilities of the historical model and the current aggregated model, thereby improving the accuracy of street view semantic understanding.

[0145] Taking FedDyn, FedIR, FedProx, and MOOM as baseline models, the balance parameters of FedDyn and FedProx can be 0.005 and 0.01. The target global street view understanding model and the baseline models are respectively tested on the Cityscapes dataset and the CamVid dataset, and the mean intersection over union (mIoU) is used to evaluate the model performance. The mean intersection over union of the target global street view understanding model in the embodiments of the present application is 7.12% higher than that of the baseline model.

[0146] Please refer to Figure 8 , the embodiments of the present application also provide a vehicle control device that can implement the above vehicle control method. The vehicle control device includes:

[0147] A model acquisition module 810, configured to acquire the target global street view understanding model sent by the server; wherein, the target global street view understanding model is trained according to the above distributed model training method based on exponential moving average;

[0148] An image acquisition module 820, configured to acquire a target street view image locally collected by the target vehicle;

[0149] A street view understanding module 830, configured to perform street view semantic understanding on the target street view image through the target global street view understanding model to obtain target street view semantics;

[0150] A control module 840, configured to control the target vehicle to travel according to the target street view semantics.

[0151] The specific implementation manner of the vehicle control device is substantially the same as the specific embodiments of the above vehicle control method, and will not be elaborated herein.

[0152] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned distributed model training method or vehicle control method based on exponential moving average is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0153] Please refer to Figure 9 , Figure 9 which shows the hardware structure of the electronic device in another embodiment. The electronic device includes:

[0154] A processor 910, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0155] A memory 920, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 920 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 920, and are called by the processor 910 to execute the distributed model training method or vehicle control method based on exponential moving average of the embodiments of the present application;

[0156] An input / output interface 930, which is used to implement information input and output;

[0157] A communication interface 940, which is used to implement communication interaction between this device and other devices, and can implement communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0158] A bus 950, which transmits information between various components of the device (such as the processor 910, the memory 920, the input / output interface 930, and the communication interface 940);

[0159] Among them, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are communicatively connected to each other inside the device through the bus 950.

[0160] An embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-described distributed model training method or vehicle control method based on exponential moving average is implemented.

[0161] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0163] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0166] In the description of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one) of the following" or its similar expressions refer to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0169] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, in each embodiment of the present application, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0172] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A distributed model training method based on exponential moving average, characterized in that The method includes: Each vehicle terminal obtains the global street view understanding model obtained by the server in the previous training round, updates the model parameters of the global street view understanding model based on the locally collected street view image dataset to obtain a local street view understanding model, and sends the local street view understanding model and the number of subsamples of the street view image dataset to the server; wherein, the global street view understanding model is obtained by the server based on the historical global street view understanding model trained locally and the historical local street view understanding models sent by each vehicle terminal in the previous training round; The server determines the total number of samples according to the number of subsamples sent by each vehicle terminal, calculates the aggregation weight corresponding to each vehicle terminal according to the number of subsamples and the total number of samples, and aggregates the local street view understanding models corresponding to each vehicle terminal according to each aggregation weight to obtain an initial aggregation model; The server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street view understanding model; wherein, the exponential moving average factor is used to control the fusion ratio of the previous global street view understanding model during fusion, and the exponential moving average factor includes a dynamic exponential factor, and there is an inverse relationship between the dynamic exponential factor and the fusion ratio; The server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street view understanding model, including: Calculating the aggregation factor of the initial aggregation model according to the exponential moving average factor; wherein, the sum of the exponential moving average factor and the aggregation factor is 1; multiplying the exponential moving average factor by the model parameters of the previous global street view understanding model to obtain a first candidate model; multiplying the aggregation factor by the model parameters of the initial aggregation model to obtain a second candidate model; adding the model parameters of the first candidate model and the model parameters of the second candidate model to obtain the target global street view understanding model; After the server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain a target global street view understanding model, the method further includes: Obtaining the communication rate of each vehicle terminal, determining the communication weight of each vehicle terminal according to the communication rate of each vehicle terminal; calculating the model similarity between the target global street view understanding model and the current local street view understanding model of each vehicle terminal; determining the vehicle weight of each vehicle terminal according to the communication weight, the aggregation weight and the model similarity; screening each vehicle terminal according to the vehicle weight to obtain a target vehicle terminal; sending the target global street view understanding model to the target vehicle terminal.

2. The method according to claim 1, characterized in that, The street view image dataset includes a plurality of street view images and the reference street view semantics of each street view image. Updating the model parameters of the global street view understanding model based on the locally collected street view image dataset to obtain a local street view understanding model includes: Performing street view semantic understanding on each street view image through the global street view understanding model to obtain the predicted street view semantics of each street view image; Calculating the target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image; Performing iterative update on the model parameters of the global street view understanding model for a preset number of times according to the target semantic loss to obtain the local street view understanding model.

3. The method according to claim 2, wherein The calculating the target semantic loss according to the reference street view semantics and the predicted street view semantics of each street view image includes: Calculating the negative entropy regularization loss according to the predicted street view semantics of each street view image; Calculating the cross-entropy loss according to the reference street view semantics and the corresponding predicted street view semantics of each street view image; Performing loss summation on the negative entropy regularization loss and the cross-entropy loss to obtain the target semantic loss.

4. The method according to claim 1, characterized in that Before the server fuses the previous global street view understanding model and the initial aggregation model based on the exponential moving average factor to obtain the target global street view understanding model, the method further includes: Obtaining the dynamic exponential factor; Calculating the ratio between the preset constant and the dynamic exponential factor to obtain the exponential moving average factor.

5. A vehicle control method, characterized in that, The method includes: Obtaining the target global street view understanding model sent by the server; wherein, the target global street view understanding model is trained according to the distributed model training method based on exponential moving average according to any one of claims 1 to 4; Obtaining the target street view image collected locally by the target vehicle; Performing street view semantic understanding on the target street view image through the target global street view understanding model to obtain the target street view semantics; Controlling the driving of the target vehicle according to the target street view semantics.

6. A vehicle control device, characterized in that, The device includes: A model acquisition module, configured to obtain the target global street view understanding model sent by the server; wherein, the target global street view understanding model is trained according to the distributed model training method based on exponential moving average according to any one of claims 1 to 4; An image acquisition module, configured to obtain the target street view image collected locally by the target vehicle; A street view understanding module, configured to perform street view semantic understanding on the target street view image through the target global street view understanding model to obtain the target street view semantics; A control module, configured to control the driving of the target vehicle according to the target street view semantics.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the distributed model training method based on exponential moving average according to any one of claims 1 to 4 or the vehicle control method according to claim 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the distributed model training method based on exponential moving average according to any one of claims 1 to 4 or the vehicle control method according to claim 5.

Images