Federal learning-based data processing method and system, and electronic equipment

Abstract

The invention provides a data processing method and system based on federated learning and electronic equipment, and the method comprises the steps: a member object of federated learning determines the Kalman gains of local various types of sensors, and uploads the collected data features of the various types of sensors and the corresponding Kalman gains to a centralized calculation platform of federated learning, a plurality of types of sensors are applied to unmanned driving of a target type of vehicle. And the centralized computing platform performs weighted fusion on the acquired data features from different member objects of each sensor type according to corresponding Kalman gains, and trains a global deep learning model based on the fused acquired data features corresponding to each sensor type. And the member objects adjust the local deep learning model based on the trained target model parameters provided by the centralized computing platform. Wherein the global deep learning model and the local deep learning model are both used for predicting unmanned driving control parameters of the vehicle of the target type.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving, and in particular to a data processing method, system and electronic equipment based on federated learning. Background Art

[0002] Existing unmanned driving technology generally uses a linear weighted method to fuse multi-sensor information, thereby achieving unmanned driving control based on the fusion results. In complex environments, these sensors will be affected by factors such as lighting, motion, and texture, and will show different levels of interference. If a simple linear weighted method is used to realize multi-sensor information, the final fusion result will be unstable due to the influence of sensors with greater interference, which will lead to a decrease in the accuracy of unmanned driving. Summary of the invention

[0003] The purpose of this application is to provide a data processing method, system and electronic equipment based on federated learning, which can solve the problem of unstable multi-sensor information fusion in the field of unmanned driving.

[0004] In order to achieve the above purpose, the embodiment of the present application is implemented as follows: In a first aspect, a data processing method based on federated learning is provided, comprising: The member objects of the federated learning determine the Kalman gains of the local multiple types of sensors, and upload the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to the unmanned driving of the target type of transportation vehicle; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type, wherein the global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle; The centralized computing platform sends the target model parameters of the global deep learning model that has been trained to the member objects; The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0005] In a second aspect, a data processing system based on federated learning is provided, comprising: a member object device and a centralized computing platform, wherein: The member object determines the Kalman gains of multiple types of local sensors, and uploads the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to the unmanned driving of the target type of transportation vehicle; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type, wherein the global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle; The centralized computing platform sends the target model parameters of the global deep learning model that has been trained to the member objects; The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the method described in the first aspect.

[0007] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the method described in the first aspect when executed by a processor.

[0008] The embodiment of the present application trains a deep learning model under a federated learning architecture to predict the unmanned driving control parameters of a target type of vehicle. The federated learning architecture includes multiple member objects and a centralized computing platform. The deep learning model configured by the member object is called a local deep learning model, and the deep learning model configured by the centralized computing platform is called a global deep learning model. In the training phase, the member object determines the Kalman gain of the local multiple types of sensors, and uploads the acquisition data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform. The centralized computing platform performs weighted fusion of the acquisition data features from different member objects of each sensor type according to the corresponding Kalman gain to obtain the fused acquisition data features corresponding to each sensor type, and trains the global deep learning model based on the fused acquisition data features corresponding to each sensor type, so as to send the target model parameters of the trained global deep learning model to the member objects, and the member objects adjust the local local deep learning model. It should be understood that the Kalman gain characterizes the degree of uncertainty of sensor data. Therefore, after weighted fusion according to the Kalman gain, the impact of sensors with lower contribution on model training can be weakened, such as sensors that are severely interfered by environmental factors, thereby improving the quality of the trained target model parameters. After adjusting the local deep learning model, more accurate unmanned driving control parameters can be predicted in the subsequent application stage to ensure the accuracy of unmanned driving control. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A flowchart of a data processing method based on federated learning according to an embodiment of the present application.

[0011] Figure 2 A schematic diagram of the application architecture of a data processing method based on federated learning according to an embodiment of the present application.

[0012] Figure 3 A schematic diagram of feature extraction of a data processing method based on federated learning in an embodiment of the present application.

[0013] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0015] An embodiment of the present application provides a data processing method based on federated learning, which is applied to unmanned driving of target types of vehicles. Figure 1 It is a flowchart of the data processing method of the industry, including the following steps: S102, the member objects of the federated learning determine the Kalman gains of the local multiple types of sensors, and upload the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning. The multiple types of sensors are applied to the unmanned driving of the target type of vehicles.

[0016] Federated Learning (FL) is a distributed machine learning technology that allows multiple participants to collaboratively train a shared model without directly sharing the original data. Its core idea is to build deep learning models based on data sets distributed across multiple devices or institutions while preventing data leakage, thereby achieving joint modeling and model optimization while ensuring data privacy and security.

[0017] refer to Figure 2As shown, in this embodiment, the architecture of federated learning consists of multiple member objects and a centralized computing platform in the cloud. Among them, the member objects of this embodiment can be manufacturers or users of target type vehicles (such as vehicles, drones, unmanned boats, etc.). Taking the member object as a manufacturer as an example, although there are some differences in the configuration of the target type of vehicles of each manufacturer, the types of sensors used are roughly the same, that is, the various types of sensors described in this article are used on the vehicles of each member object. In addition, in federated learning, each member object and the centralized computing platform each maintains a local deep learning model for predicting the unmanned driving control parameters of the target type of vehicle. This article defines the deep learning model deployed on the member object side as a local deep learning model, and defines the deep learning model deployed on the centralized computing platform side as a global deep learning model. The process of federated learning is as follows: each member object uses a local deep learning model to extract features from the sensor data detected by various types of local sensors to obtain the collected data features of various types of sensors (features of the data collected by the sensors); then, each member object provides the collected data features of various types of sensors to the centralized computing platform, which uses its abundant computing resources to complete the training of the local global deep learning model; then, the centralized computing platform feeds back the target model parameters of the trained global deep learning model to each member object, and the member object adjusts the local deep learning model according to the target model parameters.

[0018] It can be seen that federated learning uses the local sensor data of multiple member objects to train the model on the centralized platform side, thus enriching the diversity of training data and improving the quality and accuracy of the model. At the same time, due to the differences in sensors of different member objects, this difference helps to improve the generalization ability of the model, enabling it to achieve better performance on unseen data.

[0019] Based on the above-mentioned federated learning, this embodiment uses Kalman gain to assist training.

[0020] Kalman filter is a linear minimum variance estimator used to estimate a random process or system in the presence of noise. It is a recursive filter that can estimate the state of a system through a series of incomplete and noisy data.

[0021] Kalman gain is a core concept in Kalman filter theory. In this embodiment, it is used to characterize the degree of change of uncertainty in the data during the fusion process. It should be understood that if the uncertainty is reduced after the fusion of new data, then the gain is positive; conversely, if the uncertainty increases, then the gain is negative. Therefore, this embodiment uses Kalman gain to ensure the contribution of each sensor. It should be noted that for the same type of sensor, different member objects have their own sensor configurations, so there are differences in the detection sensor data, corresponding to different Kalman gains.

[0022] As an example, for a member object, the calculation formula of the Kalman gain of any type of sensor is: , (Formula 1); in, ; Indicates the moment; represents the Kalman gain of the sensor at time k; represents the state covariance matrix estimated a priori based on the characteristics of the collected data at time k; Represents the state transfer matrix (also called observation matrix) determined by the motion state of the vehicle based on the target type. The initial value can be set to ; represents the covariance matrix of the sensor observation noise, , the 0.1 in the upper left corner (the value can be calculated based on the error of the actual value) can be expressed as the variance of the position error, and the 0.1 in the lower right corner can be expressed as the variance of the speed error; represents the process error; A represents the relationship between the motion state at the previous moment and the motion state at the next moment, A= .

[0023] Among them, the iterative process of Kalman gain includes: Prediction step: Use the optimal estimate value and system model of the previous moment to predict the state value and error covariance matrix at the current moment.

[0024] Calculation steps: Formula 1; Update step: Use the observed and predicted values ​​at the current moment, as well as the Kalman gain, to update the state value and error covariance matrix.

[0025] The Kalman gain can be viewed as a trade-off factor used to find an optimal balance between the predicted value and the observed value. When it is smaller, the observed value is more reliable and the Kalman gain increases, making the updated state value closer to the observed value.

[0026] In addition, different types of sensors also require cross-modal retrieval to facilitate calculation. The core of cross-modal retrieval is to achieve the correspondence between data of different modalities, and the basis of this correspondence is feature extraction. Through feature extraction, data of different modalities can be converted into a unified feature vector representation, so as to perform similarity measurement and matching in a shared feature space. This allows users to retrieve data of one modality through information of another modality, realizing the effective association of multimodal data.

[0027] Assume that federated learning has K member objects (K=1, 2, ..., k, ..., K), each member object has M types of sensors, and the member objects need to train local deep learning models. Multiple rounds (n rounds) of iterations are required to train local deep learning models.

[0028] Then, after the nth round, the sensor data of the nth round is used to extract the features of the time series. Here, it is necessary to consider that the features of the same type of sensors in adjacent rounds have a certain similarity, the feature data within the modality have a certain similarity, and the features extracted by the same sensor in multiple rounds over a period of time have a certain similarity.

[0029] For ease of understanding, refer to Figure 3 As shown in the figure, two sensors of different modes are used as an example. If the sensor data in the same mode Sensor data are similar, then after feature extraction and It should be similar, The value of is very small. Therefore, the following optimization objectives can be designed for the similarity preservation within the modality: ,(Formula 2); in, represents the feature transformation of n data of sensor p in round N; , represents the feature transformation of n data of sensor q in round N; Indicates the similarity of the data features collected within the modality, ( ) represents the characteristics of the collected data of the same mode Data collection characteristics The similarity measure between if The corresponding The value is 1, otherwise the corresponding The value is 0, express Next-level neighbors.

[0030] It can be organized into the following formula 2 , (Formula 3); Similarly, in the original data features of each modality, the similarity information between modalities as feature conversion information also needs to be considered separately. The associated features between different modal data describing the same scene for the same client also need to be maintained after transformation. and It is a pair of samples describing the same scene on different sensors for the same client. and The original similarity information should also be retained. To this end, the inter-modal similarity preservation can be optimized as: , (Formula 4); in, Representing modality and If different sensors collect sensor data of the same modality within a specific time interval in the same round, then The value of is 1, otherwise The value of is 0; and Indicates the coefficient determined based on the correlation between sensors.

[0031] It can be organized into the following formula: , (Formula 5); In order to obtain the mapping relationship between the collected sensor data and the feature extraction, it is necessary to minimize Formula 2 and Formula 4: , (Formula 6); The feature extraction formula of the sensor can be obtained through Formula 6, and the data collected by the two sensors are converted into corresponding collected data features U and collected data features V.

[0032] The above feature extraction formula realizes cross-modal retrieval without relying on manual annotation.

[0033] S104, the centralized computing platform performs weighted fusion on the collected data features from different member objects of each sensor type according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains the global deep learning model based on the fused collected data features corresponding to each sensor type. The global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0034] It should be understood that after weighted fusion according to the Kalman gain, the influence of sensors with lower contribution on the training of the global deep learning model can be weakened.

[0035] It should be noted that training unmanned driving control parameters for deep learning model prediction belongs to the prior art, and this embodiment does not specifically limit the training method of the global deep learning model.

[0036] S106, the centralized computing platform sends the target model parameters of the trained global deep learning model to the member objects.

[0037] S108, receiving the member object of the target model parameters, adjusting the local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0038] As an example, if the local deep learning model is completely consistent with the global deep learning model, the member object can directly configure the local local deep learning model according to the target model parameters. In addition, if the member object does not want to rely entirely on the training results of the centralized computing platform, it can also configure the local local deep learning model according to the target model parameters and then train the local deep learning model in combination with the local unique collection data features.

[0039] In summary, the method of this embodiment trains a deep learning model under a federated learning architecture to predict the unmanned driving control parameters of a target type of vehicle. Among them, the federated learning architecture includes multiple member objects and a centralized computing platform. The deep learning model configured by the member object is called a local deep learning model, and the deep learning model configured by the centralized computing platform is called a global deep learning model. In the training stage, the member object determines the Kalman gain of the local multiple types of sensors, and uploads the acquisition data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform. The centralized computing platform performs weighted fusion of the acquisition data features from different member objects of each sensor type according to the corresponding Kalman gain to obtain the fused acquisition data features corresponding to each sensor type, and trains the global deep learning model based on the fused acquisition data features corresponding to each sensor type, so as to send the target model parameters of the trained global deep learning model to the member object, and the member object adjusts the local local deep learning model. It should be understood that the Kalman gain characterizes the degree of uncertainty of sensor data. Therefore, after weighted fusion according to the Kalman gain, the impact of sensors with lower contribution on model training can be weakened, such as sensors that are severely interfered by environmental factors, thereby improving the quality of the trained target model parameters. After adjusting the local deep learning model, more accurate unmanned driving control parameters can be predicted in the subsequent application stage to ensure the accuracy of unmanned driving control.

[0040] In addition, in this embodiment, the centralized computing platform can also use the test set provided by each member object to test the trained global deep learning model, and determine the contribution of each member object to federated learning with the goal of minimizing the total test loss; wherein the total test loss is associated with the contribution of each member object to federated learning and the test loss corresponding to the test set provided by each member object in the test. Afterwards, the centralized computing platform performs matching incentive processing on the member objects according to the determined contribution. For example: the centralized computing platform marks the member objects whose contribution does not reach the preset contribution threshold as low-contribution member objects, so as to no longer obtain the collected data features used for training the global deep learning model from the low-contribution member objects, thereby reducing the computational complexity. For another example: the centralized computing platform adjusts the weight coefficient of the weighted fusion of the collected data features provided by each member object according to the contribution of each member object, so that the collected data features provided by the member objects with high contribution have a greater impact on the training of the global deep learning model.

[0041] As an example, the objective function for minimizing the total test loss is: , (Formula 7) in, Indicates the contribution of the kth member object to the global deep model during the federated optimization process. Represents a member object from the test collection The fusion data features randomly acquired for testing are Indicates that the kth member object uses the parameters of the global deep model And randomly obtain The test loss is calculated. Through n rounds of federated learning, the value of Formula 7 is minimized and the contribution of each member object to the global deep model is calculated.

[0042] In addition, corresponding to Figure 1 Another embodiment of this embodiment also provides a data processing system based on federated learning. The data processing system can use Figure 2 The application architecture shown includes: member object devices and a centralized computing platform. Among them: The member object determines the Kalman gains of multiple types of local sensors, and uploads the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to the unmanned driving of the target type of transportation vehicle; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type, wherein the global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle; The centralized computing platform sends the target model parameters of the global deep learning model that has been trained to the member objects; The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0043] Optionally, after the centralized computing platform trains the global deep learning model based on the fused sensor features corresponding to each sensor type, the centralized computing platform uses the test set provided by each of the member objects to test the trained global deep learning model, and determines the contribution of each of the member objects to the federated learning with the goal of minimizing the total test loss; wherein the total test loss is associated with the contribution of each of the member objects to the federated learning and the test loss corresponding to the test set provided by each of the member objects in the test. The centralized computing platform performs matching incentive processing on the member objects according to the determined contribution.

[0044] Optionally, the centralized computing platform performs matching incentive processing on the member objects according to the determined contribution: the centralized computing platform marks the member objects whose contribution does not reach a preset contribution threshold as low-contribution member objects, so as to no longer obtain collected data features used for training the global deep learning model from the low-contribution member objects.

[0045] Optionally, the centralized computing platform performs matching incentive processing on the member objects according to the determined contribution: the centralized computing platform adjusts the weight coefficient for weighted fusion of the collected data features provided by each member object according to the contribution of each member object.

[0046] Optionally, the Kalman gain of any type of sensor is calculated as: ; in, ; Indicates the moment; represents the Kalman gain of the sensor at time k; represents the state covariance matrix estimated a priori based on the characteristics of the collected data at time k; represents a covariance matrix determined based on the motion state of a vehicle of the target type; represents the covariance matrix of sensor observation noise; Represents process error; A represents the relationship between the motion state at the previous moment and the motion state at the next moment.

[0047] Optionally, the features of the collected data of the multiple types of sensors provided by the member object are obtained by extracting features of the collected data of the multiple types of sensors by the local deep learning model completed by the local Nth round of training; The feature extraction formula is: ; in, ; ; , represents the feature transformation of n data of sensor p in round N; , represents the feature transformation of n data of sensor q in round N; Indicates the similarity of the data features collected within the modality, ( ) represents the characteristics of the collected data of the same mode Data collection characteristics The similarity measure between if The corresponding The value is 1, otherwise the corresponding The value is 0; express order neighbors; Representing modality and If different sensors collect sensor data of the same modality within a specific time interval in the same round, then The value of is 1, otherwise The value of is 0; and Indicates the coefficient determined based on the correlation between sensors.

[0048] Optionally, the target type includes at least one of a vehicle, a drone, and an unmanned boat.

[0049] It should be noted that the data processing system based on federated learning in this embodiment can be used as Figure 1 The execution subject of the method shown can thus realize Figure 1 The steps and functions in the method shown.

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0051] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0052] The memory is used to store the computer program. Specifically, the computer program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides the computer program to the processor.

[0053] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming the above-mentioned data processing system based on federated learning at the logical level. Correspondingly, the processor executes the program stored in the memory and is specifically used to perform the following operations: The member objects of the federated learning determine the Kalman gains of multiple types of local sensors, and upload the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning. The multiple types of sensors are applied to the unmanned driving of the target type of vehicles.

[0054] The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type. The global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0055] The centralized computing platform sends the target model parameters of the trained global deep learning model to the member objects.

[0056] The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0057] The above is as in this manual Figure 1The method disclosed in the illustrated embodiment can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0058] Of course, in addition to software implementation, the electronic device of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0059] In addition, the present application also provides a computer-readable storage medium, which stores one or more computer programs, wherein the one or more computer programs include instructions. When the instructions are executed by a portable electronic device including multiple application programs, the portable electronic device can execute Figure 1 The steps in the method shown include: The member objects of the federated learning determine the Kalman gains of the local multiple types of sensors, and upload the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to the unmanned driving of the target type of transportation vehicle; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type. The global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0060] The centralized computing platform sends the target model parameters of the trained global deep learning model to the member objects.

[0061] The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

[0062] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The above are only embodiments of this specification and are not intended to limit this specification. For those skilled in the art, this specification may be subject to various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims of this specification. In addition, all other embodiments obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of this document.

Claims

1. A data processing method based on federated learning, characterized in that: include: The member objects of the federated learning determine the Kalman gains of the local multiple types of sensors, and upload the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to the unmanned driving of the target type of transportation vehicle; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type, wherein the global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle; The centralized computing platform sends the target model parameters of the global deep learning model that has been trained to the member objects; The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

2. The method according to claim 1, characterized in that After the centralized computing platform trains the global deep learning model based on the fused sensor features corresponding to each sensor type, the method further includes: The centralized computing platform uses the test set provided by each of the member objects to test the trained global deep learning model, and determines the contribution of each of the member objects to the federated learning with the goal of minimizing the total test loss; wherein the total test loss is associated with the contribution of each of the member objects to the federated learning and the test loss corresponding to the test set provided by each of the member objects in the test. The centralized computing platform performs matching incentive processing on the member objects according to the determined contribution degrees.

3. The method according to claim 2, characterized in that The centralized computing platform performs matching incentive processing on the member objects according to the determined contribution, including: The centralized computing platform marks the member objects whose contributions do not reach a preset contribution threshold as low-contribution member objects, so as to no longer obtain the collected data features used for training the global deep learning model from the low-contribution member objects.

4. The method according to claim 1, characterized in that: The centralized computing platform performs matching incentive processing on the member objects according to the determined contribution: The centralized computing platform adjusts the weight coefficient for weighted fusion of the collected data features provided by each of the member objects according to the contribution of each of the member objects.

5. The method according to claim 1, characterized in that The Kalman gain for any type of sensor is calculated as: ； in, ; Indicates the moment; represents the Kalman gain of the sensor at time k; represents the state covariance matrix estimated a priori based on the characteristics of the collected data at time k; represents a covariance matrix determined based on the motion state of a vehicle of the target type; represents the covariance matrix of sensor observation noise; Represents process error; A represents the relationship between the motion state at the previous moment and the motion state at the next moment.

6. The method according to claim 1, characterized in that The collected data features of the multiple types of sensors provided by the member object are obtained by extracting features from the collected data of the multiple types of sensors by the local deep learning model completed by the local Nth round of training; The feature extraction formula is: ； in, ; ; , represents the feature transformation of n data of sensor p in round N; , represents the feature transformation of n data of sensor q in round N; Indicates the similarity of the data features collected within the modality, ( ) represents the characteristics of the collected data of the same mode Data collection characteristics The similarity measure between if The corresponding The value is 1, otherwise the corresponding The value is 0; express order neighbor; Representing modality and If different sensors collect sensor data of the same modality within a specific time interval in the same round, then The value of is 1, otherwise The value of is 0; and Indicates the coefficient determined based on the correlation between sensors.

7. The method according to any one of claims 1 to 6, characterized in that: The target type of transportation includes at least one of a vehicle, a drone, and an unmanned boat.

8. A data processing system based on federated learning, member object devices and a centralized computing platform, characterized in that: The member object determines Kalman gains of multiple types of local sensors, and uploads the collected data features and corresponding Kalman gains of the multiple types of sensors to the centralized computing platform of the federated learning, wherein the multiple types of sensors are applied to unmanned driving of target types of vehicles; The centralized computing platform performs weighted fusion on the collected data features of each sensor type from different member objects according to the corresponding Kalman gain to obtain the fused collected data features corresponding to each sensor type, and trains a global deep learning model based on the fused collected data features corresponding to each sensor type, wherein the global deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle; The centralized computing platform sends the target model parameters of the global deep learning model that has been trained to the member objects; The member object that receives the target model parameters adjusts a local deep learning model based on the target model parameters, where the local deep learning model is used to predict the unmanned driving control parameters of the target type of vehicle.

9. An electronic device, comprising: processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer-readable storage medium storing a computer program, characterized in that: The computer program is operable to cause a computer to perform the method according to any one of claims 1 to 7.

Images