An intelligent driving perception method based on federated learning and semi-supervised learning

Abstract

The application discloses an intelligent driving perception method based on federated learning and semi-supervised learning. The method firstly uses existing labeled data to train a perception model for an intelligent driving terminal in a supervised manner, and deploys the perception model on the intelligent driving terminal. Secondly, a horizontal federated learning system is constructed, in which each intelligent driving terminal is a federated participant, and a cloud server is a federated aggregator. Finally, when a certain amount of perception data is obtained at each intelligent driving terminal, the aforementioned semi-supervised model training method is adopted, and federated learning is combined to iteratively update the perception model by multiple participants. The method based on federated learning and semi-supervised learning is used to improve the reliability and efficiency of intelligent driving perception software.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, specifically to an intelligent driving perception method based on federated learning and semi-supervised learning. Background Technology

[0002] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and is not necessarily to be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

[0003] Perception is crucial in intelligent driving technology, characterized by abundant sensing devices, diverse sensing data modalities, massive data volumes, and complex task outputs. Machine learning techniques are widely applied in this field, such as computer vision, natural language processing, and recommendation systems. The successful application of these machine learning techniques relies on a foundation of large amounts of data. However, in real-world applications, the data we can obtain is often small-scale or requires annotation by industry experts, making it difficult to acquire high-quality, large-volume training data. When people use a service, the data generated is collected and transmitted to a central point with a computing cluster capable of using this massive amount of data to train machine learning models. However, as artificial intelligence applications continue to be implemented across various industries, public concern for user privacy and data security is constantly increasing, especially in sectors like finance and healthcare where user data privacy is paramount. In the field of intelligent driving, due to the widespread distribution of vehicles and the large volume of sensing data, timely collection and model iteration updates are difficult. Furthermore, data sharing between different intelligent driving technology companies is even more challenging.

[0004] Federated learning aims to build a federated learning model based on distributed datasets. It is designed to be a form that does not require direct data exchange or collection and allows several participants to collaboratively train a machine learning model. While protecting user privacy and data security, it also enables each party to obtain a model that is better than the one they trained themselves. In this process, the consumption of data traffic will also be greatly reduced.

[0005] Semi-supervised learning is a key research area in pattern recognition and machine learning, combining supervised and unsupervised learning methods. It utilizes a large amount of unlabeled data alongside labeled data for pattern recognition. Semi-supervised learning requires minimal human intervention while achieving relatively high accuracy, thus gaining increasing attention. In autonomous driving terminals, the sensing devices continuously collect and aggregate new sensory data. Traditional supervised learning methods require full manual annotation of the data before model training. However, semi-supervised learning significantly reduces annotation costs while maintaining efficient model updates. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides an intelligent driving perception method based on federated learning and semi-supervised learning, which improves the reliability and efficiency of intelligent driving perception software.

[0007] To solve the aforementioned technical problem, the technical solution adopted by the present invention is: an intelligent driving perception method based on federated learning and semi-supervised learning, comprising the following steps:

[0008] S01) Use existing labeled data to train a perception model for intelligent driving terminals in a supervised manner and deploy it on intelligent driving terminals;

[0009] S02) Construct a horizontal federated learning system with each intelligent driving terminal as a federated participant and the cloud server as a federated aggregator. The horizontal federated learning system enables each intelligent driving terminal to collaboratively train the perception model.

[0010] S03) When a certain amount of detection data is acquired by each intelligent driving terminal, a semi-supervised model training method is adopted, combined with federated learning, and multiple parties work together to iteratively update the perception model.

[0011] S04), the updated perception model is used for detection during intelligent driving.

[0012] Furthermore, step S03) specifically involves:

[0013] S31) Each federated learning participant, i.e., the intelligent driving terminal, replicates the existing perception model into a teacher model and a student model;

[0014] S32) After weak augmentation of the unlabeled new data, it is input into the teacher model to obtain the prediction results. Then, some potential erroneous samples are filtered out by a pre-set confidence threshold δ to generate pseudo-labels.

[0015] S33) After strongly augmenting the unlabeled new data, it is combined with the pseudo-labels generated in S32) and input into the student model for supervised forward and backward propagation updates.

[0016] S34) The EMA algorithm is used to update the weights of the teacher model in combination with the weights updated in the student model;

[0017] S35) Send the weights of the teacher model and the student model to the cloud server respectively;

[0018] S36) After receiving the encrypted model weights sent by each intelligent driving terminal, the cloud server uses the FedAvg algorithm to aggregate the teacher model and student model respectively, and then determines whether the loss of the student model has converged. The aggregated teacher model and student model weights are then sent to each intelligent driving terminal. If the loss of the student model has converged, the federated learning process ends.

[0019] S37) After receiving the aggregated model from the cloud server, the intelligent driving terminal makes a status judgment. If the loss of the student model in S36) has converged, that is, the federated learning process has ended, then the model being tested in actual operation is replaced with the student model. If the loss of the student model in S36 has not yet converged, then steps S32 to S37 are repeated.

[0020] Furthermore, in step S35), the weights of the teacher model and the student model are encrypted using homomorphic encryption and then sent to the cloud server. In step S37), if the intelligent driving terminal determines that the loss of the student model in S36) has converged, the student model is decrypted first and then the model actually tested is replaced with the student model.

[0021] Furthermore, the horizontal federated learning system employs a federated averaging method for federated learning.

[0022] Furthermore, this method is used for obstacle detection, lane line detection, drivable area detection, and obstacle motion prediction in intelligent driving.

[0023] Furthermore, this method is used for intelligent question answering in autonomous driving.

[0024] The beneficial effects of this invention are as follows: The intelligent driving perception method described in this invention is based on federated learning and semi-supervised learning. Federated learning enables each intelligent driving terminal to participate in local model training and upload model parameters as a federated participant. While protecting user privacy and data security, it also allows each party to obtain a better model than it trained itself. In this process, data traffic consumption is also greatly reduced. Semi-supervised learning is used to train machine learning models using a large number of unlabeled samples and a relatively small number of labeled samples, solving the problem of insufficient labeled samples. It reduces the human and material costs of data labeling while ensuring that the final model performance is minimally affected. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a federated learning system. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] This embodiment discloses an intelligent driving perception method based on federated learning and semi-supervised learning. The method includes the following steps:

[0029] S01) Use existing labeled data to train a perception model for intelligent driving terminals in a supervised manner and deploy it on intelligent driving terminals;

[0030] S02) Construct a horizontal federated learning system with each intelligent driving terminal as a federated participant and the cloud server as a federated aggregator. The horizontal federated learning system enables each intelligent driving terminal to collaboratively train the perception model.

[0031] S03) When a certain amount of detection data is acquired by each intelligent driving terminal, a semi-supervised model training method is adopted, combined with federated learning, and multiple parties work together to iteratively update the perception model.

[0032] S04), the updated perception model is used for detection during intelligent driving.

[0033] The federated learning system described in this embodiment enables each intelligent driving terminal to collaboratively train its perception model, thereby allowing each party to obtain a better model than it trained independently. For example... Figure 1 As shown, the federated learning system includes various intelligent driving terminals and a cloud server. During the federated learning process, each intelligent driving terminal acts as a federated participant, performing local model training and uploading model parameters, while the cloud server acts as a federated aggregator, initiating federated learning tasks, collecting, aggregating, and distributing model parameters from participating parties.

[0034] To ensure data privacy, the model parameters will be sent using homomorphic encryption. Homomorphic encryption is a cryptographic technique based on the computational complexity theory of mathematical problems. Processing homomorphically encrypted data yields an output; decrypting this output produces the same result as processing the unencrypted original data using the same method. A homomorphic encryption function refers to two plaintext... a , b Satisfy Dec( En ( a )⊙ En ( b ))= a ⊕ b The encryption function, where En ⊙ represents encryption, Dec represents decryption, and ⊙ and ⊕ correspond to operations on the plaintext and ciphertext fields, respectively.

[0035] In this embodiment, the federated learning process will be carried out using the federated averaging (FedAvg) method.

[0036] This embodiment also employs a semi-supervised learning method to train a machine learning model using a large number of unlabeled samples and a relatively small number of labeled samples, thus solving the problem of insufficient labeled samples. This reduces the human and material costs associated with data labeling while ensuring minimal loss in the final model's performance.

[0037] The following uses obstacle detection based on computer vision as an example to illustrate the specific steps of this method:

[0038] 1. Collect and label a certain amount of monocular camera image data, use the SSD algorithm to train a target detection model that can detect common road obstacles such as people, vehicles, and cones, and deploy it on the intelligent driving terminal, so as to predict the position of obstacles in the camera's line of sight in real time during the operation of the intelligent driving vehicle, thereby realizing deceleration or detour;

[0039] 2. Use virtual private networks (VPNs) and other means to ensure network communication between the cloud server and each intelligent driving terminal, and deploy federated learning software on the cloud server and each intelligent driving terminal respectively;

[0040] 3. After the intelligent driving terminal collects a certain amount of new data, the cloud server issues a federated learning task, selecting two or more intelligent driving terminals to jointly iterate and update the model:

[0041] a. Each participant in the federated learning process (i.e., the intelligent driving terminal) replicates the existing obstacle detection model as both a teacher model and a student model;

[0042] b. After weak augmentation of the unlabeled new data, it is input into the teacher model to obtain the prediction results. Then, some potential erroneous samples are filtered out by a pre-set confidence threshold δ to generate pseudo-labels.

[0043] c. After strongly augmenting the unlabeled new data, combine it with the pseudo-labels generated in b, and input it into the student model for supervised forward and backward propagation updates;

[0044] d. Use the EMA algorithm, combined with the weights updated in the student model, to update the weights in the teacher model;

[0045] e. Encrypt the weights of the teacher model and the student model using homomorphic encryption and send them to the cloud server;

[0046] f. After receiving the encrypted model weights sent by each intelligent driving terminal, the cloud server uses the FedAvg algorithm to aggregate the teacher model and student model respectively, and then determines whether the student model loss has converged. The aggregated teacher model and student model weights are then sent to each intelligent driving terminal. If the student model loss has converged, the federated learning process ends.

[0047] g. After receiving the aggregated model from the cloud server, the intelligent driving terminal makes a status judgment. If the loss of the student model in f has converged, that is, the federated learning process has ended, the student model is decrypted and the model that actually runs the prediction and inference is replaced with the student model. If the loss of the student model in f has not converged, steps b to f are repeated.

[0048] In addition to the obstacle detection method described in this embodiment, the perception method can also be used for lane line detection, drivable area detection, obstacle movement prediction in intelligent driving, or for intelligent question-and-answer systems in intelligent driving.

[0049] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An intelligent driving perception method based on federated learning and semi-supervised learning, characterized in that: Comprise the following steps: S01), using existing labeled data to train the perception model for intelligent driving terminal in a supervised manner, and deploy it on the intelligent driving terminal; S02), a horizontal federated learning system is constructed, in which each intelligent driving terminal is a federated participant and the cloud server is a federated aggregator, the horizontal federated learning system enables each intelligent driving terminal to collaboratively train the perception model; S03), when a certain amount of detection data is obtained at each intelligent driving terminal, a semi-supervised model training method is adopted, combined with federated learning, multiple participants jointly update the perception model iteratively; Step S03) is specifically: S31), each federated learning participant, i.e., intelligent driving terminal, copies the existing perception model as a teacher model and a student model; S32), after weak enhancement of unlabeled new data, the teacher model is input to obtain a prediction result, and a pre-set confidence threshold δ is used to filter part of the potential error samples to generate pseudo labels; S33), after strong enhancement of the unlabeled new data, the pseudo labels generated in S32) are combined and input into the student model for supervised forward propagation and backward propagation update; S34), the EMA algorithm is adopted to update the weight of the teacher model combined with the updated weight of the student model; S35), the weights of the teacher model and the student model are sent to the cloud server respectively; S36), after receiving the encrypted model weights sent by each intelligent driving terminal, the cloud server aggregates the teacher model and the student model respectively using the FedAvg algorithm, then judges whether the loss of the student model converges, and respectively distributes the aggregated teacher model and student model weights to each intelligent driving terminal, if the loss of the student model has converged, the federated learning process is ended; S37), after receiving the aggregated model distributed by the cloud server, the intelligent driving terminal performs state judgment, if the loss of the student model in S36) has converged, i.e., the federated learning process has ended, the actually running detection model is replaced by the student model, if the loss of the student model in S36) has not converged, steps S32 to S37) are repeated; S04), the updated perception model is used for detection in the intelligent driving process.

2. The intelligent driving perception method based on federated learning and semi-supervised learning according to claim 1, characterized in that: In step S35), the weights of the teacher model and the student model are encrypted by homomorphic encryption method and sent to the cloud server, in step S37), if the intelligent driving terminal judges that the loss of the student model in S36) has converged, the student model is decrypted before replacing the actually running detection model with the student model. 3.The intelligent driving perception method based on federated learning and semi-supervised learning according to claim 1, wherein: The horizontal federated learning system adopts the federated average method for federated learning. 4.The intelligent driving perception method based on federated learning and semi-supervised learning according to claim 1, wherein: The method is used for obstacle detection, lane line detection, drivable area detection, and obstacle motion prediction in intelligent driving. 5.The intelligent driving perception method based on federated learning and semi-supervised learning according to claim 1, wherein: The method is used for intelligent question and answer in intelligent driving.

Images