Driving control method and system based on personalized federated contrastive learning

Abstract

The present invention relates to intelligent decision-making technology, and discloses a driving control method based on personalized federated contrastive learning, including: a client obtains driving data, and uses a multimodal processor to extract features from the driving data to obtain multimodal driving feature data; performs data processing on the multimodal driving feature data to obtain multimodal driving contrast feature data; trains a contrastive learning driving prediction model to obtain a target driving prediction model, and a central server aggregates global neuron parameters to obtain fusion parameters and distributes them to multiple clients, and the client uses the fusion parameters to perform personalized parameter optimization on the target driving prediction model; the client receives real-time driving data of the user to perform driving prediction and obtain the user's driving prediction result. The present invention also proposes a driving control system based on personalized federated contrastive learning. The present invention can realize personalized driving prediction of driving control based on personalized federated contrastive learning.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology, and in particular to a driving control method and system based on personalized federated comparative learning. Background Art

[0002] Autonomous driving is an emerging field with the potential to revolutionize the way humans travel. Existing autonomous driving methods are mostly based on machine learning, especially deep learning techniques that require large-scale training data. Many studies have explored the ability to derive end-to-end driving strategies directly from perception data, and the results of these methods have been applied to different application scenarios, such as lane following, autonomous navigation in complex environments, autonomous driving on artificial roads, and drone navigation. However, most of these methods focus on improving the accuracy of the system by using pre-collected datasets, while ignoring the privacy of user data. At the same time, these trained large models also require a large number of labeled datasets, and there are currently no datasets that can meet such requirements. In addition, large models are usually not able to adapt to the driving habits of every driver. The differences between driving prediction models in existing control methods are usually determined by driving scenarios. However, in many similar driving scenarios, different drivers may exhibit very different driving behaviors. Some drivers may frequently change lanes to overtake, while others prefer to stay in the current lane. This difference is difficult to solve through a consistent driving prediction model, and it is impossible to meet the personalized recommendation of driving behavior for users. Summary of the invention

[0003] The present invention provides a driving control method and device based on personalized federated contrastive learning, which can realize personalized driving prediction of driving control based on personalized federated contrastive learning.

[0004] To achieve the above object, the present invention provides a driving control method based on personalized federated contrastive learning, comprising:

[0005] The client obtains driving data, and uses a preset multimodal processor to extract features from the driving data to obtain multimodal driving feature data, wherein the driving data includes multi-view images, LiDAR data, and system information;

[0006] The client processes the multimodal driving feature data using a preset contrast learning algorithm to obtain multimodal driving contrast feature data;

[0007] The client trains the pre-built comparative learning driving prediction model according to the multi-modal driving comparative feature data, and obtains the target driving prediction model after the training is completed;

[0008] The central server selects a plurality of clients participating in the target driving prediction model training, and obtains the global neuron parameters of the target driving prediction model in the plurality of clients, and the central server aggregates the global neuron parameters to obtain fusion parameters;

[0009] The central server distributes the fusion parameters to the multiple clients, and the clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client to obtain a personalized target driving prediction model;

[0010] The client receives the user's real-time driving data, and inputs the user's real-time driving data into the personalized target driving prediction model to perform driving prediction, thereby obtaining the user's driving prediction result.

[0011] Optionally, the extracting features of the driving data using a preset multimodal processor to obtain multimodal driving feature data includes:

[0012] Extracting features from the multi-view images in the driving data using a temporal Q-Former processing algorithm to obtain multi-view image features;

[0013] Extracting features from the LiDAR data in the driving data using an encoder, and converting the extracted features using a ternary processing algorithm to obtain LiDAR features;

[0014] Extracting features of the system information in the driving data using a transformer encoder to obtain system information features;

[0015] A feature fusion operation is performed on the multi-view image features, the LiDAR features, and the system information features to obtain the multimodal driving feature data.

[0016] Optionally, the extracting features of the multi-view images in the driving data by using a temporal Q-Former processing algorithm to obtain multi-view image features includes:

[0017] Extract all multi-view images between the current timestamp and a preset historical timestamp;

[0018] The multi-view image at the preset historical timestamp is subjected to feature extraction using the temporal Q-Former processing algorithm to obtain a multi-view image tag embedding at the preset historical timestamp;

[0019] The multi-view image markers from the preset historical timestamp to the current timestamp are sequentially extracted and embedded to obtain the multi-view image features.

[0020] Optionally, the extracting features of the LiDAR data in the driving data by using an encoder, and converting the extracted features by using a ternary processing algorithm to obtain LiDAR features, includes:

[0021] Extracting features from the LiDAR data using an encoder to obtain LiDAR initial features;

[0022] The LiDAR initial features are converted into features of an obstacle state, an obstacle-free state, and an uncertain obstacle state by using a ternary processing algorithm to obtain the LiDAR features.

[0023] Optionally, performing a feature fusion operation on the multi-view image features, the LiDAR features, and the system information features to obtain the multimodal driving feature data includes:

[0024] The LiDAR feature converted according to the ternary processing algorithm meets the fusion requirement of feature fusion with the multi-view image feature, and the LiDAR feature is fused with the multi-view image feature to obtain a fused feature;

[0025] Performing feature alignment on the fusion feature and the system information feature, and constructing a feature comparison matrix;

[0026] Feature extraction is performed from the feature comparison matrix to obtain the multimodal driving feature data.

[0027] Optionally, inputting the user's real-time driving data into the personalized target driving prediction model to perform driving prediction to obtain the user's driving prediction result includes:

[0028] Obtaining personalized decision information from the user's real-time driving data, and performing feature encoding on the personalized decision information to obtain a personalized decision information feature matrix, wherein the personalized decision information includes system information, driving decision strategy information, and user driving prompt information;

[0029] Encoding the image information and LiDAR information in the real-time driving data of the user, and fusing the image information and LiDAR information to obtain real-time fusion information features;

[0030] Calculate the matching degree between the real-time fusion information feature and the personalized decision information feature matrix, and extract the personalized decision information corresponding to the personalized decision information feature matrix with the highest matching degree;

[0031] The corresponding driving decision strategy information is extracted from the personalized decision information with the highest matching degree to obtain the user driving prediction result.

[0032] Optionally, the using the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client includes:

[0033] The following formula is used to optimize the personalized parameters of the target driving prediction model in the client:

[0034] in, Personalize parameters for the target driving prediction model, is the original parameter of the target driving prediction model, η is the learning rate, Predict model gradients for target driving.

[0035] In order to solve the above problems, the present invention also provides a driving control system based on personalized federated contrastive learning, wherein the system includes a central server and a plurality of clients respectively connected to the central server.

[0036] Optionally, the client is disposed in a vehicle.

[0037] Optionally, the central server obtains the global neuron parameters of the target driving prediction model in the multiple clients, aggregates the global neuron parameters to obtain fusion parameters, and then distributes the fusion parameters to the multiple clients. The clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client.

[0038] The present invention uses a preset multimodal processor to perform feature extraction on the driving data, so as to more comprehensively extract the category features of different data in the driving data, thereby ensuring the comprehensiveness of the extracted features. In addition, by performing data processing on the multimodal driving feature data through a preset contrastive learning algorithm, it can be ensured that the subsequent contrastive learning driving prediction model can make corresponding correct responses in different situations, thereby improving the robustness and adaptability of the system. In addition, a central server is used to distribute the fusion parameters to the multiple clients, and the clients receive the fusion parameters and use the fusion parameters to optimize the personalized parameters of the target driving prediction model in the client, so that the model can be more in line with the driver's driving habits and personalized needs and ensure the privacy and security of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a model training of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a model decision process of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a contrastive learning driving prediction model framework of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention;

[0042] Figure 4 A technical detail diagram of a multi-information fusion module of a decision model of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention;

[0043] Figure 5 A framework flow chart of personalized federated learning of a driving control method based on personalized federated comparative learning provided by an embodiment of the present invention;

[0044] Figure 6 A personalized encoder training flow chart of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention.

[0045] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0047] The embodiment of the present application provides a driving control method based on personalized federated contrastive learning. The execution subject of the driving control method based on personalized federated contrastive learning includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the driving control method based on personalized federated contrastive learning can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0048] Reference Figure 1 FIG. 2 is a flow chart of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention. In this embodiment, the driving control method based on personalized federated contrastive learning includes:

[0049] S1. The client obtains driving data and uses a preset multimodal processor to extract features from the driving data to obtain multimodal driving feature data, wherein the driving data includes multi-view images, LiDAR data, and system information; the client uses a preset contrastive learning algorithm to process the multimodal driving feature data to obtain multimodal driving contrast feature data; the client trains a pre-constructed contrastive learning driving prediction model according to the multimodal driving contrast feature data, and obtains a target driving prediction model after the training is completed.

[0050] In the embodiment of the present invention, the client refers to a storage device for driving data in federated learning.

[0051] In the embodiment of the present invention, the driving data refers to the data recorded when the user drives a vehicle on the road, including multi-view images, LiDAR data, and system information. LiDAR stands for Light Detection and Ranging.

[0052] As an embodiment of the present invention, the step of extracting features from the driving data using a preset multimodal processor to obtain multimodal driving feature data includes:

[0053] Extracting features from the multi-view images in the driving data using a temporal Q-Former processing algorithm to obtain multi-view image features;

[0054] Extracting features from the LiDAR data in the driving data using an encoder, and converting the extracted features using a ternary processing algorithm to obtain LiDAR features;

[0055] Extracting features of the system information in the driving data using a transformer encoder to obtain system information features;

[0056] A feature fusion operation is performed on the multi-view image features, the LiDAR features, and the system information features to obtain the multimodal driving feature data.

[0057] In an embodiment of the present invention, the temporal Q-Former processing algorithm is a lightweight transformer that uses a set of learnable query vectors to extract visual features from a frozen visual model to achieve visual-language alignment between the visual model and a large language model.

[0058] In the embodiment of the present invention, the ternary processing algorithm is a technology that divides data in a set of continuous intervals into three different segments so as to maximize the data differentiation between the three segments. In the embodiment of the present invention, the ternary processing is performed on LiDAR information.

[0059] In the embodiment of the present invention, the multi-view image can be represented as Where T represents the time length, N I Indicates the number of views, H and W indicate the height and width of the image; LiDAR data is ternary data L∈{Y,N,O}, where Y indicates that there is an obstacle ahead, N indicates that there is no obstacle ahead, and O indicates that there may be an obstacle ahead but it is uncertain; System message Where N M Represents the number of system information tags, which are a summary of vehicle speed, driving decisions, and mission definitions.

[0060] Furthermore, the extracting features of the multi-view images in the driving data using a temporal Q-Former processing algorithm to obtain multi-view image features includes:

[0061] Extract all multi-view images between the current timestamp and a preset historical timestamp;

[0062] The multi-view image at the preset historical timestamp is subjected to feature extraction using the temporal Q-Former processing algorithm to obtain a multi-view image tag embedding at the preset historical timestamp;

[0063] The multi-view image markers from the preset historical timestamp to the current timestamp are sequentially extracted and embedded to obtain the multi-view image features.

[0064] Exemplarily, the extracting features of the multi-view images in the driving data by using a temporal Q-Former processing algorithm to obtain multi-view image features comprises the following steps:

[0065] Step 1: Set each view at timestamp-T Enter ViT and have N Q Randomly initialize the query Q-Former to get the image tag embedding

[0066] Step 2: Embed using image tag As a query for Q-Former, we get the image token embedding for the next timestamp by performing the first step again. Repeat these two steps until you get the image tag embedding for the current timestamp Collect all time information from -T to 0.

[0067] The embodiment of the present invention uses a temporal Q-Former processing algorithm to process multi-view images from a preset historical timestamp -T to 0 (current timestamp), thereby avoiding a linear increase in resources required to process time series data as the time length increases.

[0068] In the embodiment of the present invention, the ViT is a model based on the Transformer architecture, which is specifically used for processing image data.

[0069] Furthermore, the extracting features of the LiDAR data in the driving data by using an encoder, and converting the extracted features by using a ternary processing algorithm to obtain LiDAR features, includes:

[0070] Extracting features from the LiDAR data using an encoder to obtain LiDAR initial features;

[0071] The LiDAR initial features are converted into features of an obstacle state, an obstacle-free state, and an uncertain obstacle state by using a ternary processing algorithm to obtain the LiDAR features.

[0072] For example, for LiDAR data, in order to reduce the processing burden of the on-board computer, a three-valued LiDAR data processing method is adopted. The data collected by the LiDAR sensor is extracted by the encoder, and then these features are converted into three states of data by the decoder, which are represented as {Y, N, O}, respectively, where Y indicates that there is an obstacle ahead, N indicates that there is no obstacle ahead, and O indicates that there may be an obstacle ahead but the state is uncertain, and the LiDAR features obtained by the three-valued processing are recorded as

[0073] In the embodiment of the present invention, performing a feature fusion operation on the multi-view image features, the LiDAR features, and the system information features to obtain the multimodal driving feature data includes:

[0074] The LiDAR feature converted according to the ternary processing algorithm meets the fusion requirement of feature fusion with the multi-view image feature, and the LiDAR feature is fused with the multi-view image feature to obtain a fused feature;

[0075] Performing feature alignment on the fusion feature and the system information feature, and constructing a feature comparison matrix;

[0076] Feature extraction is performed from the feature comparison matrix to obtain the multimodal driving feature data.

[0077] Exemplarily, the step of performing a feature fusion operation on the multi-view image features, the LiDAR features, and the system information features to obtain the multimodal driving feature data includes:

[0078] Step 1: Due to the heterogeneity of LiDAR data and multi-view images, it is necessary to fuse these two types of data into a processable data form. Thanks to the previous ternary processing, the LiDAR features are fused with the multi-view image features to obtain fused data.

[0079] Step 2: Fusion data System Information Features Align and construct a contrast matrix C. In the contrast matrix, the elements on the diagonal are matched positive samples, while the elements on the off-diagonal are unmatched negative samples. Therefore, through N positive samples, there are N 2 -N negative samples can make the positive samples as close as possible and the negative samples as far away as possible, and use the InfoNCE function to calculate the loss:

[0080]

[0081] in is the characteristic representation of the i-th system information, is the fusion data (positive sample) that matches it, is a negative sample, It is a sample and is the similarity measure between them, and τ is the temperature parameter used to control the shape of the loss function.

[0082] S2. The central server selects multiple clients participating in the target driving prediction model training, and obtains the global neuron parameters of the target driving prediction model in the multiple clients. The central server aggregates the global neuron parameters to obtain fusion parameters. The central server distributes the fusion parameters to the multiple clients. The clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the clients to obtain a personalized target driving prediction model.

[0083] In the embodiment of the present invention, the central server refers to a server responsible for coordinating various clients to perform model training and updating.

[0084] As an embodiment of the present invention, the method of using the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client includes:

[0085] The following formula is used to optimize the personalized parameters of the target driving prediction model in the client:

[0086]

[0087] in, Personalize parameters for the target driving prediction model, is the original parameter of the target driving prediction model, η is the learning rate, Predict model gradients for target driving.

[0088] Exemplarily, the step of using the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client includes the following steps:

[0089] In the tth round of training, the central server randomly selects a group of participating clients C t In a group of participating clients C t A client k is randomly selected, and the central server selects a set of p from the global model k Active neurons, where p k ∈(0,1] is the proportion of active neurons, p k The value of p depends on the system characteristics of client k. More powerful clients k have larger p k , and the parameters A of these neurons k (w t ) is sent to k, and client k uses the received A k (w t ) Update the local model Get the intermediate model Then client k uses stochastic descent (SGD) to train the model Update to get a new model

[0090]

[0091] Among them, among them, Personalize parameters for the target driving prediction model, is the original parameter of the target driving prediction model, η is the learning rate, F k The gradient with respect to the active parameter, i.e.

[0092]

[0093] S3. The client receives the user's real-time driving data, and inputs the user's real-time driving data into the personalized target driving prediction model to perform driving prediction, thereby obtaining the user's driving prediction result.

[0094] In the embodiment of the present invention, the real-time driving data of the user refers to data that is acquired in real time and records the driving behavior of the user.

[0095] As an embodiment of the present invention, the step of inputting the user's real-time driving data into the personalized target driving prediction model to perform driving prediction and obtain the user's driving prediction result includes:

[0096] Obtaining personalized decision information from the user's real-time driving data, and performing feature encoding on the personalized decision information to obtain a personalized decision information feature matrix, wherein the personalized decision information includes system information, driving decision strategy information, and user driving prompt information;

[0097] Encoding the image information and LiDAR information in the real-time driving data of the user, and fusing the image information and LiDAR information to obtain real-time fusion information features;

[0098] Calculate the matching degree between the real-time fusion information feature and the personalized decision information feature matrix, and extract the personalized decision information corresponding to the personalized decision information feature matrix with the highest matching degree;

[0099] The corresponding driving decision strategy information is extracted from the personalized decision information with the highest matching degree to obtain the user driving prediction result.

[0100] In an embodiment of the present invention, the user driving prompt information may include user needs and driving road type information, etc.

[0101] Exemplarily, inputting the user's real-time driving data into the personalized target driving prediction model to perform driving prediction includes the following steps:

[0102] In order to make the model more in line with the driver's habits when making specific decisions, a personalized encoder momentum encoder is used. After the pre-built contrast learning driving prediction model is trained according to the multi-modal driving contrast feature data, the parameters are copied. Then use the local data to update the text encoder in the contrastive learning driving prediction model to obtain another parameter The momentum encoder is combined with two parameters to update the text encoder in the prediction model to obtain a personalized encoder.

[0103]

[0104] in, is the gradient calculated by the momentum encoder using local data at time t, is the final updated gradient of the momentum encoder at time t, and m is the momentum hyperparameter.

[0105] As an embodiment of the present invention, a driving control system based on a driving control method of personalized federated contrastive learning includes a central server and a plurality of clients respectively connected to the central server.

[0106] Furthermore, the client is arranged in a vehicle.

[0107] Furthermore, the central server obtains global neuron parameters of the target driving prediction models in the multiple clients, aggregates the global neuron parameters to obtain fusion parameters, and then distributes the fusion parameters to the multiple clients. The clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction models in the clients.

[0108] The present invention uses a preset multimodal processor to perform feature extraction on the driving data, so as to more comprehensively extract the category features of different data in the driving data, thereby ensuring the comprehensiveness of the extracted features. In addition, by performing data processing on the multimodal driving feature data through a preset contrastive learning algorithm, it can be ensured that the subsequent contrastive learning driving prediction model can make corresponding correct responses in different situations, thereby improving the robustness and adaptability of the system. In addition, a central server is used to distribute the fusion parameters to the multiple clients, and the clients receive the fusion parameters and use the fusion parameters to optimize the personalized parameters of the target driving prediction model in the client, so that the model can be more in line with the driver's driving habits and personalized needs and ensure the privacy and security of the user.

[0109] Reference Figure 1 , which is a schematic diagram of model training of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention.

[0110] Reference Figure 2 , which is a schematic diagram of a model decision process of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention.

[0111] Reference Figure 4 , which is a technical detail diagram of a multi-information fusion module of a decision model of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention.

[0112] Reference Figure 5 As shown, it is a framework flow chart of personalized federated learning of a driving control method based on personalized federated comparative learning provided by one embodiment of the present invention.

[0113] Reference Figure 6 As shown, it is a personalized encoder training flow chart of a driving control method based on personalized federated contrastive learning provided by an embodiment of the present invention.

[0114] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0115] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0116] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0117] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0118] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A driving control method based on personalized federated contrastive learning, characterized in that: The method comprises: The client obtains driving data, and uses a preset multimodal processor to extract features from the driving data to obtain multimodal driving feature data, wherein the driving data includes multi-view images, LiDAR data, and system information; The client uses a preset contrast learning algorithm to process the multimodal driving feature data to obtain multimodal driving contrast feature data, wherein the use of a preset multimodal processor to extract features from the driving data to obtain multimodal driving feature data includes: extracting features from the multi-view images in the driving data using a time Q-Former processing algorithm to obtain multi-view image features; extracting features from the LiDAR data in the driving data using an encoder, and converting the extracted features using a ternary processing algorithm to obtain LiDAR features; extracting features from the system information in the driving data using a transformer encoder to obtain system information features; extracting features from the multi-view images using a time Q-Former processing algorithm to obtain multi-view image features; extracting features from the LiDAR data in the driving data using an encoder, and converting the extracted features using a ternary processing algorithm to obtain LiDAR features; extracting features from the system information in the driving data using a transformer encoder to obtain system information features; The image feature, the LiDAR feature and the system information feature perform a feature fusion operation to obtain the multimodal driving feature data, wherein the multi-view image feature, the LiDAR feature and the system information feature perform a feature fusion operation to obtain the multimodal driving feature data, including: the LiDAR feature converted according to the ternary processing algorithm meets the fusion requirement for feature fusion with the multi-view image feature, the LiDAR feature is feature fused with the multi-view image feature to obtain a fused feature; the fused feature is feature aligned with the system information feature and a feature comparison matrix is ​​constructed; and feature extraction is performed from the feature comparison matrix to obtain the multimodal driving feature data; The client trains the pre-built comparative learning driving prediction model according to the multi-modal driving comparative feature data, and obtains the target driving prediction model after the training is completed; The central server selects a plurality of clients participating in the target driving prediction model training, and obtains the global neuron parameters of the target driving prediction model in the plurality of clients, and the central server aggregates the global neuron parameters to obtain fusion parameters; The central server distributes the fusion parameters to the multiple clients, and the clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client to obtain a personalized target driving prediction model; The client receives the user's real-time driving data, and inputs the user's real-time driving data into the personalized target driving prediction model to perform driving prediction, thereby obtaining the user's driving prediction result.

2. The driving control method based on personalized federated contrastive learning as claimed in claim 1, characterized in that: The extracting features of the multi-view images in the driving data by using a temporal Q-Former processing algorithm to obtain multi-view image features includes: Extract all multi-view images between the current timestamp and a preset historical timestamp; The multi-view image at the preset historical timestamp is subjected to feature extraction using the temporal Q-Former processing algorithm to obtain a multi-view image tag embedding at the preset historical timestamp; The multi-view image markers from the preset historical timestamp to the current timestamp are sequentially extracted and embedded to obtain the multi-view image features.

3. The driving control method based on personalized federated contrastive learning as claimed in claim 1, characterized in that: The extracting features of the LiDAR data in the driving data by using an encoder, and converting the extracted features by using a ternary processing algorithm to obtain LiDAR features, includes: Extracting features from the LiDAR data using an encoder to obtain LiDAR initial features; The LiDAR initial features are converted into features of an obstacle state, an obstacle-free state, and an uncertain obstacle state by using a ternary processing algorithm to obtain the LiDAR features.

4. The driving control method based on personalized federated contrastive learning as claimed in claim 1, characterized in that: The step of inputting the user's real-time driving data into the personalized target driving prediction model to perform driving prediction and obtain the user's driving prediction result includes: Obtaining personalized decision information from the user's real-time driving data, and performing feature encoding on the personalized decision information to obtain a personalized decision information feature matrix, wherein the personalized decision information includes system information, driving decision strategy information, and user driving prompt information; Encoding the image information and LiDAR information in the real-time driving data of the user, and fusing the image information and LiDAR information to obtain real-time fusion information features; Calculate the matching degree between the real-time fusion information feature and the personalized decision information feature matrix, and extract the personalized decision information corresponding to the personalized decision information feature matrix with the highest matching degree; The corresponding driving decision strategy information is extracted from the personalized decision information with the highest matching degree to obtain the user driving prediction result.

5. The driving control method based on personalized federated contrastive learning as claimed in claim 1, characterized in that: The using of the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client includes: The following formula is used to optimize the personalized parameters of the target driving prediction model in the client: in, Personalize parameters for the target driving prediction model, is the original parameter of the target driving prediction model, η is the learning rate, Predict model gradients for target driving.

6. A driving control system based on the driving control method based on personalized federated contrastive learning according to any one of claims 1 to 5, characterized in that: It includes a central server and multiple clients respectively connected to the central server.

7. The driving control system according to claim 6, characterized in that: The client is arranged in the vehicle.

8. The driving control system according to claim 6, characterized in that: The central server obtains the global neuron parameters of the target driving prediction model in the multiple clients, aggregates the global neuron parameters to obtain fusion parameters, and then distributes the fusion parameters to the multiple clients. The clients receive the fusion parameters and use the fusion parameters to perform personalized parameter optimization on the target driving prediction model in the client.

Images