A heavy truck driving behavior recognition method and system based on vehicle-cloud cooperation

Abstract

The application discloses a heavy truck driving behavior recognition method and system based on vehicle-cloud cooperation, and relates to the technical field of intelligent transportation. The method comprises the following steps: a vehicle-end system collects multi-modal driving data, and obtains gradients by processing the data with a local model; a cloud-end system aggregates multiple vehicle-end gradients to update a global model, and distributes the global model to the vehicle-end to update the local model. Before aggregation, the cloud-end aligns multi-source data across domains and heterogeneously, eliminates domain distribution differences by minimizing maximum mean difference and adversarial training; when aggregating, a dynamic weighting strategy is adopted to balance the contribution of each vehicle-end system. The cloud-end further runs a deep multi-modal fusion model to perform spatio-temporal modeling and risk prediction, and the results are distributed to the vehicle-end for early warning. Through vehicle-cloud cooperation and federated learning, the application protects data privacy and overcomes the limitation of vehicle-end computing power, solves the problem of insufficient model generalization caused by data distribution differences through a cross-domain alignment mechanism, and improves the precision and safety of heavy truck driving behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and driving behavior analysis technology, and more specifically to a method and system for recognizing the driving behavior of heavy-duty vehicles based on vehicle-cloud collaboration. Background Technology

[0002] Heavy-duty vehicles play a crucial role in logistics, transportation, and mining operations, but their accident rate and severity are significantly higher than those of ordinary vehicles. Current safety management relies heavily on post-accident monitoring and experience-based management, lacking real-time, accurate methods for identifying driving behavior and providing risk warnings. With the development of vehicle-to-everything (V2X) and intelligent sensing technologies, multimodal data collection has become possible, but faces three major technical bottlenecks: first, data privacy and data silos prevent centralized training of data from different fleets; second, limited computing power on the vehicle side makes it difficult to deploy high-precision models; and third, significant differences in cross-domain data distribution hinder model generalization capabilities.

[0003] Therefore, there is an urgent need for a driving behavior recognition method that can achieve distributed collaborative modeling, adapt to vehicle-side computing power limitations, and have cross-domain adaptability while protecting privacy. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a heavy vehicle driving behavior recognition method based on vehicle-cloud collaboration to overcome or at least partially solve the above problems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution, specifically including the following steps:

[0006] S1. The vehicle-side system collects multimodal driving data, uses a local model to process the multimodal driving data in real time to obtain local recognition results, and extracts model gradients and feature representations of the driving data. S2. The cloud system receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems. S3. The cloud system aggregates the model gradients based on the federated learning algorithm to update the global model; at the same time, it runs a deep multimodal fusion model on the feature representations after cross-domain heterogeneous alignment to generate deep recognition results. S4. The updated global model parameters are sent to the vehicle-side system, and the vehicle-side system updates its local model based on the updated global model parameters; the cloud system sends the depth recognition results to the vehicle-side system. S5. According to the preset dynamic weighting strategy, the local recognition result is fused with the deep recognition result to generate the final decision result for real-time warning or driving intervention.

[0007] Preferably, the cross-domain heterogeneous alignment in step S2 includes: achieving it by minimizing the maximum mean difference (MMD).

[0008] in, This represents the feature map in the reproducing kernel Hilbert space. and These represent the number of samples in the source domain and the target domain, respectively. Represents the first term of the source domain One sample; Represents the first of the target domains One sample; The symbol representing the regenerated kernel Hilbert space.

[0009] Preferably, the cross-domain heterogeneous alignment step further introduces an adversarial training mechanism to further optimize the feature alignment effect by minimizing the adversarial loss function:

[0010] in Indicates a feature extractor; Representation domain discriminator; Represents the adversarial loss function. and Losses in the game; Feature extractor corresponding MMD loss; This represents a tradeoff coefficient used to balance the contribution of feature alignment to adversarial constraints.

[0011] Preferably, in step S3, the global optimization objective function of the federated learning algorithm is:

[0012] in, Represents global model parameters; Indicates the number of vehicle-side systems; Indicates the first Local model parameters for each vehicle-side system; Indicates the first Local loss function for each vehicle-side system; This represents the dynamic penalty parameter.

[0013] Preferably, the local model and / or the global model are multi-task learning models, and their overall loss function is:

[0014] in, Indicates the total number of tasks; Indicates the first The weighting coefficients for each task; This represents the loss function corresponding to each task.

[0015] Preferably, in step S3, a sparse vehicle-side system personalized aggregation algorithm is used, and the aggregation weight of each vehicle-side system is dynamically adjusted according to its activity, data quality, and task relevance:

[0016]

[0017] in, This represents the aggregated global model parameters; Indicates the first Local model parameters for each vehicle-side system; This represents the total number of vehicle-side systems. For the first Aggregated weight of individual vehicle-side systems This represents the task relevance coefficient; This is a data confidence index.

[0018] Preferably, the step S3 of running the deep multimodal fusion model to generate the deep recognition result includes the following sub-steps: Step A: The cloud system receives the multimodal feature representation after cross-domain heterogeneous alignment in step S2. The multimodal feature representation includes visual embedding features, inertial temporal features, and vehicle state vector. Step B: Perform a bird's-eye view spatial transformation on the visual embedding features and map them to a unified BEV space to obtain geometrically structured spatial semantic features. Step C: Perform time-series modeling on the spatial semantic features, inertial temporal features, and vehicle state vectors. Capture the dynamic evolution of long-term driving behavior through a hierarchical time window attention mechanism and output time-enhanced feature representation. Step D: The visual embedding features, inertial temporal features, and vehicle state vector in the temporal enhancement feature representation are weighted and interacted through a cross-modal attention mechanism to generate fused semantic features; Step E: Construct a relationship graph between driving behavior and environmental factors, using the fused semantic features as node attributes, determining edge weights based on feature similarity and temporal dependence, and realizing joint reasoning of behavioral patterns and environmental semantics through the message passing mechanism of graph convolutional networks to output deep recognition results. Step F: Send the depth recognition results to the corresponding vehicle-side system.

[0019] Preferably, in step S4, a federated knowledge distillation mechanism is used for knowledge transfer between the cloud system and the vehicle system, with the deep multimodal fusion model serving as the teacher model, periodically guiding local model updates through distillation. The distillation loss function is:

[0020] in Represents cross-entropy loss, This represents the Kullback–Leibler divergence between the predicted distributions of teachers and students. Indicates the balance coefficient; This represents the predicted probability distribution of the deep multimodal fusion model; This represents the predicted probability distribution of the local model.

[0021] Preferably, the method for determining the dynamic fusion weights in step S5 includes: When the network state parameters are lower than the first preset threshold, the fusion weight of the deep recognition results is reduced; when the confidence level of the local model output is lower than the second preset threshold, the fusion weight of the deep recognition results is increased.

[0022] Based on the same inventive concept, this invention also discloses a heavy-duty vehicle driving behavior recognition system based on vehicle-cloud collaboration, used to implement any of the above methods, including: Vehicle-side system, used for: Multimodal driving data is collected, and the local model is used to process the multimodal driving data in real time to obtain local recognition results. The model gradient and feature representation of the driving data are extracted. Receive updated global model parameters and depth recognition results from the cloud system; Update the local model based on the updated global model parameters; According to a preset dynamic weighting strategy, the local recognition results and the deep recognition results are fused to generate a final decision result for real-time warning or driving intervention. Cloud systems are used for: It receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems; The model gradients are aggregated using a federated learning algorithm to update the global model; A deep multimodal fusion model is run on the feature representations after cross-domain heterogeneous alignment to generate deep recognition results; The updated global model parameters and the depth recognition results are sent to the corresponding vehicle-side system.

[0023] The beneficial effects of the above-mentioned technical solution provided by the present invention include at least the following: By constructing a vehicle-cloud collaborative federated learning architecture, only model gradients, rather than raw driving data, are uploaded to the vehicle. The cloud aggregates gradients, updates the global model, and distributes it to the vehicle. This achieves distributed joint modeling of multiple vehicle fleets while protecting data privacy, effectively solving the data silo problem. At the same time, deploying complex model training in the cloud overcomes the bottleneck of limited computing power on the vehicle. On this basis, the cloud introduces a cross-domain heterogeneous alignment step before aggregating gradients to distribute and align the feature representations uploaded by different vehicles, eliminating data distribution differences between different vehicles and driving environments, and solving the problem of insufficient generalization ability of the global model caused by cross-domain data heterogeneity. Furthermore, through hierarchical collaboration between real-time recognition by the vehicle's local model and high-precision recognition by the cloud's deep multimodal fusion model, and by using a dynamic weight strategy to fuse the local and cloud recognition results to generate the final decision, the advantages of real-time response on the vehicle and high-precision spatiotemporal modeling on the cloud are combined. Thus, a reliable heavy-duty vehicle driving behavior recognition and safety warning system is built without relying on a continuous network connection. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a flowchart of a heavy-duty vehicle driving behavior recognition method based on vehicle-cloud collaboration provided in an embodiment of the present invention; Figure 2 This is a framework diagram of a heavy-duty vehicle driving behavior recognition system based on vehicle-cloud collaboration provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown in the figure, this invention discloses a method for recognizing the driving behavior of heavy-duty vehicles based on vehicle-cloud collaboration, including the following steps: S1. The vehicle-side system collects multimodal driving data, uses a local model to process the multimodal driving data in real time to obtain local recognition results, and extracts model gradients and feature representations of driving data. S2. The cloud system receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems. S3, the cloud system aggregates model gradients based on federated learning algorithms to update the global model; at the same time, it runs a deep multimodal fusion model on the cross-domain heterogeneous aligned feature representations to generate deep recognition results; S4. The updated global model parameters are sent to the vehicle system, and the vehicle system updates its local model based on the updated global model parameters; the cloud system sends the depth recognition results to the vehicle system. S5. Based on the preset dynamic weighting strategy, the local recognition results and deep recognition results are fused to generate the final decision result for real-time warning or driving intervention.

[0028] In one embodiment, for step S1, the vehicle-side system integrates a multimodal perception unit for collecting multi-source data during the driving process. The multimodal perception unit includes at least a visual sensor (such as an onboard camera), an inertial measurement unit (IMU), and a vehicle bus interface (such as a CAN bus), acquiring driving scene images, vehicle motion states (acceleration, angular velocity), and internal vehicle signals (vehicle speed, brake pedal opening, steering angle, etc.), respectively. By fusing visual, inertial, and vehicle dynamic data, the perception limitations of a single sensor under conditions of changing lighting, occlusion, or noise can be effectively overcome, providing robust input features for subsequent behavior recognition.

[0029] On the vehicle-side system side, the model architecture employs a deep network design that balances lightweightness and efficiency. MobileFormer and Edge-Swin Transformer are selected as the basic model architecture. MobileFormer combines the efficient deep separable convolutions of MobileNet with the global attention mechanism of Transformer, maintaining high recognition accuracy on low-computing-power terminals. Edge-Swin Transformer introduces a local window attention mechanism and a multi-scale feature modeling strategy, enhancing its ability to capture spatiotemporal variations in driving behavior. To reduce communication load, the local model only uploads model gradients or partial parameter updates, rather than the complete model weights.

[0030] The optimization objective of the vehicle-side system (i.e., the vehicle-end system) can be expressed as:

[0031] in, Indicates the first The local loss function of the vehicle-side system For local model parameters, For task loss items, This represents the number of local samples. , The first Input and label for each sample.

[0032] In one embodiment, the cross-domain heterogeneous alignment in step S2 includes: achieving it by minimizing the maximum mean difference (MMD).

[0033] in, This represents the feature map in the reproducing kernel Hilbert space. and These represent the number of samples in the source domain and the target domain, respectively. Represents the first term of the source domain One sample; Represents the first of the target domains One sample; The symbol representing the regenerated kernel Hilbert space.

[0034] Specifically, considering the statistical heterogeneity and task offset of data distribution among different vehicles, drivers, and fleets, this invention introduces a cross-domain heterogeneous alignment step. This step aims to achieve semantic unification and distribution reconstruction of data from different sources, thereby improving the generalization ability and robustness of the global model while ensuring privacy protection.

[0035] At the feature space level, a domain adaptation and multi-domain feature sharing mechanism is introduced. By constraining the high-dimensional feature distribution of samples from different domains through the maximum mean difference constraint, the statistical difference between cross-domain feature means is minimized, thereby achieving the unification of semantic features across domains. The MMD constraint can be expressed as:

[0036] In one embodiment, the cross-domain heterogeneous alignment step also introduces an adversarial training mechanism to further optimize the feature alignment effect by minimizing the adversarial loss function:

[0037] in Indicates a feature extractor; Representation domain discriminator; Represents the adversarial loss function. and Losses in the game; Feature extractor corresponding MMD loss; This represents a tradeoff coefficient used to balance the contribution of feature alignment to adversarial constraints.

[0038] Specifically, to further improve semantic layer alignment capabilities, an adversarial discriminator is embedded in the feature extraction network to construct a feature distribution matching mechanism based on adversarial training. The discriminator minimizes the adversarial loss. This forces the features of different domains to become indistinguishable, thereby enhancing cross-domain consistency. The joint optimization objective can be defined as:

[0039] Furthermore, a domain-shared hypernetwork-domain-specific adapter structure is introduced from the HAMUR model, enabling models corresponding to different fleets, regions, and driving styles to possess differentiated adaptability. The hypernetwork is responsible for learning cross-domain shared knowledge, while the adapter generates parameters for a specific domain, thereby achieving efficient and loosely coupled multi-domain feature alignment and transfer within the federated framework.

[0040] In one embodiment, in step S3, the global optimization objective function of the federated learning algorithm is:

[0041] in, Global model parameters; Number of vehicle-side systems; Indicates the first Local model parameters for each vehicle-side system; Indicates the first Local loss function for each vehicle-side system; This represents the dynamic penalty parameter.

[0042] The cloud system is responsible for receiving model gradients (or partial parameter updates) from multiple vehicle-side systems and using improved FedProx and pFedMe algorithms for global aggregation and parameter optimization. FedProx introduces a local regularization term into the traditional FedAvg framework, dynamically penalizing parameters. To suppress model drift in non-independent and identically distributed (Non-IID) data, the pFedMe algorithm, based on Moreau Envelope theory, introduces a two-layer optimization mechanism to achieve personalized modeling. Combining these two ideas, the optimization objective in the cloud can be defined as:

[0043] This optimization strategy achieves a balance between global consistency and individual adaptability, significantly improving the convergence and generalization performance of the model under heterogeneous data conditions.

[0044] In one embodiment, the local model and / or the global model is a multi-task learning model, and its overall loss function is:

[0045] in, Indicates the total number of tasks; Indicates the first The weighting coefficients for each task; This represents the loss function corresponding to each task.

[0046] Specifically, to achieve the multi-task learning objective, both the local and global models can employ a joint optimization mechanism of a shared feature extraction layer and a task-specific output layer. The shared layer is used to extract general driving feature representations, while the task layers are optimized for sub-tasks such as driving behavior recognition, risk prediction, and driving state assessment.

[0047] Taking the local model of the vehicle-side system as an example, considering the limitations of onboard computing power, its shared layer adopts a lightweight network architecture, such as MobileFormer or Edge-Swin Transformer, while the task layer outputs the probability of driving behavior categories, risk scores, and driver state levels, respectively. During training, the local model uses local data to optimize the above multi-task loss function, achieving joint learning of multiple tasks.

[0048] The cloud-based global model is also designed with a multi-task learning architecture to maintain structural consistency with the local model, facilitating parameter distribution and knowledge transfer. The shared layer of the cloud-based global model can employ more complex networks, such as ResNet or Transformer, and the task heads can be the same as or more refined than those of the local model, for example, adding tasks such as risk evolution trend prediction. During federated aggregation, the cloud aggregates model gradients from each vehicle, updating the multi-task parameters of the global model, thereby improving the generalization ability of each task.

[0049] There are T learning tasks in total, including driving behavior recognition, risk prediction, and driver state assessment. Each task... Corresponding to a loss function For example, cross-entropy loss is used for classification tasks, and mean squared error is used for regression tasks. The overall loss function is a weighted sum of the losses from each task. The weighting coefficient Adjustments can be made based on the importance of the task or the training dynamics.

[0050] This design not only enables knowledge sharing and complementarity among tasks, but also effectively improves modeling efficiency and computational resource utilization. Ultimately, the system achieves joint learning of multi-source distributed data while protecting data privacy, providing a reliable algorithmic and architectural foundation for subsequent vehicle-cloud collaborative learning and dynamic knowledge transfer.

[0051] In one embodiment, in step S3, a sparse vehicle-side system personalized aggregation algorithm is adopted at the model aggregation level, and the aggregation weight of each vehicle-side system is dynamically adjusted according to its activity, data quality, and task relevance:

[0052]

[0053] in, This represents the aggregated global model parameters; Indicates the first Local model parameters for each vehicle-side system; This represents the total number of vehicle-side systems. For the first Aggregated weight of individual vehicle-side systems This represents the task relevance coefficient; This is a data confidence index.

[0054] Specifically, at the model aggregation level, a multi-domain parameter sharing and task-weighted aggregation mechanism is proposed. Different vehicle-side systems dynamically adjust the global aggregation weights based on their data confidence, sample size, and task relevance. The optimization objective can be formalized as follows:

[0055]

[0056] Simultaneously, the AUTOSEM algorithm framework is introduced, which dynamically determines the importance of each task and the joint training ratio through automatic task selection and mixture ratio optimization mechanisms. This mechanism can achieve adaptive balancing among different tasks, improving the global model's collaborative optimization capability and multi-task convergence efficiency.

[0057] By combining the personalized modeling module, each vehicle-side system can make local fine-tuning based on shared parameters, forming a two-layer model structure that combines "shared" and "personalized" approaches. This is achieved by introducing a small number of locally adapted parameters. The vehicle-side system's local model can adaptively optimize based on its own driving style and operating condition characteristics, achieving cross-domain knowledge transfer and federated distillation. The cloud periodically calculates the similarity matrix and collaborative gain index between different domain models, and dynamically adjusts the proportion of shared parameters based on this to balance global consistency and local adaptability.

[0058] In one embodiment, step S3, which involves running a deep multimodal fusion model to generate deep identification results and risk evolution trends, includes the following sub-steps: Step A: The cloud system receives the multimodal feature representation after cross-domain heterogeneous alignment in step S2. The multimodal feature representation includes visual embedding features, inertial temporal features, and vehicle state vector. Step B: Perform a bird's-eye view spatial transformation on the visual embedding features and map them to a unified BEV space to obtain geometrically structured spatial semantic features. Step C: Perform temporal modeling of spatial semantic features, inertial temporal features, and vehicle state vectors. Capture the dynamic evolution of long-term driving behavior through a hierarchical time window attention mechanism and output a time-enhanced feature representation. Step D: Through a cross-modal attention mechanism, the visual embedding features, inertial temporal features, and vehicle state vector in the temporal enhancement feature representation are weighted and interacted to generate fused semantic features; Step E: Construct a graph showing the relationship between driving behavior and environmental factors. Use fused semantic features as node attributes and feature similarity and temporal dependence to determine edge weights. Achieve joint reasoning of behavioral patterns and environmental semantics through the message passing mechanism of graph convolutional networks, and output deep recognition results. Step F: Send the depth recognition results to the corresponding vehicle-side system.

[0059] Specifically, deep multimodal fusion models include: The BEVFormer-XL module is used to map visual features to a bird's-eye view space to obtain unified spatial semantic features. The Temporal Swin Transformer module is used to capture the dynamic evolution of long-term driving behavior and output temporally enhanced feature representations. The cross-modal attention fusion module is used to perform weighted interaction on multimodal data features to generate fused semantic features; The graph convolutional network module is used to construct a graph showing the relationship between driving behavior and environmental factors, enabling joint reasoning of behavioral patterns and environmental semantics.

[0060] The cloud-based system leverages high-performance computing power and big data resources to construct a deep multimodal driving behavior recognition model based on a fusion architecture of BEVFormer-XL and TemporalSwin Transformer. This model, with cloud-based multimodal feature fusion and global spatiotemporal modeling at its core, integrates three types of inputs: visual images, inertial sensing, and vehicle bus signals. This enables joint modeling of cross-modal semantic associations and temporal dependencies, thereby achieving refined driving behavior recognition and dynamic risk prediction in complex traffic environments and multi-condition scenarios.

[0061] At the spatial modeling level, the BEVFormer-XL module maps multi-view visual features to a bird's-eye view (BEV) space, forming structured spatial semantic features to enhance spatial perception capabilities of vehicles, roads, and environmental objects. This module utilizes a learnable projection function. This enables a unified mapping from image planar features to the BEV space, thereby reducing the impact of viewpoint bias on feature fusion.

[0062] At the temporal modeling level, the Temporal Swin Transformer employs a hierarchical temporal window mechanism to perform sliding modeling of multi-frame features, capturing the dynamic evolution of long-term driving behavior through multi-scale temporal attention. Its temporal feature modeling can be formalized as follows:

[0063] in Indicates the first Temporal embedding representation of time points, This represents the window length. This structure significantly improves the model's sensitivity to changes in the continuity of driving operations and potential risks.

[0064] In the feature fusion stage, the cloud system introduces a cross-modal attention mechanism (CMA) to weight and interact the visual embedding features, inertial temporal features, and vehicle state vectors in the temporally enhanced feature representation, thereby highlighting the decision contribution of key modalities. The fusion process can be expressed as follows:

[0065] in / / These represent query, key, and value vectors from different modalities. This is the global semantic representation after fusion.

[0066] In the relationship modeling stage, the model further utilizes graph convolutional networks to construct a relationship graph between driving behavior and environmental factors. Nodes include driving actions, vehicle posture, and external scene elements, and edge weights are determined by feature similarity and temporal dependency. GCN achieves joint reasoning of behavioral patterns and environmental semantics through a message passing mechanism, outputting deep recognition results of driving behavior.

[0067] Meanwhile, in addition to classifying driving behavior, the deep multimodal fusion model in the cloud system can also calculate the risk evolution trend function based on short-time series features. It is used to predict the rate of increase in driving risks in advance, and to achieve minute-level danger warnings.

[0068] In one embodiment, in step S4, a federated knowledge distillation mechanism is used for knowledge transfer between the cloud system and the vehicle system. The deep multimodal fusion model serves as the teacher model, and periodically guides the local model update through distillation. The distillation loss function is:

[0069] in Represents cross-entropy loss, This represents the Kullback–Leibler divergence between the predicted distributions of teachers and students. Indicates the balance coefficient; This represents the predicted probability distribution of the deep multimodal fusion model; This represents the predicted probability distribution of the local model.

[0070] Specifically, a vehicle-cloud collaborative optimization-driven bidirectional knowledge transfer technology is proposed to establish a closed-loop system for dynamic learning and continuous evolution. This technology aims to achieve efficient knowledge sharing and adaptive optimization between the vehicle and the cloud. Through mechanisms such as encrypted gradient synchronization, differential privacy protection, and federated knowledge distillation, it completes the secure transfer and multi-layer fusion of model parameters and feature knowledge, constructing an intelligent cycle of perception-learning-feedback-relearning.

[0071] During the uplink phase, the vehicle-side system operates according to the local training cycle. Regularly upload encrypted gradients ,in For local gradients, Noise terms are injected based on a differential privacy mechanism to prevent reverse reasoning attacks. The vehicle-side simultaneously extracts sparse feature embeddings and uploads only high-confidence features to reduce communication load. After receiving uploads from multiple vehicle fleets, the cloud performs global aggregation and dynamic model updates using a deep multimodal fusion model. The aggregation process can be formalized as follows:

[0072] in Indicates a global parameter. For learning rate, Aggregation weights are adaptively assigned based on data quality and node confidence.

[0073] During the downlink phase, the cloud will optimize the global model parameters. The data is then distributed to the vehicles, where each vehicle performs lightweight retraining and adaptive calibration based on local data distribution and task differences. Through fine-tuning of local parameters, the model can enhance its adaptability to individual driving styles and vehicle states without compromising global consistency.

[0074] To ensure the stability and security of knowledge transfer, the system incorporates end-to-end encryption and differential privacy protection mechanisms during transmission to prevent the theft or reverse engineering of original driving data or model gradients. At the knowledge level, a federated knowledge distillation framework is adopted, utilizing soft labels output by the cloud-based teacher model (i.e., the deep multimodal fusion model) to guide the updating of the vehicle-side student model (i.e., the local model), achieving cross-device knowledge sharing without original data. Its optimization objective is:

[0075] The vehicle provides real-world driving characteristics and behavioral priors through continuous perception, while the cloud optimizes the vehicle-side model performance through large-scale aggregation and model distillation. Over long-term operation, the model develops self-evolution capabilities, enabling rapid convergence and adaptive optimization based on new environments and tasks, thereby constructing an intelligent collaborative system with continuous learning and generalization capabilities.

[0076] In one embodiment, the method for determining the dynamic fusion weights in step S5 includes: When the network state parameters are lower than the first preset threshold, the fusion weight of the deep recognition results is reduced; when the confidence of the local model output is lower than the second preset threshold, the fusion weight of the deep recognition results is increased.

[0077] Specifically, the vehicle-side system fuses the cloud-based output with the local recognition results based on dynamic fusion weights to form the final decision. The dynamic fusion weights depend on the network state or the confidence level of the local model output.

[0078] It is important to note that in this method, vehicle-side recognition and cloud-based recognition do not compete on the same time scale, but rather cooperate and complement each other across different time windows. The vehicle-side local model achieves millisecond-level real-time response, ensuring immediate triggering of warnings or interventions in emergencies. The cloud-based deep multimodal fusion model, due to its large-scale computation and multi-vehicle data aggregation, has inherent latency ranging from seconds to minutes, but it offers higher recognition accuracy and can capture long-term risk evolution trends. The two are connected on the timeline through dynamic fusion weights, forming a two-layer decision-making system.

[0079] The specific interaction process is as follows: (1) Vehicle-side real-time layer: The vehicle-side system continuously collects multimodal data, and the local model outputs recognition results in real time. If a high-risk behavior is detected, an audible and visual alarm or a brief intervention (such as speed reduction) is immediately activated to ensure the safety baseline. At the same time, the vehicle-side extracts the model gradient and feature representation and uploads it to the cloud.

[0080] (2) Cloud-based deep layer: The cloud receives feature representations from multiple vehicle terminals, performs cross-domain heterogeneous alignment, and runs a deep multimodal fusion model to perform spatiotemporal modeling, generating a deep recognition result. This result is timestamped, reflecting the comprehensive judgment within a window over a past period.

[0081] (3) Distribution and Fusion: The cloud distributes the deep recognition results to the corresponding vehicle terminal. After receiving the results, the vehicle terminal dynamically calculates the fusion weight based on the current network status (such as RTT latency and packet loss rate) and the confidence level of the local model's output on the current driving behavior. For example, if the network latency is high, the credibility of the cloud result decreases, and the weight of the vehicle terminal is reduced; if the confidence level of the local model on the current scene is low (such as blurry images or unknown working conditions), the weight of the cloud result is increased.

[0082] (4) Final Decision: The vehicle-side system performs a weighted fusion of the local real-time results and the cloud-based delayed results based on dynamic fusion weights to generate the final decision result. If the cloud-based results correct local misjudgments (such as misreporting normal driving as fatigue), the vehicle-side system can deactivate the triggered alarm; if the cloud-based results supplement local missed reports (such as identifying an upward trend in fatigue risk), the vehicle-side system can issue a supplementary warning. Although this correction process has a delay of up to a second, it occurs much earlier than the critical time of the accident and can continuously optimize the online learning capability of the local model.

[0083] like Figure 2 As shown, based on the same inventive concept, this embodiment of the invention also provides a heavy-duty vehicle driving behavior recognition system based on vehicle-cloud collaboration, including: Vehicle-side system, used for: Collect multimodal driving data, use a local model to process the multimodal driving data in real time to obtain local recognition results, and extract model gradients and feature representations; Receive updated global model parameters and depth recognition results from the cloud system; Update the local model based on the updated global model parameters; Based on a preset dynamic weighting strategy, the local recognition results are fused with the deep recognition results to generate the final decision result for real-time warning or driving intervention. Cloud systems are used for: It receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems; Federated learning algorithms are used to aggregate model gradients to update the global model; Run a deep multimodal fusion model on the cross-domain heterogeneous aligned feature representations to generate deep recognition results; The updated global model parameters and depth recognition results are sent to the corresponding vehicle-side system.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing the driving behavior of heavy-duty vehicles based on vehicle-cloud collaboration, characterized in that, Includes the following steps: S1. The vehicle-side system collects multimodal driving data, uses a local model to process the multimodal driving data in real time to obtain local recognition results, and extracts model gradients and feature representations of the driving data. S2. The cloud system receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems. S3. The cloud system aggregates the model gradients based on the federated learning algorithm to update the global model; at the same time, it runs a deep multimodal fusion model on the feature representations after cross-domain heterogeneous alignment to generate deep recognition results. S4. The updated global model parameters are sent to the vehicle-side system, and the vehicle-side system updates the local model based on the updated global model parameters; The cloud system sends the depth recognition results to the vehicle system; S5. According to the preset dynamic weighting strategy, the local recognition result is fused with the deep recognition result to generate the final decision result for real-time warning or driving intervention.

2. The method according to claim 1, characterized in that, The cross-domain heterogeneous alignment in step S2 includes: achieving it by minimizing the maximum mean difference (MMD). in, This represents the feature map in the reproducing kernel Hilbert space. and These represent the number of samples in the source domain and the target domain, respectively. Represents the first term of the source domain One sample; Represents the first of the target domains One sample; The symbol representing the regenerated kernel Hilbert space.

3. The method according to claim 2, characterized in that, The cross-domain heterogeneous alignment step also introduces an adversarial training mechanism to further optimize the feature alignment effect by minimizing the adversarial loss function. in Indicates a feature extractor; Representation domain discriminator; Represents the adversarial loss function. and Losses in the game; Feature extractor corresponding MMD loss; This represents a tradeoff coefficient used to balance the contribution of feature alignment to adversarial constraints.

4. The method according to claim 1, characterized in that, In step S3, the global optimization objective function of the federated learning algorithm is: in, Represents global model parameters; Indicates the number of vehicle-side systems; Indicates the first Local model parameters for each vehicle-side system; Indicates the first Local loss function for each vehicle-side system; This represents the dynamic penalty parameter.

5. The method according to claim 1, characterized in that, The local model and / or the global model are multi-task learning models, and their overall loss function is: in, Indicates the total number of tasks; Indicates the first The weighting coefficients for each task; This represents the loss function corresponding to each task.

6. The method according to claim 1, characterized in that, In step S3, a personalized aggregation algorithm for sparse vehicle-end systems is adopted, and the aggregation weight of each vehicle-end system is dynamically adjusted according to its activity, data quality, and task relevance: in, This represents the aggregated global model parameters; Indicates the first Local model parameters for each vehicle-side system; This represents the total number of vehicle-side systems. For the first Aggregated weight of individual vehicle-side systems This represents the task relevance coefficient; This is a data confidence index.

7. The method according to claim 1, characterized in that, Step S3, which involves running the deep multimodal fusion model to generate deep recognition results, includes the following sub-steps: Step A: The cloud system receives the multimodal feature representation after cross-domain heterogeneous alignment in step S2. The multimodal feature representation includes visual embedding features, inertial temporal features, and vehicle state vector. Step B: Perform a bird's-eye view spatial transformation on the visual embedding features and map them to a unified BEV space to obtain geometrically structured spatial semantic features. Step C: Perform time-series modeling on the spatial semantic features, inertial temporal features, and vehicle state vectors. Capture the dynamic evolution of long-term driving behavior through a hierarchical time window attention mechanism and output time-enhanced feature representation. Step D: The visual embedding features, inertial temporal features, and vehicle state vector in the temporal enhancement feature representation are weighted and interacted through a cross-modal attention mechanism to generate fused semantic features; Step E: Construct a relationship graph between driving behavior and environmental factors, using the fused semantic features as node attributes, determining edge weights based on feature similarity and temporal dependence, and realizing joint reasoning of behavioral patterns and environmental semantics through the message passing mechanism of graph convolutional networks to output deep recognition results. Step F: Send the depth recognition results to the corresponding vehicle-side system.

8. The method according to claim 1, characterized in that, In step S4, the cloud system and the vehicle system use a federated knowledge distillation mechanism for knowledge transfer. The deep multimodal fusion model serves as the teacher model, and periodically guides the local model updates through distillation. The distillation loss function is: in Represents cross-entropy loss, This represents the Kullback–Leibler divergence between the predicted distributions of teachers and students. Indicates the balance coefficient; This represents the predicted probability distribution of the deep multimodal fusion model; This represents the predicted probability distribution of the local model.

9. The method according to claim 1, characterized in that, The methods for determining the dynamic fusion weights in step S5 include: When the network state parameters are lower than the first preset threshold, the fusion weight of the deep recognition results is reduced; when the confidence level of the local model output is lower than the second preset threshold, the fusion weight of the deep recognition results is increased.

10. A heavy-duty vehicle driving behavior recognition system based on vehicle-cloud collaboration, used to implement the method described in any one of claims 1-9, characterized in that, include: Vehicle-side system, used for: Multimodal driving data is collected, and the local model is used to process the multimodal driving data in real time to obtain local recognition results. The model gradient and feature representation of the driving data are extracted. Receive updated global model parameters and depth recognition results from the cloud system; Update the local model based on the updated global model parameters; According to a preset dynamic weighting strategy, the local recognition results and the deep recognition results are fused to generate a final decision result for real-time warning or driving intervention. Cloud systems are used for: It receives model gradients from multiple vehicle-end systems and performs cross-domain heterogeneous alignment of feature representations from different vehicle-end systems; The model gradients are aggregated using a federated learning algorithm to update the global model; A deep multimodal fusion model is run on the feature representations after cross-domain heterogeneous alignment to generate deep recognition results; The updated global model parameters and the depth recognition results are sent to the corresponding vehicle-side system.

Images