A method and system for fatigue identification and early warning of safety officers in cloud-based cabins under multi-vehicle monitoring scenarios

Abstract

This invention relates to the field of remote driving safety monitoring technology for intelligent vehicles, and in particular to a method and system for fatigue identification and early warning of safety personnel in a cloud-based monitoring scenario. The method includes constructing a differentiated sub-model system for different driver-vehicle ratio scenarios using an LSTM model-attention mechanism to generate personalized fatigue identification sub-models adapted to the fatigue patterns under each driver-vehicle ratio scenario; each personalized fatigue identification sub-model extracts fatigue features from corresponding multimodal data and outputs sub-fatigue identification results; the parameter information of each personalized fatigue identification sub-model is fused through multi-task federated learning to generate a global fatigue identification model; and the fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model are integrated through the global fatigue identification model to obtain a comprehensive fatigue identification result. This invention can improve the accuracy and robustness of fatigue identification and reduce the risk of remote driving monitoring failure due to fatigue.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle remote driving safety monitoring technology, specifically a cloud-based cabin safety officer fatigue identification and early warning method and system in a multi-vehicle monitoring scenario. Background Technology

[0002] The core objective of research on remote monitoring and safety assurance for intelligent vehicles is to accurately identify and warn of the fatigue state of cloud-based safety operators in multi-vehicle monitoring scenarios by developing advanced methods and tools. This will enable targeted measures to ensure the safety of remote driving operations of intelligent vehicles. Fatigue identification and early warning technology is a crucial component of remote driving safety monitoring for intelligent vehicles. Its main task is to quantify and understand the changing patterns of fatigue characteristics of cloud-based safety operators in dynamic switching scenarios with different human-vehicle ratios, providing scientific support for remote monitoring.

[0003] Among these, fatigue identification and early warning methods that adapt to dynamic changes in the human-vehicle ratio are complex technologies aimed at improving driving safety in multi-vehicle monitoring scenarios. The nature of the work of cloud-based safety operators means their fatigue risk increases significantly with changes in the human-vehicle ratio (e.g., from 1:1 to 1:3), and the cause of fatigue gradually shifts from being primarily driven by physiological energy consumption to being primarily driven by cognitive overload. This objective can more effectively reduce the risk of takeover delays due to fatigue, improve the quality of remote monitoring, and promote the sustainable development of remote driving safety monitoring for intelligent vehicles.

[0004] Traditional methods have certain limitations in practical application. Previous methods were typically designed for single-vehicle driving scenarios, rarely considering the differences in fatigue causation weights caused by dynamic changes in the driver-vehicle ratio under multi-vehicle monitoring. For example, in a 1:1 scenario, fatigue characteristics are mainly driven by basic physiological indicators, while in a 1:3 scenario, they are more reflected in increased neurocognitive load. Traditional models, using uniform indicators and fixed algorithms, struggle to capture this shift, affecting recognition accuracy. Furthermore, most existing methods are based on fixed driver-vehicle ratios, lacking a balance between feature capture and computational efficiency under different driver-vehicle ratio scenarios, limiting their application in multi-vehicle monitoring needs.

[0005] Meanwhile, existing research has certain shortcomings in terms of the continuity and pervasive effect of fatigue evolution under different driver-vehicle ratio scenarios. When the driver-vehicle ratio changes from 1:3 to 1:1, safety officers may still be fatigued due to previous high-load work. However, traditional models only focus on the current scenario data, easily overlooking the cumulative effect of fatigue, resulting in inconsistent identification results. In addition, existing early warning strategies rarely combine the coupling relationship between driver-vehicle ratio and fatigue risk, and have not established a tiered intervention mechanism, making it difficult to provide accurate safety responses based on different regulatory loads. For example, in a 1:3 high-load scenario, the impact of mild fatigue on driving safety is significantly higher than in a 1:1 low-load scenario, but existing strategies often use uniform intervention measures, affecting the effective control of accident risks.

[0006] As an emerging profession, the work characteristics of cloud-based safety operators differ fundamentally from those of on-board safety officers. Cloud-based safety operators rely on multi-screen visual monitoring for operation, resulting in a single mode of perception and an exponentially increasing workload with the number of vehicles under supervision. In contrast, on-board safety officers directly control vehicles through multi-sensory fusion, with a relatively constant workload. This difference makes the fatigue characteristics of cloud-based safety operators more complex. Traditional physiological indicators such as blinking frequency become less sensitive under high-load scenarios, necessitating the introduction of neurocognitive indicators and multi-vehicle collaborative data to improve recognition accuracy.

[0007] In summary, existing technologies have room for improvement in terms of adaptability to the human-vehicle ratio, model structure adaptability, cross-scenario continuity, hierarchical early warning strategies, and adaptability to the characteristics of remote safety officers. There is an urgent need for a fatigue identification and early warning method and system that can adapt to the dynamic changes in the human-vehicle ratio in multi-vehicle monitoring scenarios. Summary of the Invention

[0008] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method and system for fatigue identification and early warning of cloud-based safety personnel in multi-vehicle monitoring scenarios, thereby improving the accuracy and robustness of fatigue identification and reducing the risk of remote driving monitoring failure due to fatigue.

[0009] To achieve the above objectives, the specific solution of the present invention is as follows:

[0010] The first aspect of this invention provides a method for fatigue identification and early warning of safety personnel in a cloud-based vehicle monitoring scenario, comprising the following steps:

[0011] Based on different human-vehicle ratio scenarios, obtain the corresponding multimodal data for each human-vehicle ratio scenario;

[0012] A differentiated sub-model system for different human-vehicle ratio scenarios is constructed using an LSTM model-attention mechanism, generating personalized fatigue identification sub-models that adapt to fatigue patterns under each human-vehicle ratio scenario; each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs the sub-fatigue identification results.

[0013] By fusing parameter information from various personalized fatigue identification sub-models through multi-task federated learning, a global fatigue identification model is generated. The fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model are then integrated through the global fatigue identification model to obtain a comprehensive fatigue identification result.

[0014] Based on the comprehensive fatigue identification results, combined with the human-vehicle ratio parameter and fatigue risk matrix, fatigue levels are classified, and safety warning measures are generated according to the graded early warning strategy.

[0015] Furthermore, the multimodal data of the present invention includes physiological state data, behavioral operation data, and vehicle operation data; wherein, the physiological state data includes blink frequency, EEG alpha wave frequency, and heart rate variability; the behavioral operation data includes mouse or gamepad operation frequency, visual fixation duration distribution, and operation reaction time; and the vehicle operation data includes vehicle speed, acceleration, lane departure frequency, and multi-vehicle trajectory safety.

[0016] Furthermore, the step of acquiring multimodal data corresponding to different person-vehicle ratio scenarios specifically includes:

[0017] When the ratio of people to vehicles is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, and heart rate variability, supplemented by mouse operation frequency, single-screen fixation duration, and lane departure frequency.

[0018] When the ratio of human to vehicle is 1:2, add multitasking-related indicators, including operation reaction time, gaze shift frequency, and following distance.

[0019] When the ratio of human to vehicle is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, multi-vehicle trajectory safety, and takeover delay.

[0020] Furthermore, the generation of a personalized fatigue identification sub-model adapted to fatigue patterns in various vehicle-to-person ratio scenarios specifically includes:

[0021] Normalize the multimodal data to eliminate the influence of different dimensions;

[0022] Principal component analysis was performed on the normalized individual optimal fatigue characteristic indicators to extract key features.

[0023] The principal components of the samples are used as input to the LSTM model, and the state values ​​of the samples are used as output. The network coefficients are adjusted through the information transmission and error backpropagation algorithm of the LSTM model neurons to establish a personalized fatigue identification sub-model for the human-vehicle ratio scenario.

[0024] Furthermore, the present invention further includes, in part, constructing a differentiated sub-model system for different human-vehicle ratio scenarios using an LSTM model-attention mechanism, specifically comprising:

[0025] When the ratio of people to vehicles is 1:1, a single-layer LSTM model with a fixed attention mechanism is used, in which the basic physiological indicators have the highest weight.

[0026] When the ratio of people to vehicles is 1:2, a two-layer LSTM model with dynamic attention mechanism is adopted. The dynamic attention mechanism introduces the ratio of people to vehicles to dynamically adjust the feature weights.

[0027] When the ratio of people to vehicles is 1:3, a bidirectional LSTM model with an adaptive attention mechanism is adopted, the weight of neurocognitive indicators is increased, and a dynamic weight adjustment mechanism triggered by the covariance of multi-vehicle trajectories is set.

[0028] Furthermore, when switching between human-vehicle ratio scenarios, the hidden state data of the LSTM model in the previous human-vehicle ratio scenario and the fatigue feature data within time T1 are retained. The hidden state data of the previous human-vehicle ratio scenario is used as the initial input of the sub-model of the current human-vehicle ratio scenario. The fatigue feature data of the sub-model of the previous human-vehicle ratio scenario is assigned a first weight, and the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is assigned a second weight. The weights are updated every T2 time interval until the second weight of the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is equal to 1. At this time, the sub-model of the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

[0029] Furthermore, the present invention further includes the step of fusing parameter information from various personalized fatigue identification sub-models through multi-task federated learning to generate a global fatigue identification model; and the step of integrating fatigue features and sub-fatigue identification results from various personalized fatigue identification sub-models through the global fatigue identification model to obtain a comprehensive fatigue identification result, specifically including:

[0030] Regularly collect parameter information of sub-models in each person-vehicle ratio scenario, including gradient update information, model weights, fatigue features, and sub-fatigue identification results;

[0031] Based on fatigue feature information under different human-vehicle ratio scenarios, the data volume under each human-vehicle ratio scenario drives the sub-model parameters of each sub-model to be weighted, aggregated and adaptively adjusted, forming a fusion strategy for fatigue features of each human-vehicle ratio scenario, and generating a new round of global fatigue identification model.

[0032] The new round of global fatigue identification model outputs a fused global fatigue identification result based on the fatigue characteristics and sub-fatigue identification results of each sub-model. The fused global fatigue identification result is compared with the actual state value, and the new round of global fatigue identification model is optimized using an error feedback adjustment mechanism. The optimized global model parameters are sent to each sub-model, and each sub-model iteratively updates according to the optimized global model parameters. The above process is repeated until the fused global fatigue identification result meets the convergence condition, resulting in a global fatigue identification model that conforms to reality. The global fatigue identification model outputs a comprehensive fatigue identification result.

[0033] Furthermore, the present invention further includes, based on the comprehensive fatigue identification results, and in conjunction with the person-vehicle ratio parameter and the fatigue risk matrix, classifying fatigue levels, specifically including:

[0034] Based on the fatigue identification results and combined with the person-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine the fatigue level; the fatigue levels include Level I fatigue, Level II fatigue, Level III fatigue, Level IV fatigue, and Level V fatigue, which have progressively increasing risks.

[0035] When the ratio of people to vehicles is 1:1, Level I fatigue corresponds to low risk, Level II fatigue corresponds to medium risk, and Level III fatigue corresponds to high risk.

[0036] When the ratio of people to vehicles is 1:2, Level II fatigue corresponds to medium risk, Level III fatigue corresponds to high risk, and Level IV fatigue corresponds to extremely high risk.

[0037] When the ratio of people to vehicles is 1:3, Level III fatigue corresponds to high risk, Level IV fatigue corresponds to extremely high risk, and Level V fatigue corresponds to emergency risk.

[0038] Furthermore, the generation of safety warning measures based on the graded early warning strategy in this invention specifically includes:

[0039] Based on the fatigue level classification results, determine whether to activate the corresponding safety warning;

[0040] When the fatigue level is Level I, a yellow alert is activated, and a voice prompt says, "The current fatigue level is low, please remain vigilant."

[0041] When the fatigue level is Level II, an orange alert is activated, a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of vehicles under supervision", and the level is automatically downgraded. If the ratio of personnel to vehicles is ≥2, the ratio will be automatically reduced by one level.

[0042] When the fatigue level reaches Level III, a red alert is activated, a mandatory rest is initiated, the operating interface is locked, and a backup safety officer is simultaneously notified to take over one vehicle and activate autonomous driving assistance for the remaining vehicles.

[0043] When the fatigue level is IV or V, an emergency response is initiated, the entire vehicle is taken over, automatic driving is activated, the safety driver leaves the monitoring queue, and a backup safety driver takes over.

[0044] The second aspect of this invention provides a cloud-based safety officer fatigue identification and early warning system for multi-vehicle monitoring scenarios, including a central processing center, a fatigue risk identification unit, and a safety early warning unit;

[0045] The central processing center is used for data interaction and instruction distribution, and supports two-way communication with the fatigue risk identification unit and the safety early warning unit.

[0046] The fatigue risk identification unit is used to acquire multimodal data in real time under different human-vehicle ratio scenarios, and to identify fatigue by constructing a differentiated sub-model system.

[0047] The safety early warning unit is used to determine the fatigue level based on the fatigue identification results and to generate safety warning measures by adopting a graded early warning strategy.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0049] This invention achieves accurate capture of fatigue characteristics under different driver-vehicle ratio scenarios by applying a differentiated data acquisition strategy and an LSTM model-attention mechanism, thereby improving the accuracy of fatigue identification. The introduction of a federated learning framework effectively integrates information from sub-models of different driver-vehicle ratio scenarios, addressing the shortcomings of traditional models in cross-scenario continuity and penetration effects, and improving the robustness of fatigue identification. Furthermore, the combination of a fatigue risk matrix and a tiered early warning strategy significantly improves driving safety in multi-vehicle monitoring scenarios and reduces the risk of remote driving monitoring failure due to fatigue. Attached Figure Description

[0050] Figure 1 This is a structural block diagram of the cloud cabin safety officer fatigue identification and early warning system in a multi-vehicle monitoring scenario according to an embodiment of the present invention.

[0051] Figure 2 This is a structural block diagram of the fatigue risk identification unit in an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but this is not to limit the scope of the invention to this.

[0053] like Figures 1 to 2 As shown in this embodiment, a method for fatigue identification and early warning of cloud-based safety personnel in a multi-vehicle monitoring scenario is adapted to scenarios with dynamic changes in the ratio of personnel to vehicles. Specifically, it includes the following steps:

[0054] Step S100: Based on different human-vehicle ratio scenarios (human-vehicle ratio ∈ {1:1, 1:2, 1:3}), obtain the corresponding multimodal data for each human-vehicle ratio scenario.

[0055] Specifically, multimodal data directly related to the human-vehicle ratio (1:1 / 1:2 / 1:3) is collected in real time by multiple devices, providing objective and dynamic input support for the fatigue identification model. This multimodal data includes physiological state data, behavioral operation data, and vehicle operation data. Physiological state data includes blink frequency, EEG alpha wave frequency, and heart rate variability; behavioral operation data includes mouse or gamepad operation frequency, visual fixation duration distribution, and operation reaction time; and vehicle operation data includes vehicle speed, acceleration, lane departure frequency, and multi-vehicle trajectory safety.

[0056] To address the varying causes and characteristic weights of fatigue among cloud-based safety personnel in different driver-to-vehicle ratio scenarios, this embodiment employs a differentiated data collection strategy:

[0057] When the ratio of people to vehicles is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, and heart rate variability, supplemented by mouse operation frequency, single-screen fixation duration, and lane departure frequency.

[0058] When the ratio of human to vehicle is 1:2, add multitasking-related indicators, including operation reaction time, gaze shift frequency, and following distance.

[0059] When the ratio of human to vehicle is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, multi-vehicle trajectory safety, and takeover delay.

[0060] For example, a professional eye tracker is equipped at the safety officer's workstation in the cloud cabin to monitor blink frequency in real time; the EEG acquisition device uses the Emotiv Epoc X wireless EEG device manufactured by Emotiv Systems to obtain stable alpha wave signals; mouse operation behavior is recorded through a system background program; and lane departure is monitored using a virtual simulated vehicle operating environment. The eye tracker collects blink data at a sampling frequency of 120Hz, and the EEG device collects EEG signals at a sampling frequency of 1000Hz. After bandpass filtering, the alpha wave frequency is extracted; the mouse operation frequency is statistically analyzed every 10 minutes; the number of lane departures is determined in real time based on the positional relationship between the virtual vehicle and the lane lines using a preset algorithm. The final data obtained are a blink frequency of 12 times / minute, an alpha wave frequency of 11Hz, a mouse operation frequency of 8 times / 10 minutes, and a lane departure frequency of 2 times / 10 minutes.

[0061] Step S200: Construct a differentiated sub-model system for different human-vehicle ratio scenarios using the LSTM model-attention mechanism, and generate personalized fatigue identification sub-models that adapt to the fatigue patterns under each human-vehicle ratio scenario; each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs the sub-fatigue identification results.

[0062] Specifically, when the ratio of people to vehicles is 1:1, a single-layer LSTM model with a fixed attention mechanism is used, with the physiological basic indicators having the highest weight, forming sub-model A; when the ratio of people to vehicles is 1:2, a two-layer LSTM model with a dynamic attention mechanism is used, with the dynamic attention mechanism introducing the human-vehicle ratio coefficient to dynamically adjust the feature weights, forming sub-model B; when the ratio of people to vehicles is 1:3, a bidirectional LSTM model with an adaptive attention mechanism is used, with the weights of neurocognitive indicators increased, and a dynamic weight adjustment mechanism triggered by the covariance of multi-vehicle trajectories is set, forming sub-model C.

[0063] Specifically, an LSTM model with an attention mechanism is trained using multimodal data from safety officers in different driver-to-vehicle ratio scenarios within the cloud-based safety cabin, establishing a personalized fatigue identification sub-model for these scenarios. The LSTM model, through a gating mechanism, captures the long-term evolutionary characteristics of individual fatigue, addressing the problem of mining nonlinear fatigue features over long periods. The attention layer learns the weights of fatigue features through data-driven learning, dynamically capturing the features most conducive to fatigue identification as the driver-to-vehicle ratio changes, addressing the impact of the driver-to-vehicle ratio on fatigue features, and enhancing the model's accuracy and speed in identifying fatigue features under various driver-to-vehicle ratio scenarios. The LSTM model utilizes input gates, forget gates, and output gates to control information selection and extract fatigue features from time-series information. Layer normalization stabilizes network training and enhances its adaptability to fatigue characteristics under different driver-to-vehicle ratios. Based on the effectiveness of fatigue feature indicators under different driver-to-vehicle ratios, the attention layer increases the weights of key features, improving the adaptability of the personalized fatigue identification sub-model to driver-to-vehicle ratio switching, thereby improving the accuracy of fatigue identification.

[0064] This embodiment addresses the significant differences in fatigue evolution between individuals and vehicles by utilizing layer normalization of the LSTM model, as shown in the following formula:

[0065] ;

[0066] in, It is the output of the LSTM at time t. and These are the mean and standard deviation of the LSTM output under the ratio of person to vehicle r, respectively. It is a small constant that avoids division by zero. and These are trainable scaling and translation parameters that will not be erased by model aggregation. This normalizes the hidden states of the LSTM model, stabilizes model training, and preserves the fatigue evolution characteristics of the human-vehicle ratio by avoiding global aggregation.

[0067] By using an LSTM model with an attention mechanism, the parameters of the sub-models can be adjusted for different human-vehicle ratio scenarios, thus establishing a differentiated model system to meet the requirements of feature capture accuracy and computational efficiency under different loads.

[0068] Furthermore, the specific process for generating personalized fatigue identification sub-models includes: first, normalizing the multimodal data to eliminate the influence of different dimensions; then, performing principal component analysis (PCA) on the normalized individual optimal fatigue characteristic indicators to extract key features; finally, using the sample principal components as input to the LSTM model, and the sample state values ​​(e.g., mild fatigue = 0, moderate fatigue = 1, severe fatigue = 2) as output, adjusting the network coefficients through the information transmission and error backpropagation algorithm of the LSTM model neurons, thereby establishing a personalized fatigue identification sub-model for the human-vehicle ratio scenario. Each sub-model is trained using multimodal data collected in its respective human-vehicle ratio scenario.

[0069] Each sub-model learns key fatigue characteristics of individuals under the influence of the driver-vehicle ratio through an attention mechanism, which is significant for improving the model's fatigue identification efficiency and personalization level. Therefore, this embodiment retains the attention layer to avoid average aggregation smoothing out the individualized fatigue characteristics of safety officers under different driver-vehicle ratios. The weights of the attention layer are calculated as follows:

[0070] ;

[0071] in, It is the layer-normalized output vector. The attention weights are derived from training time t, and r is the human-to-vehicle ratio. It is the global adjustment coefficient for the attention weight of the human-vehicle comparison.

[0072] Step S300: The parameter information of each personalized fatigue identification sub-model is fused through multi-task federated learning to generate a global fatigue identification model; the fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model are integrated through the global fatigue identification model to obtain a comprehensive fatigue identification result.

[0073] Specifically, using a multi-task federated learning method, the parameter information of personalized fatigue identification sub-models under different human-vehicle ratio scenarios is periodically weighted and aggregated based on the data volume and local loss function value of the sub-models, to obtain a global fatigue identification model that includes both commonalities and individualities across scenarios.

[0074] Specifically, by introducing layer normalization and attention mechanisms into the sub-models for each person-vehicle ratio, the fatigue characteristics of each person-vehicle ratio are preserved while aggregating the sub-models. Based on the establishment of personalized fatigue identification sub-models for each person-vehicle ratio scenario, the aggregator of multi-task federated learning collects fatigue features and model information for each person-vehicle ratio. The fatigue features and identification results for each person-vehicle ratio are periodically uploaded to the aggregator of multi-task federated learning for global model updates. Then, the aggregator of multi-task federated learning uses a weighted averaging method to drive the weighted aggregation and adaptive adjustment of the sub-model parameters of the sub-models based on the fatigue feature information under different person-vehicle ratio scenarios, with the amount of data for each person-vehicle ratio, forming a fusion strategy for fatigue identification information of each person-vehicle ratio, and generating a new round of global fatigue identification models.

[0075] This fusion process utilizes statistical patterns of fatigue characteristics under different human-vehicle ratio scenarios and also uses them for weighted aggregation of sub-model parameters to form a new global fatigue identification model that can adapt to various human-vehicle ratio switching scenarios. The fusion process is based on multi-source weighted aggregation of sub-model parameters and fatigue feature information from sub-models under different human-vehicle ratio scenarios to form a global fatigue identification model that includes both commonalities and individual characteristics across scenarios.

[0076] The global fatigue identification model is driven by fused multi-person vehicle ratio data, enabling it to capture common fatigue patterns across different multi-person vehicle ratio scenarios. Through periodic aggregation and optimization, the adaptability of each sub-model to common and individual fatigue characteristics is continuously improved. Each sub-model continuously learns common features from the global fatigue identification model, thereby enhancing the robustness of the identification system.

[0077] Specifically, the system periodically collects parameter information from sub-models under various human-vehicle ratio scenarios, including gradient update information, model weights, fatigue features, and sub-fatigue identification results. Based on the fatigue feature information under different human-vehicle ratio scenarios, the system drives the weighted aggregation and adaptive adjustment of sub-model parameters of each sub-model using the data volume under each human-vehicle ratio scenario, forming a fusion strategy for fatigue features under different human-vehicle ratio scenarios, and generating a new round of global fatigue identification models. The sub-model parameters of each sub-model include gradient update information and model weights; the fatigue feature information under different human-vehicle ratio scenarios includes fatigue features and sub-fatigue identification results.

[0078] The new round of global fatigue identification model outputs a fused global fatigue identification result based on the fatigue features and sub-fatigue identification results of each sub-model. This fused global fatigue identification result is compared with the actual state value, and an error feedback adjustment mechanism is used to optimize the new round of global fatigue identification model. The multi-task federated learning distributor sends the optimized global model parameters to each sub-model, and each sub-model iteratively updates based on the optimized global model parameters. This process is repeated until the fused global fatigue identification result meets the convergence condition, resulting in a global fatigue identification model that conforms to reality. The global fatigue identification model comprehensively evaluates the fatigue features extracted by each sub-model and the sub-fatigue identification results, thereby outputting the final comprehensive fatigue identification result. The optimized global model parameters include the latest gradient value and gradient direction.

[0079] Specifically, the fused global fatigue identification results are compared with the actual state values ​​to obtain a deviation value. If the deviation value is greater than or equal to a preset threshold, the global model parameters of the global fatigue identification model are optimized based on the deviation value. The optimized global model parameters are then distributed to each sub-model, which updates itself based on the optimized global model parameters and is trained using corresponding multimodal data to output sub-fatigue identification results. The aggregator then periodically collects the parameter information of each updated sub-model to update the global model parameters. This process continues until the deviation value is less than the preset threshold, resulting in a global fatigue identification model that meets the actual fatigue identification capabilities of the cloud cabin safety officer, thereby improving the robustness of fatigue identification.

[0080] In actual operation, when the system identifies the current human-vehicle ratio scenario (e.g., switching to a human-vehicle ratio of 1:3), it will call the sub-model under that ratio for fatigue identification. Simultaneously, this sub-model has been injected with the sub-model parameters of the global fatigue identification model, possessing a memory of common fatigue characteristics across multiple human-vehicle ratios. During actual fatigue identification, the system first calls the corresponding sub-model based on the current human-vehicle ratio scenario to perform fatigue identification and outputs the sub-fatigue identification results.

[0081] In this embodiment, when switching between human-vehicle ratio scenarios, the hidden state data of the LSTM model in the previous human-vehicle ratio scenario and the fatigue feature data within time T1 are retained. The hidden state data of the previous human-vehicle ratio scenario is used as the initial input of the sub-model of the current human-vehicle ratio scenario. The fatigue feature data of the sub-model of the previous human-vehicle ratio scenario is assigned a first weight, and the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is assigned a second weight. The weights are updated every T2 time interval until the second weight of the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is equal to 1. At this time, the sub-model of the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

[0082] Specifically, time T1 can be determined based on the actual time window of fatigue effect penetration before and after the personnel-vehicle ratio switch. For example, setting T1 to 2 hours means retaining the hidden state data of the LSTM model under the previous personnel-vehicle ratio scenario and the fatigue feature data within 2 hours. Initially, the first weight can be set to 0.7, the second weight can be set to 0.3, and T2 can be set to 30 minutes. As time progresses, the second weight increases by 0.1 every 30 minutes until the second weight equals 1. This ensures that the previously accumulated fatigue level naturally transitions to the current personnel-vehicle ratio scenario, avoiding fatigue "resetting" and accurately identifying the fatigue state transition of the cloud cabin safety officer before and after the switch.

[0083] From a fatigue identification perspective, the importance weight of the same fatigue feature often differs depending on the person-to-vehicle ratio. For example, working hours are far more sensitive to fatigue in a 1:3 person-to-vehicle ratio scenario than in a 1:1 scenario. By using a global fatigue identification model to adaptively adjust feature weights, features such as working hours, eye movements, and operational characteristics are automatically assigned weights based on their saliency in new scenarios. This maintains the continuity of the cumulative fatigue effect while also strengthening fatigue-sensitive features in the new scenario, achieving accurate fatigue identification and risk warning.

[0084] For example, when the person-vehicle ratio switches from 1:3 or 1:2 to 1:1 (or vice versa), the system retains the output of the sub-model and its internal state (such as the hidden state of the LSTM) before the switch, using it as the initial input for fatigue identification in the new scenario. This ensures that the previously accumulated fatigue level naturally transitions to the current person-vehicle ratio scenario, avoiding a "reset" of fatigue. Then, based on the global fatigue identification model and the fatigue identification results before the switch, the system adaptively adjusts the feature weights and sub-model parameters under the current person-vehicle ratio. This reweights factors such as working hours, operational range, and eye-tracking features according to their importance in the new scenario, reflecting the differences in fatigue feature sensitivity under different person-vehicle ratio scenarios. Finally, through this state transition and weight reconstruction mechanism, the system retains the fatigue penetration effect across scenarios while fully utilizing the global fatigue patterns to optimize feature contribution, achieving accurate fatigue identification and continuous risk warning under different person-vehicle ratio scenario switching.

[0085] Step S500: Based on the comprehensive fatigue identification results, combined with the human-vehicle ratio parameter and fatigue risk matrix, fatigue levels are classified, and safety warning measures are generated according to the graded early warning strategy.

[0086] Specifically, based on the comprehensive fatigue identification results and combined with the person-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine the fatigue level. The fatigue levels include Level I fatigue, Level II fatigue, Level III fatigue, Level IV fatigue, and Level V fatigue, with gradually increasing risk. Among them, when the person-vehicle ratio is 1:1, Level I fatigue corresponds to low risk, Level II fatigue corresponds to medium risk, and Level III fatigue corresponds to high risk; when the person-vehicle ratio is 1:2, Level II fatigue corresponds to medium risk, Level III fatigue corresponds to high risk, and Level IV fatigue corresponds to extremely high risk; when the person-vehicle ratio is 1:3, Level III fatigue corresponds to high risk, Level IV fatigue corresponds to extremely high risk, and Level V fatigue corresponds to critical risk.

[0087] The tiered early warning strategy generates safety warning measures, specifically including: determining whether to activate the corresponding safety warning based on the fatigue level classification results;

[0088] When the fatigue level is Level I, a yellow alert is activated, and a voice prompt says, "Current fatigue level is low, please remain vigilant." When the fatigue level is Level II, an orange alert is activated, and a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of monitored vehicles," and the system automatically downgrades. If the ratio of personnel to vehicles is ≥2, the ratio will automatically decrease by one level (e.g., from a ratio of 1:3 to 1:2). When the fatigue level is Level III, a red alert is activated, a forced rest is initiated, the operating interface is locked, and a backup safety officer is simultaneously notified to take over one vehicle, while the remaining vehicles are activated with autonomous driving assistance. When the fatigue level is Level IV or V, an emergency response is activated, all vehicles are taken over, autonomous driving is triggered, the safety officer leaves the monitoring queue, and the backup safety officer takes over.

[0089] like Figures 1 to 2 As shown, this embodiment also provides a cloud cabin safety officer fatigue identification and early warning system for multi-vehicle monitoring scenarios, which adapts to scenarios with dynamic changes in the ratio of personnel to vehicles, specifically including a central processing center, a fatigue risk identification unit, and a safety early warning unit;

[0090] The central processing center is used for data interaction and instruction distribution, and supports two-way communication with the fatigue risk identification unit and the safety warning unit. The fatigue risk identification unit is used to acquire multimodal data in real time under different human-vehicle ratio scenarios, and to identify fatigue by constructing a differentiated sub-model system. The safety warning unit is used to determine the fatigue level based on the fatigue identification results, and to generate safety warning measures by adopting a graded warning strategy.

[0091] Specifically, the fatigue risk identification unit includes a fatigue data acquisition module, a human-vehicle ratio sub-model construction module, and a multi-task federated learning module. The fatigue data acquisition module acquires multimodal data corresponding to different human-vehicle ratio scenarios. The human-vehicle ratio sub-model construction module builds a differentiated sub-model system using an LSTM model-attention mechanism to generate personalized fatigue identification sub-models. These personalized sub-models then extract fatigue features from the current multimodal data to obtain sub-fatigue identification results. The multi-task federated learning module integrates the parameter information of each personalized fatigue identification sub-model through multi-task federated learning to generate a global fatigue identification model. Finally, the global fatigue identification model integrates the fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model to obtain a comprehensive fatigue identification result.

[0092] The safety warning unit includes a fatigue level classification module and a graded warning execution module. The fatigue level classification module is used to determine the fatigue level based on the fatigue identification results and, in conjunction with the human-vehicle ratio parameter, classify the fatigue level. The graded warning execution module determines whether to activate the corresponding safety warning based on the fatigue level classification results.

[0093] For example, taking a 1:1 human-to-vehicle ratio scenario, an eye tracker collects blink data at a sampling frequency of 120Hz, and an EEG device collects EEG signals at a sampling frequency of 1000Hz. After bandpass filtering, the alpha wave frequency is extracted. Mouse operation frequency is statistically analyzed every 10 minutes. Lane departures are determined in real time based on the positional relationship between the virtual vehicle and the lane lines using a preset algorithm. The final data obtained are: blink frequency of 12 times / minute, EEG alpha wave frequency of 11Hz, mouse operation frequency of 8 times / 10 minutes, and lane departures of 2 times / 10 minutes.

[0094] The collected physiological and behavioral data were processed using standardization methods to ensure that the data conformed to a standard normal distribution with a mean of 0 and a standard deviation of 1, so as to facilitate model input.

[0095] The single-layer LSTM model contains 128 neurons. Standardized data is input into the model sequentially in time series, with a 5-minute time step. A fixed attention mechanism calculates the correlation between various features and fatigue state, assigning a weight of 0.4 to EEG alpha waves, 0.3 to blink frequency, and the remaining 0.3 to other features. The model processes the input data using trained parameters, ultimately outputting a result indicating moderate fatigue. The fatigue risk identification unit transmits the moderate fatigue result to the central processing center, which then sends it to the safety warning unit. Based on the fatigue risk matrix, the safety warning unit determines the current level as Level II fatigue (orange warning). The graded warning execution module, based on the orange warning, sends instructions to the cloud cabin safety officer's interface. A prominent red warning box pops up on the interface, and a voice prompt is issued through the built-in speaker: "Fatigue level upgraded, recommended to reduce the number of monitored vehicles." Furthermore, the central processing center stores this warning information and related data for subsequent analysis and review.

[0096] For example, taking a scenario with a human-to-vehicle ratio of 1:3 as an example, a high-precision pupil detector is used to monitor the pupil diameter in real time; multi-vehicle trajectory safety data is obtained through the vehicle positioning system and cloud real-time communication technology; the takeover delay time is determined by recording the time interval between the takeover command issued by the system and the actual operation by the safety officer; finally, data with a pupil diameter of 1.8mm, a multi-vehicle trajectory covariance of 0.85, and a takeover delay of 3.5 seconds are collected.

[0097] Then, the collected neurocognitive indicators and multi-vehicle collaborative data are standardized and input into a bidirectional LSTM model. The bidirectional LSTM model can extract features from time-series data from both forward and backward directions, capturing more comprehensive information from the data. The adaptive attention mechanism automatically increases the weight of vehicle data to 0.5 when the multi-vehicle trajectory covariance is detected to be greater than 0.6, based on the characteristics of the input data. After calculation, the model finally outputs a Level V fatigue result, with the warning level being an emergency response. Based on the warning level, the graded warning execution module immediately triggers a full vehicle takeover command. First, the autonomous driving system quickly starts, sending control commands to each monitored vehicle via cloud communication technology to adjust vehicle speed, direction, and other driving parameters to ensure safe vehicle operation.

[0098] The tiered early warning execution module notifies backup safety personnel to take over via SMS, voice calls, and other means. The system records the timeline of the entire intervention process; in addition, the central processing center will back up detailed data of this emergency intervention, including all collected data, model calculation processes, and early warning trigger records, for subsequent accident analysis and system optimization.

[0099] The above description is only a preferred embodiment of the present invention. Therefore, any equivalent changes or modifications made to the structure, features and principles described in the claims of this patent application are included within the protection scope of this patent application.

Claims

1. A method for fatigue identification and early warning of safety personnel in a cloud-based vehicle monitoring scenario, characterized in that, Includes the following steps: Based on different human-vehicle ratio scenarios, obtain the corresponding multimodal data for each human-vehicle ratio scenario; By constructing a differentiated sub-model system for different human-vehicle ratio scenarios through the LSTM model-attention mechanism, a personalized fatigue identification sub-model adapted to the fatigue patterns under each human-vehicle ratio scenario is generated. Each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs the sub-fatigue identification results. By fusing parameter information from various personalized fatigue identification sub-models through multi-task federated learning, a global fatigue identification model is generated. By integrating the fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model into the global fatigue identification model, a comprehensive fatigue identification result is obtained. Based on the comprehensive fatigue identification results, combined with the human-vehicle ratio parameter and fatigue risk matrix, fatigue levels are classified, and safety warning measures are generated according to the graded early warning strategy. The step of acquiring multimodal data corresponding to different person-vehicle ratio scenarios specifically includes: When the ratio of people to vehicles is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, and heart rate variability, supplemented by mouse operation frequency, single-screen fixation duration, and lane departure frequency. When the ratio of human to vehicle is 1:2, add multitasking-related indicators, including operation reaction time, gaze shift frequency, and following distance. When the ratio of human to vehicle is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, multi-vehicle trajectory safety and takeover delay. The system of differentiated sub-models for different human-vehicle ratio scenarios constructed using the LSTM model-attention mechanism specifically includes: When the ratio of people to vehicles is 1:1, a single-layer LSTM model with a fixed attention mechanism is used, in which the basic physiological indicators have the highest weight. When the ratio of people to vehicles is 1:2, a two-layer LSTM model with dynamic attention mechanism is adopted. The dynamic attention mechanism introduces the ratio of people to vehicles to dynamically adjust the feature weights. When the ratio of people to vehicles is 1:3, a bidirectional LSTM model with an adaptive attention mechanism is adopted, the weight of neurocognitive indicators is increased, and a dynamic weight adjustment mechanism triggered by the covariance of multi-vehicle trajectories is set.

2. The cloud cabin safety officer fatigue identification and early warning method according to claim 1, characterized in that, The generation of a personalized fatigue identification sub-model adapted to fatigue patterns under different vehicle-to-person ratio scenarios specifically includes: Normalize the multimodal data to eliminate the influence of different dimensions; Principal component analysis was performed on the normalized individual optimal fatigue characteristic indicators to extract key features. The principal components of the samples are used as input to the LSTM model, and the state values ​​of the samples are used as output. The network coefficients are adjusted through the information transmission and error backpropagation algorithm of the LSTM model neurons to establish a personalized fatigue identification sub-model for the human-vehicle ratio scenario.

3. The cloud cabin safety officer fatigue identification and early warning method according to claim 1, characterized in that, When switching between human-vehicle ratio scenarios, the hidden state data of the LSTM model in the previous human-vehicle ratio scenario and the fatigue feature data within time T1 are retained. The hidden state data of the previous human-vehicle ratio scenario is used as the initial input of the sub-model in the current human-vehicle ratio scenario. The fatigue feature data of the sub-model in the previous human-vehicle ratio scenario is assigned a first weight, and the fatigue feature data of the sub-model in the current human-vehicle ratio scenario is assigned a second weight. The weights are updated every T2 time interval until the second weight of the fatigue feature data of the sub-model in the current human-vehicle ratio scenario is equal to 1. At this point, the sub-model in the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

4. The cloud cabin safety officer fatigue identification and early warning method according to claim 1, characterized in that, The parameter information of each personalized fatigue identification sub-model is fused through multi-task federated learning to generate a global fatigue identification model. By integrating the fatigue features and sub-fatigue identification results of various personalized fatigue identification sub-models into a global fatigue identification model, a comprehensive fatigue identification result is obtained, which specifically includes: Regularly collect parameter information of sub-models in each person-vehicle ratio scenario, including gradient update information, model weights, fatigue features, and sub-fatigue identification results; Based on fatigue feature information under different human-vehicle ratio scenarios, the data volume under each human-vehicle ratio scenario drives the sub-model parameters of each sub-model to be weighted, aggregated and adaptively adjusted, forming a fusion strategy for fatigue features of each human-vehicle ratio scenario, and generating a new round of global fatigue identification model. The new round of global fatigue identification model outputs a fused global fatigue identification result based on the fatigue characteristics and sub-fatigue identification results of each sub-model. The fused global fatigue identification result is compared with the actual state value, and the new round of global fatigue identification model is optimized using an error feedback adjustment mechanism. The optimized global model parameters are sent to each sub-model, and each sub-model iteratively updates according to the optimized global model parameters. The above process is repeated until the fused global fatigue identification result meets the convergence condition, resulting in a global fatigue identification model that conforms to reality. The global fatigue identification model outputs a comprehensive fatigue identification result.

5. The cloud cabin safety officer fatigue identification and early warning method according to claim 1, characterized in that, The fatigue level classification based on the comprehensive fatigue identification results, combined with the human-vehicle ratio parameter and fatigue risk matrix, specifically includes: Based on the fatigue identification results and combined with the person-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine the fatigue level; the fatigue levels include Level I fatigue, Level II fatigue, Level III fatigue, Level IV fatigue, and Level V fatigue, which have progressively increasing risks. When the ratio of people to vehicles is 1:1, Level I fatigue corresponds to low risk, Level II fatigue corresponds to medium risk, and Level III fatigue corresponds to high risk. When the ratio of people to vehicles is 1:2, Level II fatigue corresponds to medium risk, Level III fatigue corresponds to high risk, and Level IV fatigue corresponds to extremely high risk. When the ratio of people to vehicles is 1:3, Level III fatigue corresponds to high risk, Level IV fatigue corresponds to extremely high risk, and Level V fatigue corresponds to emergency risk.

6. The cloud cabin safety officer fatigue identification and early warning method according to claim 5, characterized in that, The generation of safety warning measures based on the tiered early warning strategy specifically includes: Based on the fatigue level classification results, determine whether to activate the corresponding safety warning; When the fatigue level is Level I, a yellow alert is activated, and a voice prompt says, "The current fatigue level is low, please remain vigilant." When the fatigue level is Level II, an orange alert is activated, a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of vehicles under supervision", and the level is automatically downgraded. If the ratio of personnel to vehicles is ≥2, the ratio will be automatically reduced by one level. When the fatigue level reaches Level III, a red alert is activated, a mandatory rest is initiated, the operating interface is locked, and a backup safety officer is simultaneously notified to take over one vehicle and activate autonomous driving assistance for the remaining vehicles. When the fatigue level is IV or V, an emergency response is initiated, the entire vehicle is taken over, automatic driving is activated, the safety driver leaves the monitoring queue, and a backup safety driver takes over.

7. A cloud-based safety officer fatigue identification and early warning system for multi-vehicle monitoring scenarios, characterized in that, The method for implementing the cloud cabin safety officer fatigue identification and early warning method as described in any one of claims 1 to 6 includes a central processing center, a fatigue risk identification unit, and a safety early warning unit. The central processing center is used for data interaction and instruction distribution, and supports two-way communication with the fatigue risk identification unit and the safety early warning unit. The fatigue risk identification unit is used to acquire multimodal data in real time under different human-vehicle ratio scenarios, and to identify fatigue by constructing a differentiated sub-model system. The safety early warning unit is used to determine the fatigue level based on the fatigue identification results and to generate safety warning measures by adopting a graded early warning strategy.

Images