Cloud cabin safety officer fatigue identification early warning method and system under multi-vehicle supervision scene

Abstract

The invention relates to the technical field of intelligent vehicle remote driving safety monitoring, in particular to a cloud cabin safety officer fatigue identification early warning method and system under a multi-vehicle supervision scene, and the method comprises the steps: constructing a differentiated sub-model system under different person-to-vehicle ratio scenes through an LSTM model-attention mechanism; generating a personalized fatigue identification sub-model adapted to a fatigue rule in each person-to-vehicle ratio scene; performing fatigue feature extraction on the corresponding multi-modal data by each personalized fatigue identification sub-model, and outputting a sub-fatigue identification result; fusing the parameter information of each personalized fatigue identification sub-model through multi-task federal learning, and generating a global fatigue identification model; and integrating the fatigue characteristics and the sub-fatigue identification results of the individual fatigue identification sub-models through a global fatigue identification model to obtain a comprehensive fatigue identification result. According to the invention, the accuracy and robustness of fatigue identification can be improved, and the risk of remote driving supervision failure caused by fatigue is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote driving safety monitoring of intelligent vehicles, and specifically provides a fatigue identification and early warning method and system for cloud cabin safety officers in a multi-vehicle supervision scenario. Background Art

[0002] The core goal of research on remote monitoring safety for intelligent vehicles is to develop advanced methods and tools to accurately identify and warn of fatigue in cloud cabin safety officers in multi-vehicle monitoring scenarios, enabling targeted measures to ensure the safety of intelligent vehicle remote driving operations. Fatigue identification and warning technology is a crucial component of remote monitoring safety for intelligent vehicles. Its primary task is to quantify and understand the changing patterns of fatigue characteristics of cloud cabin safety officers in dynamic switching scenarios with varying vehicle-to-vehicle ratios, providing scientific support for remote monitoring.

[0003] Fatigue identification and early warning methods that adapt to dynamic changes in the driver-vehicle ratio are complex technical measures aimed at improving driving safety in multi-vehicle monitoring scenarios. The nature of cloud cabin safety officers' work dictates that their fatigue risk increases significantly with changes in the driver-vehicle ratio (e.g., from 1:1 to 1:3), with fatigue drivers gradually shifting from being dominated by physiological energy consumption to being dominated by cognitive overload. This goal can more effectively reduce the risk of fatigue-related takeover delays, improve the quality of remote monitoring, and promote the sustainable development of remote driving safety monitoring for intelligent vehicles.

[0004] Traditional methods have certain limitations in actual operation. Previous methods are usually designed for single-vehicle driving scenarios, and rarely consider the differences in fatigue-causing factors caused by the dynamic changes in the driver-vehicle ratio under multi-vehicle supervision. For example, in a 1:1 scenario, fatigue characteristics are mainly driven by physiological basic indicators, while in a 1:3 scenario, they are more manifested as an increase in neurocognitive load. Traditional models use unified indicators and fixed algorithms, which makes it difficult to capture this transformation, affecting the recognition accuracy. In addition, most existing methods are based on a fixed driver-vehicle ratio, lacking a balance between feature capture and computational efficiency in different driver-vehicle ratio scenarios, which limits their application in multi-vehicle supervision needs.

[0005] At the same time, existing research has certain deficiencies in the continuity and penetration effect of fatigue evolution in scenarios across human-to-vehicle ratios. When the human-to-vehicle ratio switches from 1:3 to 1:1, the safety officer may still be in a fatigued state due to the previous high-load work, but the traditional model only focuses on the current scenario data, which easily ignores the cumulative effect of fatigue, resulting in inconsistent identification results. In addition, existing early warning strategies rarely combine the coupling relationship between the human-to-vehicle ratio and fatigue risk, and no graded intervention mechanism has been established, making it difficult to provide accurate safety responses based on different regulatory loads. For example, in a 1:3 high-load scenario, the impact of mild fatigue on driving safety is significantly higher than in a 1:1 low-load scenario, but existing strategies often adopt consistent intervention measures, which affects the effective control of accident risks.

[0006] As an emerging profession, cloud cabin safety officers have fundamentally different work characteristics from onboard safety officers. Cloud cabin safety officers rely on multi-screen visual monitoring for operation, with a single sensory approach and a workload that increases exponentially with the number of vehicles under their supervision. Onboard safety officers, on the other hand, directly control vehicles through multi-sensory fusion, with a relatively constant workload. This difference complicates fatigue characteristics for cloud cabin safety officers, and traditional physiological indicators such as blink rate become less sensitive in high-load scenarios. There is an urgent need to introduce neurocognitive indicators and multi-vehicle collaborative data to improve identification accuracy.

[0007] In summary, the existing technology has room for improvement in terms of adaptability of the human-vehicle ratio, model structure adaptability, cross-scenario continuity, graded warning strategy, and adaptation of remote safety officer characteristics. There is an urgent need for a fatigue identification and warning method and system that can adapt to the dynamic changes in the human-vehicle ratio in multi-vehicle supervision scenarios. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a cloud cabin safety officer fatigue identification and warning method and system in a multi-vehicle supervision scenario, improve the accuracy and robustness of fatigue identification, and reduce the risk of remote driving supervision failure due to fatigue.

[0009] To achieve the above object, the specific solutions of the present invention are as follows:

[0010] A first aspect of the present invention provides a method for identifying and warning fatigue of a cloud cabin safety officer in a multi-vehicle monitoring scenario, comprising the following steps:

[0011] According to different scenarios of human-vehicle ratio, obtain the corresponding multimodal data under each scenario;

[0012] Through the LSTM model-attention mechanism, a differentiated sub-model system is constructed for different driver-vehicle ratio scenarios, generating personalized fatigue identification sub-models that adapt to the fatigue patterns in each driver-vehicle ratio scenario. Each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs a sub-fatigue identification result.

[0013] Through multi-task federated learning, the parameter information of each personalized fatigue identification sub-model is integrated to generate a global fatigue identification model. The fatigue characteristics and sub-fatigue identification results of each personalized fatigue identification sub-model are integrated through the global fatigue identification model to obtain a comprehensive fatigue identification result.

[0014] Based on the comprehensive fatigue identification results, combined with the driver-to-vehicle ratio parameters and the fatigue risk matrix, fatigue levels are divided, and safety warning measures are generated according to the graded warning strategy.

[0015] The present invention further provides that the multimodal data includes physiological state data, behavioral operation data and vehicle operation data; wherein the physiological state data includes blinking frequency, EEG alpha wave frequency and heart rate variability; the behavioral operation data includes mouse or handle operation frequency, visual gaze duration distribution and operation reaction time; the vehicle operation data includes vehicle speed, acceleration, number of lane deviations and multi-vehicle trajectory safety.

[0016] The present invention further provides that, according to different driver-to-vehicle ratio scenarios, obtaining corresponding multimodal data under each driver-to-vehicle ratio scenario specifically includes:

[0017] When the driver-vehicle ratio is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, heart rate variability, supplemented by mouse operation frequency, single-screen gaze time, and lane deviation times;

[0018] When the driver-vehicle ratio is 1:2, multitasking-related indicators are added, including operation reaction time, gaze shift frequency, and following distance;

[0019] When the human-vehicle ratio is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, and multi-vehicle trajectory safety and takeover delay.

[0020] The present invention further provides that generating a personalized fatigue identification sub-model adapted to fatigue patterns in various driver-vehicle ratio scenarios specifically includes:

[0021] Normalize multimodal data to eliminate the influence of different dimensions;

[0022] Perform principal component analysis on the normalized individual optimal fatigue characteristic index to extract key features;

[0023] The principal components of the samples are used as the input of the LSTM model, and the state values ​​of the samples are used as the output. The network coefficients are adjusted through information transmission of LSTM model neurons and error back propagation algorithm to establish a personalized fatigue identification sub-model for this driver-vehicle ratio scenario.

[0024] The present invention further constructs a differentiated sub-model system for different driver-vehicle ratio scenarios through the LSTM model-attention mechanism, specifically including:

[0025] When the vehicle-to-person ratio is 1:1, a single-layer LSTM model with a fixed attention mechanism is used, in which the physiological basis indicator has the highest weight.

[0026] When the vehicle-to-pedestrian ratio is 1:2, a two-layer LSTM model plus a dynamic attention mechanism is used. The dynamic attention mechanism introduces the vehicle-to-pedestrian ratio coefficient to dynamically adjust the feature weights.

[0027] When the vehicle-to-pedestrian ratio is 1:3, a bidirectional LSTM model plus an adaptive attention mechanism is adopted, the weight of neurocognitive indicators is increased, and a dynamic weight adjustment mechanism triggered by the covariance of multiple vehicle trajectories is set.

[0028] The present invention further retains the hidden state data of the LSTM model in the previous human-vehicle ratio scenario and the fatigue feature data within T1 time when the human-vehicle ratio scenario is switched, and uses the hidden state data of the previous human-vehicle ratio scenario as the initial input of the sub-model of the current human-vehicle ratio scenario, assigns a first weight to the fatigue feature data of the sub-model of the previous human-vehicle ratio scenario, assigns a second weight to the fatigue feature data of the sub-model of the current human-vehicle ratio scenario, and updates the weights at intervals of T2 until the second weight of the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is equal to 1, at which point the sub-model of the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

[0029] The present invention further comprises the following steps: generating a global fatigue identification model by fusing parameter information of each personalized fatigue identification sub-model through multi-task federated learning; and integrating fatigue characteristics and sub-fatigue identification results of each personalized fatigue identification sub-model through the global fatigue identification model to obtain a comprehensive fatigue identification result, which specifically includes:

[0030] Regularly collect parameter information of the sub-models under each driver-vehicle ratio scenario, including gradient update information, model weights, fatigue characteristics, and sub-fatigue identification results;

[0031] Based on fatigue characteristic information under different driver-to-vehicle ratio scenarios, the weighted aggregation and adaptive adjustment of sub-model parameters of each sub-model are driven by the data volume under each driver-to-vehicle ratio scenario, forming a fusion strategy for fatigue characteristics of each driver-to-vehicle ratio and generating a new round of global fatigue identification model;

[0032] The new round of global fatigue identification model outputs the fused global fatigue identification result based on the fatigue characteristics and sub-fatigue identification results of each sub-model. The fused global fatigue identification result is compared with the actual state value. The new round of global fatigue identification model is optimized using the error feedback adjustment mechanism, and the optimized global model parameters are sent to each sub-model. Each sub-model is iteratively updated according to the optimized global model parameters. The above process is repeated until the fused global fatigue identification result meets the convergence condition and a global fatigue identification model that conforms to reality is obtained. The global fatigue identification model outputs a comprehensive fatigue identification result.

[0033] The present invention further divides the fatigue level based on the comprehensive fatigue identification result, combined with the driver-vehicle ratio parameter and the fatigue risk matrix, specifically including:

[0034] Based on the fatigue identification results and combined with the driver-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine fatigue levels; fatigue levels include level I fatigue, level II fatigue, level III fatigue, level IV fatigue, and level V fatigue, with gradually increasing risks;

[0035] When the vehicle-to-vehicle ratio is 1:1, level I fatigue corresponds to low risk, level II fatigue corresponds to medium risk, and level III fatigue corresponds to high risk;

[0036] When the vehicle-to-vehicle ratio is 1:2, level II fatigue corresponds to medium risk, level III fatigue corresponds to high risk, and level IV fatigue corresponds to extremely high risk;

[0037] When the driver-vehicle ratio is 1:3, level III fatigue corresponds to high risk, level IV fatigue corresponds to extremely high risk, and level V fatigue corresponds to emergency risk.

[0038] Furthermore, the present invention generates safety warning measures according to the graded warning strategy, specifically including:

[0039] Based on the fatigue level classification results, determine whether to enable the corresponding safety warning;

[0040] When the fatigue level reaches Level I, a yellow warning is activated and a voice prompt appears: "The current fatigue level is low, please remain alert."

[0041] When the fatigue level reaches Level II, an orange warning is activated, and a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of supervised vehicles", and the vehicle automatically downshifts. If the vehicle-to-vehicle ratio is ≥ 2, the vehicle-to-vehicle ratio is automatically reduced by one gear.

[0042] When the fatigue level reaches Level III, a red alert is activated, forcing the driver to take a break, locking the operating interface, notifying the backup safety officer to take over one vehicle, and enabling automated driving assistance for the remaining vehicles.

[0043] When the fatigue level is IV or V, an emergency response is initiated, all vehicles are taken over, automatic driving is triggered, the safety officer exits the supervision queue, and the backup safety officer takes over.

[0044] A second aspect of the present invention provides a cloud cabin safety officer fatigue identification and warning system in a multi-vehicle supervision scenario, comprising a central processing center, a fatigue risk identification unit, and a safety warning unit;

[0045] The central processing center is used for data exchange and command distribution, and supports two-way communication with the fatigue risk identification unit and the safety warning unit;

[0046] The fatigue risk identification unit is used to obtain multimodal data in different driver-vehicle ratio scenarios in real time and perform fatigue identification by constructing a differentiated sub-model system;

[0047] The safety warning unit is used to determine the fatigue level according to the fatigue identification result, and adopt a graded warning strategy to generate safety warning measures.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are:

[0049] This invention achieves precise capture of fatigue characteristics in different driver-to-vehicle ratio scenarios through the application of differentiated data collection strategies and LSTM model-attention mechanisms, thereby improving the accuracy of fatigue identification. The introduction of a federated learning framework effectively integrates information from sub-models for different driver-to-vehicle ratio scenarios, addressing the shortcomings of traditional models in terms of cross-scenario continuity and penetration effects, and improving the robustness of fatigue identification. In addition, the combination of a fatigue risk matrix and a graded warning strategy significantly improves driving safety in multi-vehicle supervision scenarios and reduces the risk of remote driving supervision failure due to fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a structural block diagram of the cloud cabin safety officer fatigue identification and warning system in a multi-vehicle supervision scenario in an embodiment of the present invention.

[0051] Figure 2 4 is a structural block diagram of a fatigue risk identification unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the scope of implementation of the present invention is not limited thereto.

[0053] like Figures 1 to 2 As shown, the cloud cabin safety officer fatigue identification and warning method in a multi-vehicle supervision scenario described in this embodiment is adapted to the scenario where the vehicle-to-person ratio changes dynamically, and specifically includes the following steps:

[0054] Step S100: According to different human-to-vehicle ratio scenarios (human-to-vehicle ratio∈{1:1, 1:2, 1:3}), obtain the corresponding multimodal data under each human-to-vehicle ratio scenario.

[0055] Specifically, multimodal data directly related to the driver-vehicle ratio (1:1 / 1:2 / 1:3) is collected in real time through multiple source devices to provide objective and dynamic input support for the fatigue identification model. This multimodal data includes physiological state data, behavioral operation data, and vehicle operation data. Physiological state data includes blink rate, EEG alpha wave frequency, and heart rate variability; behavioral operation data includes mouse or gamepad operation frequency, visual gaze duration distribution, and operation reaction time; and vehicle operation data includes vehicle speed, acceleration, lane departure times, and multi-vehicle trajectory safety.

[0056] In view of the different fatigue causes and feature weights of cloud cabin safety officers in different human-vehicle ratio scenarios, this embodiment adopts a differentiated data collection strategy:

[0057] When the driver-vehicle ratio is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, heart rate variability, supplemented by mouse operation frequency, single-screen gaze time, and lane deviation times;

[0058] When the driver-vehicle ratio is 1:2, multitasking-related indicators are added, including operation reaction time, gaze shift frequency, and following distance;

[0059] When the human-vehicle ratio is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, multi-vehicle trajectory safety and takeover delay.

[0060] For example, a professional eye tracker is installed at the cloud cabin safety officer's workstation to monitor blink rate in real time. The Emotiv Epoc X wireless EEG device, manufactured by Emotiv Systems, captures stable alpha wave signals. Mouse operation activity is recorded via a system background program. Lane departures are monitored using a virtual simulated vehicle operating environment. The eye tracker collects blink data at a sampling rate of 120Hz, while the EEG device collects EEG signals at a sampling rate of 1000Hz. Alpha wave frequencies are extracted after bandpass filtering. Mouse operation frequency is counted every 10 minutes. Lane departures are determined in real time using a pre-set algorithm based on the positional relationship between the virtual vehicle and the lane markings. The resulting data shows a blink rate of 12 times per minute, an alpha wave frequency of 11Hz, a mouse operation frequency of 8 times per 10 minutes, and a lane departure rate of 2 times per 10 minutes.

[0061] Step S200: Build a differentiated sub-model system for different driver-to-vehicle ratio scenarios through the LSTM model-attention mechanism, and generate a personalized fatigue identification sub-model that adapts to the fatigue rules in each driver-to-vehicle ratio scenario; each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs the sub-fatigue identification results.

[0062] Specifically, when the human-vehicle ratio is 1:1, a single-layer LSTM model plus a fixed attention mechanism is used, in which the physiological basic indicators have the highest weight, forming sub-model A; when the human-vehicle ratio is 1:2, a two-layer LSTM model plus a dynamic attention mechanism is used. The dynamic attention mechanism introduces the human-vehicle ratio coefficient to dynamically adjust the feature weights, forming sub-model B; when the human-vehicle ratio is 1:3, a bidirectional LSTM model plus an adaptive attention mechanism is used, the weight of neurocognitive indicators is increased, and a dynamic weight adjustment mechanism triggered by the covariance of multiple vehicle trajectories is set, forming sub-model C.

[0063] Specifically, a LSTM model equipped with an attention mechanism is trained using multimodal data collected by cloud cabin safety officers in different passenger-to-vehicle ratio scenarios to establish a personalized fatigue identification sub-model for each passenger-to-vehicle ratio. The LSTM model, through a gating mechanism, captures the long-term evolution of individual fatigue, solving the problem of mining nonlinear fatigue features over long periods of time. The attention layer uses data-driven learning to weight fatigue features, dynamically capturing the most beneficial features for fatigue identification as the passenger-to-vehicle ratio changes. This addresses the impact of passenger-to-vehicle ratio on fatigue features and enhances the model's ability to accurately and rapidly identify fatigue features in various passenger-to-vehicle ratio scenarios. The LSTM model uses input, forget, and output gates to control information selection and extraction, mining fatigue features from time series information. Layer normalization stabilizes network training and enhances adaptability to fatigue characteristics across various passenger-to-vehicle ratios. The attention layer increases the weights of key features based on the effectiveness of fatigue feature indicators at each passenger-to-vehicle ratio, improving the adaptability of the personalized fatigue identification sub-model to changes in passenger-to-vehicle ratios and thus enhancing fatigue identification accuracy.

[0064] This embodiment addresses the significant differences in fatigue evolution between different driver-vehicle ratios by utilizing layer normalization in the LSTM model. The formula is as follows: ;

[0065] in, is the output of LSTM at time t, and are the mean and standard deviation of LSTM output under the vehicle-to-people ratio r, is a small constant that avoids division by zero, and These are trainable scaling and translation parameters that are not erased by model aggregation. This normalizes the hidden state of the LSTM model, stabilizes model training, and preserves the fatigue evolution characteristics of the driver-vehicle ratio by avoiding global aggregation.

[0066] In this way, the LSTM model plus the attention mechanism will adjust the sub-model parameters for different human-vehicle ratio scenarios and establish a differentiated model system to meet the feature capture accuracy and computational efficiency under different loads.

[0067] Furthermore, the specific process for generating personalized fatigue identification sub-models includes: first, normalizing the multimodal data to eliminate the influence of different dimensions; then, performing principal component analysis (PCA) on the normalized individual optimal fatigue characteristic indicators to extract key features; finally, using the sample principal components as input to an LSTM model, with the sample status value (e.g., mild fatigue = 0, moderate fatigue = 1, severe fatigue = 2) as output. Through information transfer and error backpropagation algorithms within the LSTM model neurons, the network coefficients are adjusted to establish a personalized fatigue identification sub-model for the specific driver-to-vehicle ratio scenario. Each sub-model is trained using multimodal data collected from its respective driver-to-vehicle ratio scenario.

[0068] Each sub-model learns key individual fatigue characteristics under the influence of the driver-vehicle ratio through the attention mechanism, which is of great significance to improving the model's fatigue identification efficiency and personalization level. Therefore, this embodiment retains the attention layer to avoid averaging and smoothing out the safety officer's personalized fatigue characteristics under different driver-vehicle ratios. The weight calculation of the attention layer is as follows: ;

[0069] in, is the normalized output vector of the layer, is the attention weight parameter of time t obtained from training, r is the ratio of people to vehicles, It is the global adjustment coefficient of the attention weight of the human-vehicle pairing.

[0070] Step S300: Generate a global fatigue identification model by fusing parameter information of each personalized fatigue identification sub-model through multi-task federated learning; and obtain a comprehensive fatigue identification result by integrating fatigue characteristics and sub-fatigue identification results of each personalized fatigue identification sub-model through the global fatigue identification model.

[0071] Specifically, using the multi-task federated learning method, the parameter information of the personalized fatigue identification sub-model under different human-vehicle ratio scenarios is regularly weighted and aggregated according to the data volume and local loss function value of the sub-model to obtain a global fatigue identification model that contains commonalities and individualities across scenarios.

[0072] Specifically, by introducing layer normalization and an attention mechanism into the sub-models for each driver-vehicle ratio, the fatigue characteristics of each driver-vehicle ratio are retained while the sub-models are aggregated. Based on the establishment of a personalized fatigue identification sub-model for each driver-vehicle ratio scenario, the multi-task federated learning aggregator collects fatigue characteristics and model information for each driver-vehicle ratio and regularly uploads the fatigue characteristics and identification results for each driver-vehicle ratio to the multi-task federated learning aggregator for global model updates. The multi-task federated learning aggregator then uses a weighted average method to weightedly aggregate and adaptively adjust the sub-model parameters of the sub-models based on the fatigue characteristics information for different driver-vehicle ratio scenarios, using the data volume of each driver-vehicle ratio to form a fusion strategy for fatigue identification information for each driver-vehicle ratio and generate a new round of global fatigue identification models.

[0073] This fusion action leverages the statistical patterns of fatigue characteristics across different driver-to-vehicle ratio scenarios and also acts as a weighted aggregation of sub-model parameters to form a new global fatigue identification model that can adapt to various driver-to-vehicle ratios. The fusion action is based on multi-source weighted aggregation of sub-model parameters and fatigue characteristics across different driver-to-vehicle ratio scenarios, forming a global fatigue identification model that incorporates both commonalities and specific characteristics across scenarios.

[0074] The global fatigue identification model is driven by fused multi-person-to-vehicle ratio data and can capture common fatigue patterns across different ratios. This process continuously improves the adaptability of each sub-model to both common and individual fatigue characteristics through periodic aggregation optimization. Each sub-model continuously learns common characteristics from the global fatigue identification model, thereby enhancing the robustness of the identification system.

[0075] Specifically, parameter information of the sub-models under each driver-to-vehicle ratio scenario, including gradient update information, model weights, fatigue characteristics, and sub-fatigue identification results, is regularly collected. Based on the fatigue characteristic information under different driver-to-vehicle ratio scenarios, the sub-model parameters of each sub-model are weighted aggregated and adaptively adjusted using the data volume under each driver-to-vehicle ratio scenario to form a fusion strategy for the fatigue characteristics of each driver-to-vehicle ratio and generate a new round of global fatigue identification models. The sub-model parameters of each sub-model include gradient update information and model weights. The fatigue characteristic information under different driver-to-vehicle ratio scenarios includes fatigue characteristics and sub-fatigue identification results.

[0076] The new global fatigue identification model outputs a fused global fatigue identification result based on the fatigue characteristics and sub-fatigue identification results of each sub-model. This fused global fatigue identification result is compared with the actual state value and optimized using an error feedback adjustment mechanism. The multi-task federated learning distributor sends the optimized global model parameters to each sub-model, which then iteratively updates them based on the optimized global model parameters. This process is repeated until the fused global fatigue identification result meets the convergence criteria, resulting in a realistic global fatigue identification model. The global fatigue identification model comprehensively evaluates the fatigue characteristics and sub-fatigue identification results extracted from each sub-model to output the final comprehensive fatigue identification result. The optimized global model parameters include the latest gradient value and gradient direction.

[0077] Specifically, the fused global fatigue identification result is compared with the actual state value to obtain a deviation value; if the deviation value is greater than or equal to a preset threshold, the deviation value is used to optimize the global model parameters of the global fatigue identification model, and the optimized global model parameters are sent to each sub-model. Each sub-model is updated according to the optimized global model parameters and trained with the corresponding multimodal data to output the sub-fatigue identification result. The aggregator then regularly collects the parameter information of each updated sub-model to update the global model parameters; in this way, until the deviation value is less than the preset threshold, a global fatigue identification model that meets the actual fatigue identification capability of the cloud cabin safety officer is obtained, thereby improving the robustness of fatigue identification.

[0078] During actual operation, when the system identifies the current driver-to-vehicle ratio scenario (for example, switching to a 1:3 ratio), it calls the sub-model for that ratio to perform fatigue identification. This sub-model has been imbued with the sub-model parameters of the global fatigue identification model and retains the common fatigue characteristics of multiple vehicle ratios. During actual fatigue identification, the system first calls the corresponding sub-model based on the current driver-to-vehicle ratio scenario for fatigue identification and outputs the sub-fatigue identification results.

[0079] In this embodiment, when the human-vehicle ratio scenario is switched, the hidden state data of the LSTM model under the previous human-vehicle ratio scenario and the fatigue feature data within T1 time are retained, and the hidden state data of the previous human-vehicle ratio scenario is used as the initial input of the sub-model of the current human-vehicle ratio scenario. The fatigue feature data of the sub-model of the previous human-vehicle ratio scenario is assigned a first weight, and the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is assigned a second weight. The weight is updated every T2 time until the second weight of the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is equal to 1. At this time, the sub-model of the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

[0080] Specifically, the T1 time can be determined based on the actual time window of fatigue effect penetration before and after the human-vehicle ratio switch. For example, setting T1 = 2 hours means retaining the hidden state data of the LSTM model in the previous human-vehicle ratio scenario and the fatigue feature data within 2 hours. Initially, the first weight can be set to 0.7, the second weight can be set to 0.3, and T2 can be set to 30 minutes. As time goes by, the second weight increases by 0.1 every 30 minutes until the second weight is equal to 1. This ensures that the previously accumulated fatigue level naturally transitions to the current human-vehicle ratio scenario, avoiding fatigue "resetting" and accurately identifying the fatigue state transition of the cloud cabin safety officer before and after the switch.

[0081] From the perspective of fatigue identification, the importance of the same fatigue feature often varies in different driver-to-vehicle ratio scenarios. For example, working hours are much more sensitive to fatigue in a 1:3 driver-to-vehicle ratio than in a 1:1 ratio. A global fatigue identification model is used to adaptively modify feature weighting, automatically assigning weights based on the significance of features such as working hours, eye movements, and operational characteristics in new scenarios. This not only maintains the continuity of fatigue accumulation effects, but also strengthens fatigue sensitivity in new scenarios after switching, enabling accurate fatigue identification and risk warning.

[0082] For example, when the driver-vehicle ratio switches from 1:3 or 1:2 to 1:1 (or vice versa), the system retains the output of the sub-model and its internal state (such as the hidden state of the LSTM) before the switch, and uses it as the initial input for fatigue identification in the new scenario. This ensures that the previously accumulated fatigue level naturally transitions to the current driver-vehicle ratio scenario, preventing fatigue from being "reset." Then, based on the global fatigue identification model and combined with the fatigue identification results before the switch, the feature weights and sub-model parameters for the current driver-vehicle ratio are adaptively adjusted. This reweights features such as work duration, operation amplitude, and eye movement characteristics according to their importance in the new scenario, reflecting the differences in fatigue feature sensitivity across different driver-vehicle ratio scenarios. Finally, through this state transfer and weight reconstruction mechanism, the fatigue permeation effect across scenarios is preserved while fully leveraging the global fatigue law to optimize feature contribution, achieving accurate fatigue identification and continuous risk warning in different driver-vehicle ratio scenarios.

[0083] Step S500: Based on the comprehensive fatigue identification result, combined with the driver-vehicle ratio parameter and the fatigue risk matrix, fatigue levels are divided, and safety warning measures are generated according to the graded warning strategy.

[0084] Specifically, based on the comprehensive fatigue identification results and combined with the driver-to-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine the fatigue level; the fatigue levels include level I fatigue, level II fatigue, level III fatigue, level IV fatigue and level V fatigue with gradually increasing risks; among them, when the driver-to-vehicle ratio is 1:1, level I fatigue corresponds to low risk, level II fatigue corresponds to medium risk, and level III fatigue corresponds to high risk; when the driver-to-vehicle ratio is 1:2, level II fatigue corresponds to medium risk, level III fatigue corresponds to high risk, and level IV fatigue corresponds to extremely high risk; when the driver-to-vehicle ratio is 1:3, level III fatigue corresponds to high risk, level IV fatigue corresponds to extremely high risk, and level V fatigue corresponds to emergency risk.

[0085] The graded warning strategy generates safety warning measures, including: determining whether to activate the corresponding safety warning based on the fatigue level classification results;

[0086] When the fatigue level is Level I, a yellow warning is activated, and a voice prompt says "The current fatigue level is low, please remain alert"; when the fatigue level is Level II, an orange warning is activated, and a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of supervised vehicles", and the gear is automatically downshifted. If the driver-vehicle ratio is ≥2, the driver-vehicle ratio is automatically reduced by one gear (such as switching from a driver-vehicle ratio of 1:3 to a driver-vehicle ratio of 1:2); when the fatigue level is Level III, a red warning is activated, a forced rest is taken, the operation interface is locked, the backup safety officer is notified to take over one vehicle, and automatic driving assistance is enabled for the remaining vehicles; when the fatigue level is Level IV or V, an emergency response is initiated, all vehicles are taken over, automatic driving is triggered, the safety officer exits the supervision queue, and the backup safety officer takes over.

[0087] like Figures 1 to 2 As shown, this embodiment also provides a cloud cabin safety officer fatigue identification and warning system in a multi-vehicle supervision scenario, which adapts to the scenario of dynamic changes in the ratio of people to vehicles, and specifically includes a central processing center, a fatigue risk identification unit, and a safety warning unit;

[0088] The central processing center is used for data interaction and command distribution, and supports two-way communication with the fatigue risk identification unit and the safety warning unit; the fatigue risk identification unit is used to obtain multimodal data in different human-vehicle ratio scenarios in real time, and perform fatigue identification by constructing a differentiated sub-model system; the safety warning unit is used to determine the fatigue level based on the fatigue identification results, and adopt a graded warning strategy to generate safety warning measures.

[0089] Specifically, the fatigue risk identification unit includes a fatigue data acquisition module, a human-vehicle ratio sub-model construction module, and a multi-task federated learning module. The fatigue data acquisition module is used to obtain multimodal data corresponding to the current human-vehicle ratio scenario according to different human-vehicle ratio scenarios. The human-vehicle ratio sub-model construction module is used to construct a differentiated sub-model system through the LSTM model-attention mechanism, generate a personalized fatigue identification sub-model, and use the personalized fatigue identification sub-model to extract fatigue features from the current multimodal data to obtain sub-fatigue identification results. The multi-task federated learning module is used to fuse the parameter information of each personalized fatigue identification sub-model through multi-task federated learning to generate a global fatigue identification model; and integrate the fatigue features and sub-fatigue identification results of each personalized fatigue identification sub-model through the global fatigue identification model to obtain a comprehensive fatigue identification result.

[0090] The safety warning unit includes a fatigue level classification module and a graded warning execution module; the fatigue level classification module is used to determine the fatigue level based on the fatigue identification results, and to perform fatigue level classification based on the human-vehicle ratio parameters; the graded warning execution module determines whether to activate the corresponding safety warning based on the fatigue level classification results.

[0091] For example, in a scenario with a 1:1 vehicle-to-human ratio, the eye tracker collects blink data at a sampling rate of 120Hz, and the EEG device collects EEG signals at a sampling rate of 1000Hz. The alpha wave frequency is extracted after bandpass filtering. The mouse operation frequency is counted every 10 minutes. The number of lane departures is determined in real time based on the positional relationship between the virtual vehicle and the lane line using a preset algorithm. The final data obtained is a blink frequency of 12 times / minute, an EEG alpha wave frequency of 11Hz, a mouse operation frequency of 8 times / 10 minutes, and a lane departure frequency of 2 times / 10 minutes.

[0092] The collected physiological and behavioral data were processed using a standardization method to make the data conform to a standard normal distribution with a mean of 0 and a standard deviation of 1, so as to facilitate model input;

[0093] The single-layer LSTM model contains 128 neurons and uses a 5-minute time step. Normalized data is fed into the model sequentially in time series. A fixed attention mechanism calculates the correlation between each feature and fatigue status, assigning a weight of 0.4 to EEG alpha waves, a weight of 0.3 to blink frequency, and the remaining 0.3 to other features. The model operates on the input data using trained parameters, ultimately outputting a moderate fatigue result. The fatigue risk identification unit transmits the moderate fatigue result to the central processing center, which in turn sends the fatigue result to the safety warning unit. Based on the fatigue risk matrix, the safety warning unit determines the current orange alert for Level II fatigue. Based on the orange alert, the graded alert execution module sends instructions to the cloud cabin safety officer's work interface, prompting a prominent red warning box to appear on the interface and a voice prompt through the built-in speaker: "Fatigue level escalated, recommended to reduce supervised vehicles." The central processing center also stores this alert information and related data for subsequent analysis and review.

[0094] For example, using a 1:3 vehicle-to-person ratio scenario, a high-precision pupil detector was used to monitor pupil diameter in real time. Multi-vehicle trajectory safety data was acquired through an onboard positioning system and real-time cloud communication technology. The takeover delay was determined by recording the time interval between the system issuing the takeover command and the safety officer actually taking action. Ultimately, data was collected showing a pupil diameter of 1.8mm, a multi-vehicle trajectory covariance of 0.85, and a takeover delay of 3.5 seconds.

[0095] Then, the collected neurocognitive indicators and multi-vehicle collaborative data are standardized and input into the bidirectional LSTM model. The bidirectional LSTM model can extract features from time series data from both the forward and reverse directions, and capture the information in the data more comprehensively. The adaptive attention mechanism automatically increases the vehicle data weight to 0.5 based on the characteristics of the input data when it detects that the covariance of multi-vehicle trajectories is greater than 0.6. After calculation, the model finally outputs the result of level V fatigue, and the warning level is emergency response. The graded warning execution module immediately triggers the full vehicle takeover command according to the warning level. First, the autonomous driving system starts quickly and sends control instructions to each supervisory vehicle through cloud communication technology to adjust the vehicle's speed, direction and other driving parameters to ensure the safe operation of the vehicle;

[0096] The hierarchical warning execution module notifies the backup safety officer to take over via text messages, voice calls, and other means. The system records the time points of the entire intervention process. In addition, the central processing center backs up detailed data of the emergency intervention, including all collected data, model calculation processes, and warning trigger records, to facilitate subsequent accident analysis and system optimization.

[0097] The above description is only a preferred embodiment of the present invention. Therefore, any equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the protection scope of the patent application of the present invention.

Claims

1. A method for identifying and warning fatigue of cloud cabin safety officers in a multi-vehicle monitoring scenario, characterized by: The following steps are involved: According to different scenarios of human-vehicle ratio, obtain the corresponding multimodal data under each scenario; Through the LSTM model-attention mechanism, a differentiated sub-model system is constructed for different driver-vehicle ratio scenarios, generating a personalized fatigue identification sub-model that adapts to the fatigue patterns in each driver-vehicle ratio scenario. Each personalized fatigue identification sub-model extracts fatigue features from the corresponding multimodal data and outputs sub-fatigue identification results; Through multi-task federated learning, the parameter information of each personalized fatigue identification sub-model is integrated to generate a global fatigue identification model; The fatigue characteristics and sub-fatigue identification results of each personalized fatigue identification sub-model are integrated through the global fatigue identification model to obtain the comprehensive fatigue identification result; Based on the comprehensive fatigue identification results, combined with the driver-to-vehicle ratio parameters and the fatigue risk matrix, fatigue levels are divided, and safety warning measures are generated according to the graded warning strategy.

2. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: The multimodal data includes physiological state data, behavioral operation data and vehicle operation data; among them, physiological state data includes blinking frequency, EEG alpha wave frequency and heart rate variability; behavioral operation data includes mouse or handle operation frequency, visual gaze duration distribution and operation reaction time; vehicle operation data includes vehicle speed, acceleration, number of lane deviations and multi-vehicle trajectory safety.

3. The cloud cabin safety officer fatigue identification and warning method according to claim 2 is characterized in that: The method of obtaining multimodal data corresponding to each driver-vehicle ratio scenario according to different driver-vehicle ratio scenarios specifically includes: When the driver-vehicle ratio is 1:1, the focus is on collecting basic physiological indicators, including blink frequency, EEG alpha wave frequency, heart rate variability, supplemented by mouse operation frequency, single-screen gaze time, and lane deviation times; When the driver-vehicle ratio is 1:2, multitasking-related indicators are added, including operation reaction time, gaze shift frequency, and following distance; When the human-vehicle ratio is 1:3, new neurocognitive indicators and multi-vehicle collaboration data are added, including pupil diameter, head pitch angle, multi-task switching error rate, and multi-vehicle trajectory safety and takeover delay.

4. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: The generation of a personalized fatigue identification sub-model adapted to fatigue patterns in various driver-vehicle ratio scenarios specifically includes: Normalize multimodal data to eliminate the influence of different dimensions; Perform principal component analysis on the normalized individual optimal fatigue characteristic index to extract key features; The principal components of the samples are used as the input of the LSTM model, and the state values ​​of the samples are used as the output. The network coefficients are adjusted through information transmission of LSTM model neurons and error back propagation algorithm to establish a personalized fatigue identification sub-model for this driver-vehicle ratio scenario.

5. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: The LSTM model-attention mechanism is used to construct a differentiated sub-model system for different driver-vehicle ratio scenarios, specifically including: When the vehicle-to-person ratio is 1:1, a single-layer LSTM model with a fixed attention mechanism is used, in which the physiological basis indicator has the highest weight. When the vehicle-to-pedestrian ratio is 1:2, a two-layer LSTM model plus a dynamic attention mechanism is used. The dynamic attention mechanism introduces the vehicle-to-pedestrian ratio coefficient to dynamically adjust the feature weights. When the vehicle-to-pedestrian ratio is 1:3, a bidirectional LSTM model plus an adaptive attention mechanism is adopted, the weight of neurocognitive indicators is increased, and a dynamic weight adjustment mechanism triggered by the covariance of multiple vehicle trajectories is set.

6. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: When the human-vehicle ratio scenario switches, the hidden state data of the LSTM model under the previous human-vehicle ratio scenario and the fatigue feature data within T1 time are retained, and the hidden state data of the previous human-vehicle ratio scenario is used as the initial input of the sub-model of the current human-vehicle ratio scenario. The fatigue feature data of the sub-model of the previous human-vehicle ratio scenario is assigned a first weight, and the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is assigned a second weight. The weight is updated every T2 time until the second weight of the fatigue feature data of the sub-model of the current human-vehicle ratio scenario is equal to 1. At this time, the sub-model of the current human-vehicle ratio scenario is fully adapted to the current human-vehicle ratio scenario.

7. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: The multi-task federated learning is used to fuse the parameter information of each personalized fatigue identification sub-model to generate a global fatigue identification model; The fatigue characteristics and sub-fatigue identification results of each personalized fatigue identification sub-model are integrated through the global fatigue identification model to obtain the comprehensive fatigue identification results, including: Regularly collect parameter information of the sub-models under each driver-vehicle ratio scenario, including gradient update information, model weights, fatigue characteristics, and sub-fatigue identification results; Based on fatigue characteristic information under different driver-to-vehicle ratio scenarios, the weighted aggregation and adaptive adjustment of sub-model parameters of each sub-model are driven by the data volume under each driver-to-vehicle ratio scenario, forming a fusion strategy for fatigue characteristics of each driver-to-vehicle ratio and generating a new round of global fatigue identification model; The new round of global fatigue identification model outputs the fused global fatigue identification result based on the fatigue characteristics and sub-fatigue identification results of each sub-model. The fused global fatigue identification result is compared with the actual state value. The new round of global fatigue identification model is optimized using the error feedback adjustment mechanism, and the optimized global model parameters are sent to each sub-model. Each sub-model is iteratively updated according to the optimized global model parameters. The above process is repeated until the fused global fatigue identification result meets the convergence condition and a global fatigue identification model that conforms to reality is obtained. The global fatigue identification model outputs a comprehensive fatigue identification result.

8. The cloud cabin safety officer fatigue identification and warning method according to claim 1 is characterized in that: The fatigue level is divided based on the comprehensive fatigue identification results, combined with the driver-vehicle ratio parameter and the fatigue risk matrix, specifically including: Based on the fatigue identification results and combined with the driver-vehicle ratio parameter h, a fatigue risk matrix is ​​constructed to determine fatigue levels; fatigue levels include level I fatigue, level II fatigue, level III fatigue, level IV fatigue, and level V fatigue, with gradually increasing risks; When the vehicle-to-vehicle ratio is 1:1, level I fatigue corresponds to low risk, level II fatigue corresponds to medium risk, and level III fatigue corresponds to high risk; When the vehicle-to-vehicle ratio is 1:2, level II fatigue corresponds to medium risk, level III fatigue corresponds to high risk, and level IV fatigue corresponds to extremely high risk; When the driver-vehicle ratio is 1:3, level III fatigue corresponds to high risk, level IV fatigue corresponds to extremely high risk, and level V fatigue corresponds to emergency risk.

9. The cloud cabin safety officer fatigue identification and warning method according to claim 8 is characterized in that: The generation of safety warning measures according to the graded warning strategy specifically includes: Based on the fatigue level classification results, determine whether to enable the corresponding safety warning; When the fatigue level reaches level I, a yellow warning is activated and a voice prompt appears: "The current fatigue level is low, please remain alert." When the fatigue level reaches Level II, an orange warning is activated, and a red warning box pops up on the interface, displaying "Fatigue level upgraded, it is recommended to reduce the number of supervised vehicles", and the vehicle automatically downshifts. If the vehicle-to-vehicle ratio is ≥ 2, the vehicle-to-vehicle ratio is automatically reduced by one gear. When the fatigue level reaches Level III, a red alert is activated, forcing the driver to take a break, locking the operating interface, notifying the backup safety officer to take over one vehicle, and enabling automated driving assistance for the remaining vehicles. When the fatigue level is IV or V, an emergency response is initiated, all vehicles are taken over, automatic driving is triggered, the safety officer exits the supervision queue, and the backup safety officer takes over.

10. A cloud cabin safety officer fatigue identification and warning system in a multi-vehicle monitoring scenario, characterized by: It includes a central processing center, a fatigue risk identification unit, and a safety warning unit; The central processing center is used for data exchange and command distribution, and supports two-way communication with the fatigue risk identification unit and the safety warning unit; The fatigue risk identification unit is used to obtain multimodal data in different driver-vehicle ratio scenarios in real time and perform fatigue identification by constructing a differentiated sub-model system; The safety warning unit is used to determine the fatigue level according to the fatigue identification result, and adopt a graded warning strategy to generate safety warning measures.

Images