Operator brain assessment and monitoring system based on manned-unmanned flight cooperative task

Abstract

The invention relates to the technical field of brain power assessment, in particular to an operator brain power assessment and monitoring system based on manned-unmanned flight cooperative tasks, which comprises the following steps: acquiring brain wave signals, voice fluctuation and finger muscle activity data of an operator, and generating operator brain power load data; setting a personalized load reference value and a threshold range of an operator; based on multi-dimensional regression analysis and a self-adaptive deep learning model, a nonlinear relation model between task characteristics and brain load is constructed, the load state is dynamically evaluated, and a real-time load change prediction value is output; predicting a future high-load risk period according to the fluctuation mode of the mental load fluctuation curve; dynamically adjusting task parameters in a multi-task execution process; according to the invention, the dynamic reference setting of the mental load of the operator is realized, so that the individuation and accuracy of load evaluation are ensured, the influence of key task characteristics on the load can be quickly reduced when the load of the operator is increased, and an efficient and accurate load reduction effect is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of mental assessment, and in particular to a system for assessing and monitoring the mental performance of an operator based on a manned-unmanned flight collaborative task. Background Art

[0002] In modern flight missions, especially in complex scenarios of manned-unmanned collaborative operations, the operator's mental workload has become an important factor that directly affects the success or failure of the mission. As the mission scenarios become increasingly complex, operators need to handle multiple tasks, monitor multiple devices, and make decisions quickly, which significantly increases their mental workload. High mental workload not only reduces the operator's reaction speed and decision-making accuracy, but can also lead to excessive fatigue, which in turn causes mission errors or safety accidents.

[0003] Most of the current mental workload monitoring technologies are based on traditional physiological data monitoring methods, such as heart rate and galvanic skin response. Although these methods can reflect the operator's physiological state to a certain extent, they often cannot accurately capture the changes in the operator's mental workload under different task types, and are also difficult to cope with the complexity and real-time requirements of task scenarios. The existing mental workload monitoring system lacks accurate assessment of the dynamic changes in mental workload in a multi-tasking environment, cannot adapt to changes in task requirements in real time, and is difficult to support real-time task adjustments.

[0004] In addition, the operator's mental workload is often significantly affected by task characteristics, such as path planning complexity, the number of coordinated UAVs, and the diversity of task objectives. These task characteristics affect the operator's load status in a nonlinear and interactive manner. However, the existing technology is relatively insufficient in the study of the relationship between task characteristics and mental workload, and it is difficult to accurately identify the specific impact of key task characteristics on the load. This makes it impossible for the system to prioritize the adjustment of high-impact factors when the load exceeds the safety range, resulting in lag and inaccuracy in load management and task optimization. Summary of the invention

[0005] The present invention provides an operator mental assessment and monitoring system based on a manned-unmanned flight collaborative task.

[0006] The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks includes:

[0007] Multi-source data acquisition module: collects the operator's brain wave signals, voice fluctuations and finger muscle activity data, and performs multi-level noise reduction and signal enhancement processing on each data source to generate processed operator mental workload data;

[0008] Load benchmark generation module: Based on the operator's historical mental workload data, set the operator's personalized load benchmark value and threshold range;

[0009] Task characteristics-mental workload association module: used to analyze the impact of task characteristics on the operator's mental workload. Based on multidimensional regression analysis and adaptive deep learning models, a nonlinear relationship model between task characteristics and mental workload is constructed to identify the contribution of task characteristics to mental workload. The load status is dynamically evaluated according to different task types (such as complex path planning, multi-number collaboration between manned and unmanned aerial vehicles, and multi-target collaboration), and real-time load change prediction values ​​are output;

[0010] Dynamic mental workload assessment module: Based on the trained nonlinear relationship model, the current task type is input, the operator's current mental workload data is output, and a mental workload fluctuation curve is generated in real time. Combined with the mental workload benchmark model, the current mental workload data is compared with the benchmark value and threshold range of the load benchmark model to determine whether it exceeds the threshold range, and the future high-load risk period is predicted based on the fluctuation pattern of the mental workload fluctuation curve;

[0011] Real-time adaptive task adjustment module: Based on the output of the dynamic mental workload assessment module, the adaptive control algorithm is used to dynamically adjust task parameters during multi-task execution, including the number of coordinated drones, task complexity, and complex path optimization, to keep the operator's mental workload data within a safe threshold.

[0012] Optionally, the multi-source data acquisition module includes:

[0013] The brain wave acquisition unit collects the operator's brain wave signals through a wearable brain wave sensor , using adaptive filtering and wavelet decomposition algorithms to perform multi-level noise reduction on the signal and enhance the signal in key frequency bands, the key frequency bands include α waves and β waves to improve the accuracy and stability of EEG data; the adaptive wave is used to eliminate noise in the EEG signal, using an adaptive filter model;

[0014] Wavelet decomposition of EEG signals Perform multi-layer wavelet decomposition;

[0015] The voice fluctuation collection unit collects the operator's voice signal in real time through a wearable microphone, applies a noise suppression algorithm to remove environmental background noise, and extracts and enhances the voice tone and frequency changes through voice spectrum analysis to capture the correlation characteristics between voice fluctuations and brain power fluctuations;

[0016] The finger muscle activity acquisition unit obtains finger muscle activity data through the electromyographic sensor placed on the operator's hand, uses time domain filtering to remove noise in the electromyographic signal, and uses spectrum analysis to extract key features of finger muscle activity to enhance the stability of the electromyographic signal;

[0017] The multimodal data fusion unit receives brain wave, voice fluctuation and muscle activity data, and performs synchronous fusion processing on various types of data through weighted fusion and feature alignment to generate processed operator mental workload data, providing an accurate and reliable multi-source data basis for the mental workload assessment module. The weighted fusion generates mental workload data, and the expression is:

[0018] ,in, is the integrated mental workload data, , , Respectively represent the processing results of brain waves, speech fluctuations and electromyographic signals; , , They represent the weighting coefficients of brain waves, speech fluctuations and electromyographic signals respectively.

[0019] Optionally, the load benchmark generation module specifically includes:

[0020] The data preprocessing unit is used to receive the operator's historical mental workload data and normalize the mental workload data under different time and tasks. Mapped to a uniform numerical range [0,1];

[0021] The benchmark and threshold setting unit uses normalized mental workload data , combined with the mean of historical data and standard deviation To determine the operator's individual load baseline and threshold range.

[0022] Optionally, the operator's personalized load baseline value And the threshold range is expressed as:

[0023] ;

[0024] ;

[0025] ;

[0026] in, is the standard deviation, is the average value of historical load data, is the number of historical data points, is a safety factor used to adjust the threshold range.

[0027] Optionally, the task characteristic-mental workload association module specifically includes:

[0028] Task feature extraction unit, extracts task feature vector from the current task ,in, is the number of task features, each task feature (i∈m) represents specific mission parameters, including path planning complexity, number of cooperative UAVs, and multi-target density of the mission, which are numerically quantified to form a mission characteristic data set;

[0029] Multidimensional regression analysis unit, task characteristic vector and operator historical mental workload data Conduct multidimensional regression analysis, establish a nonlinear relationship model, calculate the linear and nonlinear relationship between task characteristics and load changes, and obtain the predicted mental workload value ;

[0030] The adaptive deep learning model unit uses the deep neural network model of the deep learning model to model the complex nonlinear relationship between task characteristics and mental workload based on the results of multidimensional regression analysis. The loss function of the deep learning model is to minimize the difference between the predicted workload and the actual workload:

[0031] ,in, is the loss function of the deep learning model, The model predicts The mental workload value of each task instance, is the actual mental workload value in historical data, is the number of samples; the model parameters are optimized through back propagation and gradient descent, so that the model can adapt to the complex nonlinear relationship between different task characteristics;

[0032] The task characteristic contribution recognition unit performs task characteristic sensitivity analysis on the trained model and calculates the contribution of each task characteristic. Contribution to mental workload, i.e. contribution of task characteristics ;

[0033] Dynamic load forecasting unit, based on the current task characteristic vector The trained model is used to predict the current mental workload change trend in real time and output the real-time workload change prediction value. and provide dynamic load status prediction to subsequent modules to support real-time load monitoring and task parameter adjustment.

[0034] Optionally, the nonlinear relationship model is expressed as:

[0035] ,in, represents the 𝑖th task characteristic parameter, which represents the quantized value of the task-related characteristics (such as path planning complexity, number of coordinated UAVs, multi-target coordination, etc.), Indicates Task characteristic parameters, is the predicted mental workload value, is the intercept term of the regression model, which indicates the initial mental workload without the influence of task characteristics. For task characteristics The linear regression coefficient of Direct impact on mental workload, is the second-order interaction regression coefficient between task characteristics, which is used to model the nonlinear effects between task characteristics.

[0036] Optionally, the task characteristic contribution is calculated as: ,in, For task characteristics Predicting mental workload The contribution of Sensitivity to changes in mental workload, Indicates the predicted load Task Characteristics The partial derivative of is used to calculate the task characteristics degree of impact.

[0037] Optionally, in the dynamic mental workload assessment module, the formula is: , input the task characteristic vector of the current task type , output the operator's current mental workload value ,in, is a function expression of the deep learning model, representing the task feature vector The nonlinear relationship between the mental workload and the training time (obtained from the nonlinear relationship model based on deep neural networks mentioned above), It is a set of parameters of the deep learning model, including all weights and biases, which predicts the current mental workload. Stored in time series data to update and plot mental workload fluctuation curves in real time , as the task execution time The fluctuation curve is expressed as: ;

[0038] Get the operator's personalized load baseline value and threshold range , the current mental workload data Compare with the threshold range to determine whether the operator's mental workload exceeds the safe range:

[0039] like or , it is considered that the operator's mental workload exceeds the safety threshold and an overload alarm will be issued.

[0040] Using volatility curves The load trend and fluctuation frequency are analyzed by using the trend analysis method to determine whether there is an upward trend in load or high-frequency fluctuation.

[0041] Optionally, the task real-time adaptive adjustment module specifically includes:

[0042] Task load monitoring unit: receives the output of the dynamic mental load assessment module to determine whether the operator's mental load exceeds the safety threshold range;

[0043] Adaptive control algorithm unit: based on operator mental workload data When the safety threshold is exceeded, the contribution of task characteristics Determine the optimal adjustment direction and prioritize the task characteristics with high contribution according to the impact of the task characteristics on the load. , to reduce mental workload, adjust the amount The calculation of is as follows:

[0044] ,in, It is a task feature The adjustment amount reflects the adjustment range of the task characteristics. is the adjustment coefficient, which is used to control the adjustment speed and sensitivity of the task parameters. : Deviation between current load data and reference value;

[0045] Task parameter adjustment unit, based on the adjustment amount Prioritize adjusting the characteristics of tasks with high contribution ,The task parameters are dynamically adjusted during multi-tasking execution to keep the operator's load within the safety threshold.

[0046] Optionally, the task parameter adjustment unit includes:

[0047] Adjustment of the number of coordinated drones: If the contribution of the number of drones High, it reduces the number of drones that operators need to manage simultaneously : ;

[0048] Task complexity adjustment: If the contribution of task complexity If the task is high, the operation steps or automation level in the task scenario are simplified, and the frequency of information interaction is reduced to reduce the task load of the operator;

[0049] Path optimization adjustment: If the contribution of path planning complexity If the load is high, the path planning scheme with low load will be given priority to reduce the complexity of the path and reduce the load on the operator.

[0050] Beneficial effects of the present invention:

[0051] The present invention collects the operator's brain waves, voice fluctuations and electromyographic signal data in real time through a multi-source data acquisition module to generate mental workload data. In combination with the task characteristic-mental workload association module, a nonlinear relationship model between task characteristics and mental workload is constructed through multidimensional regression analysis and an adaptive deep learning model, so that the system can output accurate mental workload change prediction values ​​in real time according to different task characteristics. In addition, by setting personalized benchmark values ​​and threshold ranges based on historical load data, a dynamic benchmark setting for the operator's mental workload is achieved, thereby ensuring the personalization and accuracy of load assessment.

[0052] The present invention can identify and preferentially adjust the task characteristics that have the greatest impact on the operator's load by calculating the contribution of task characteristics to mental workload. When the system load exceeds the safety threshold range, the system uses an adaptive control algorithm to preferentially adjust high-impact task characteristics according to the contribution, such as the number of coordinated drones, task complexity, and path optimization. The adjustment amount is based on the deviation between the operator's current load data and the benchmark value, combined with the dynamic setting of the task characteristic contribution to ensure the accuracy and flexibility of the adjustment. The contribution guides the adjustment mechanism, so that the system can quickly reduce the impact of key task characteristics on the load when the operator's load increases, achieve efficient and accurate load reduction effects, and significantly improve the intelligence and response speed of the system.

[0053] The present invention can monitor the load change pattern of the operator in real time and predict the future high-load risk period through mental load fluctuation curve and load trend analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A schematic diagram of functional modules of an evaluation and monitoring system according to an embodiment of the present invention;

[0056] Figure 2 Schematic diagram of a task characteristic-mental workload association module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0058] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiments", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0059] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0060] like Figure 1-Figure 2 As shown, the operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks includes:

[0061] Multi-source data acquisition module: collects the operator's brain wave signals, voice fluctuations and finger muscle activity data, and performs multi-level noise reduction and signal enhancement processing on each data source to generate processed operator mental workload data;

[0062] Load benchmark generation module: Based on the operator's historical mental workload data, set the operator's personalized load benchmark value and threshold range;

[0063] Task characteristics-mental workload association module: used to analyze the impact of task characteristics on the operator's mental workload. Based on multidimensional regression analysis and adaptive deep learning models, a nonlinear relationship model between task characteristics and mental workload is constructed to identify the contribution of task characteristics to mental workload. The load status is dynamically evaluated according to different task types (such as complex path planning, multi-number collaboration between manned and unmanned aerial vehicles, and multi-target collaboration), and real-time load change prediction values ​​are output;

[0064] Dynamic mental workload assessment module: Based on the trained nonlinear relationship model, the current task type is input, the operator's current mental workload data is output, and a mental workload fluctuation curve is generated in real time. Combined with the mental workload benchmark model, the current mental workload data is compared with the benchmark value and threshold range of the load benchmark model to determine whether it exceeds the threshold range, and the future high-load risk period is predicted based on the fluctuation pattern of the mental workload fluctuation curve;

[0065] Real-time adaptive task adjustment module: Based on the output of the dynamic mental workload assessment module, the adaptive control algorithm is used to dynamically adjust task parameters during multi-task execution, including the number of coordinated drones, task complexity, and complex path optimization, to keep the operator's mental workload data within the safety threshold.

[0066] The multi-source data acquisition module includes:

[0067] The brain wave acquisition unit collects the operator's brain wave signals through a wearable brain wave sensor , adaptive filtering and wavelet decomposition algorithm are used to reduce the multi-level noise of the signal and enhance the signal of the key frequency band, which includes α wave and β wave to improve the accuracy and stability of EEG data; adaptive wave is used to eliminate the noise in the EEG signal, and the adaptive filter model is used, and the expression is: ,in, is the filtered EEG signal, It is the original collected brain wave signal. is the delayed reference noise signal, It is The filter weight coefficient of order, is the order of the filter, which is selected according to the noise complexity;

[0068] Wavelet decomposition of EEG signals Perform multi-layer wavelet decomposition to extract Waves (8-12Hz) and The signal of wave (12-30Hz) is expressed as: ,in, It is the original brain wave signal. is the wavelet decomposition coefficient, The first coefficients, It is Layer A wavelet function, is the number of wavelet decomposition levels;

[0069] The voice fluctuation collection unit collects the operator's voice signal in real time through a wearable microphone, applies a noise suppression algorithm to remove environmental background noise, and extracts and enhances the voice tone and frequency changes through voice spectrum analysis to capture the correlation characteristics between voice fluctuations and brain power fluctuations;

[0070] Adaptive noise suppression: Adaptive noise suppression is performed on the original speech signal to reduce the interference of environmental noise on the speech fluctuation characteristics. Based on the adaptive gain function , the expression is as follows:

[0071] ;

[0072] ;

[0073] in, It represents the expression of the denoised speech signal in the frequency domain. represents the spectrum representation of the original speech signal, represents the spectrum representation of the ambient noise, represents the adaptive gain function, which controls the strength of noise suppression;

[0074] The voice frequency characteristics and tone changes are extracted by fast Fourier transform to analyze the operator's emotional (brain power) fluctuations, which can be expressed as:

[0075] ,in, represents the frequency domain representation of the speech signal, represents the original collected speech signal, Indicates the signal length (number of sampling points), obtained through spectrum analysis The main frequency components and energy distribution in the speech are used to judge the tone characteristics and emotional fluctuations of speech;

[0076] The finger muscle activity acquisition unit obtains finger muscle activity data through the electromyographic sensor placed on the operator's hand, uses time domain filtering to remove noise in the electromyographic signal, and uses spectrum analysis to extract key features of finger muscle activity to enhance the stability of the electromyographic signal;

[0077] Time domain filtering: EMG signals Perform low-pass filtering to reduce high-frequency noise. The expression is: ,in, is the filtered electromyographic signal, is the original electromyographic signal, It is The filter coefficients of order, is the filter order;

[0078] The amplitude analysis of the filtered electromyographic signal is performed, and the expression is:

[0079] ,in, is the average amplitude of the electromyographic signal, is the filtered electromyographic signal, is the number of sampling points;

[0080] Extract the frequency characteristics of finger muscle activity through fast Fourier transform:

[0081] ,in, is the frequency domain representation of the electromyographic signal, is the raw electromyographic signal;

[0082] The multimodal data fusion unit receives brain wave, voice fluctuation and muscle activity data, and performs synchronous fusion processing on various types of data through weighted fusion and feature alignment to generate processed operator mental workload data, providing an accurate and reliable multi-source data basis for the mental workload assessment module. The weighted fusion generates mental workload data, and the expression is:

[0083] ,in, is the integrated mental workload data, , , Respectively represent the processing results of brain waves, speech fluctuations and electromyographic signals; , , They represent the weighting coefficients of brain waves, speech fluctuations and electromyographic signals respectively.

[0084] Feature alignment: The sampling frequencies of multi-source data are different, and the interpolation method is used to align the features: ,in, represents the interpolated alignment data, , represents the adjacent data points. The time series index representing the data is the position of the sampled data point in time.

[0085] The load benchmark generation module specifically includes:

[0086] The data preprocessing unit is used to receive the operator's historical mental workload data and use the normalization method to convert the mental workload data under different time and tasks into Mapped to a uniform numerical range [0,1];

[0087] The benchmark and threshold setting unit uses normalized mental workload data , combined with the mean of historical data and standard deviation To determine the operator's individual load baseline and threshold range.

[0088] Operator's personalized load base And the threshold range is expressed as:

[0089] ;

[0090] ;

[0091] ;

[0092] in, is the standard deviation, is the average value of historical load data, is the number of historical data points, is a safety factor used to adjust the threshold range.

[0093] Task characteristics-mental workload related modules specifically include:

[0094] Task feature extraction unit, extracts task feature vector from the current task ,in, is the number of task features, each task feature (i∈m) represents specific mission parameters, including path planning complexity, number of cooperative UAVs, and multi-target density of the mission, which are numerically quantified to form a mission characteristic data set;

[0095] Multidimensional regression analysis unit, task feature vector and operator historical mental workload data Conduct multidimensional regression analysis, establish a nonlinear relationship model, calculate the linear and nonlinear relationship between task characteristics and load changes, and obtain the predicted mental workload value ;

[0096] The adaptive deep learning model unit uses the deep neural network model of the deep learning model to model the complex nonlinear relationship between task characteristics and mental workload based on the results of multidimensional regression analysis. The loss function of the deep learning model is to minimize the difference between the predicted workload and the actual workload:

[0097] ,in, is the loss function of the deep learning model, The model predicts The mental workload value of each task instance, is the actual mental workload value in historical data, is the number of samples; the model parameters are optimized through back propagation and gradient descent, so that the model can adapt to the complex nonlinear relationship between different task characteristics;

[0098] The task characteristic contribution recognition unit performs task characteristic sensitivity analysis on the trained model and calculates the contribution of each task characteristic. Contribution to mental workload, i.e. contribution of task characteristics ;

[0099] Dynamic load forecasting unit, based on the current task characteristic vector The trained model is used to predict the current mental workload change trend in real time and output the real-time workload change prediction value. and provide dynamic load status prediction to subsequent modules to support real-time load monitoring and task parameter adjustment.

[0100] The nonlinear relationship model is expressed as:

[0101] ,in, represents the 𝑖th task characteristic parameter, which represents the quantized value of the task-related characteristics (such as path planning complexity, number of coordinated UAVs, multi-target coordination, etc.), Indicates Task characteristic parameters, is the predicted mental workload value, is the intercept term of the regression model, which indicates the initial mental workload without the influence of task characteristics. For task characteristics The linear regression coefficient of Direct impact on mental workload, is the second-order interaction regression coefficient between task characteristics, which is used to model the nonlinear effects between task characteristics.

[0102] The task characteristic contribution is calculated as: ,in, For task characteristics Predicting mental workload The contribution of Sensitivity to changes in mental workload, Indicates the predicted load Task Characteristics The partial derivative of is used to calculate the task characteristics The degree of influence of the main task characteristics on mental workload is identified through contribution analysis, providing a basis for task allocation and task parameter adjustment.

[0103] In the dynamic mental workload assessment module, the formula is: , input the task characteristic vector of the current task type , output the operator's current mental workload value ,in, is a function expression of the deep learning model, representing the task feature vector The nonlinear relationship between the mental workload and the training time (obtained from the nonlinear relationship model based on deep neural networks mentioned above), It is a set of parameters of the deep learning model, including all weights and biases, obtained after model training. This formula is the functional expression of the aforementioned nonlinear relationship model, which converts the current mental workload prediction value Stored in time series data to update and plot mental workload fluctuation curves in real time , as the task execution time The fluctuation curve is expressed as: ;

[0104] Get the operator's personalized load baseline value and threshold range , the current mental workload data Compare with the threshold range to determine whether the operator's mental workload exceeds the safe range:

[0105] like or , it is considered that the operator's mental workload exceeds the safety threshold and an overload alarm will be issued.

[0106] Using volatility curves Analyze the load trend and fluctuation frequency based on the data, and use the trend analysis method to determine whether there is an upward trend in load or high-frequency fluctuation:

[0107] , For the The mental workload data at each time point; by setting the window width , calculate the nearest The average load change trend of a period; if the trend value continues to rise and approaches , then warn of the future high-load risk period;

[0108] High load period prediction: by analyzing the slope of current load changes and fluctuation patterns, predicting the possibility of future load exceeding the threshold. If it remains positive and is accompanied by a high frequency of fluctuations, it is considered that a high-load risk period is about to begin. Warnings can be given in advance and task parameters can be adjusted appropriately to alleviate future loads.

[0109] The task real-time adaptive adjustment module specifically includes:

[0110] Task load monitoring unit: receives the output of the dynamic mental load assessment module to determine whether the operator's mental load exceeds the safety threshold range;

[0111] Adaptive control algorithm unit: based on operator mental workload data When the safety threshold is exceeded, the contribution of task characteristics Determine the optimal adjustment direction and prioritize the task characteristics with high contribution according to the impact of the task characteristics on the load. , to reduce mental workload, adjust the amount The calculation of is as follows:

[0112] ,in, It is a task feature The adjustment amount reflects the adjustment range of the task characteristics. is the adjustment coefficient, which is used to control the adjustment speed and sensitivity of the task parameters. : Deviation between current load data and reference value;

[0113] Task parameter adjustment unit, based on the adjustment amount Prioritize adjusting the characteristics of tasks with high contribution ,The task parameters are dynamically adjusted during multi-tasking execution to keep the operator's load within the safety threshold.

[0114] The task parameter adjustment unit includes:

[0115] Adjustment of the number of coordinated drones: If the contribution of the number of drones High, it reduces the number of drones that operators need to manage simultaneously : ;

[0116] Task complexity adjustment: If the contribution of task complexity If the task is high, the operation steps or automation level in the task scenario are simplified, and the frequency of information interaction is reduced to reduce the task load of the operator;

[0117] Path optimization adjustment: If the contribution of path planning complexity If the load is high, the path planning scheme with low load will be given priority to reduce the complexity of the path and reduce the load on the operator.

[0118] After adjusting the task parameters, re-monitor the changes in the operator's mental workload data. If the workload returns to within the safety threshold range, maintain the current adjustment; if the expected effect is not achieved, continue to iteratively adjust the task parameters until the workload data returns to a safe range.

[0119] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks, characterized by: include: Multi-source data acquisition module: collects the operator's brain wave signals, voice fluctuations and finger muscle activity data, and performs multi-level noise reduction and signal enhancement processing on each data source to generate the operator's mental workload data; Load benchmark generation module: Based on the operator's historical mental workload data, set the operator's personalized load benchmark value and threshold range; Task characteristics-mental workload association module: used to analyze the impact of task characteristics on the operator's mental workload. Based on multidimensional regression analysis and adaptive deep learning models, a nonlinear relationship model between task characteristics and mental workload is constructed to identify the contribution of task characteristics to mental workload. The module also dynamically evaluates the load status according to different task types and outputs real-time load change prediction values. Dynamic mental workload assessment module: Based on the trained nonlinear relationship model, the current task type is input, the operator's current mental workload data is output, and a mental workload fluctuation curve is generated in real time. Combined with the mental workload benchmark model, the current mental workload data is compared with the benchmark value and threshold range of the load benchmark model to determine whether it exceeds the threshold range, and the future high-load risk period is predicted based on the fluctuation pattern of the mental workload fluctuation curve; Real-time adaptive task adjustment module: Based on the output of the dynamic mental workload assessment module, the adaptive control algorithm is used to dynamically adjust task parameters during multi-task execution, including the number of coordinated drones, task complexity, and complex path optimization, to keep the operator's mental workload data within the safety threshold.

2. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 1 is characterized in that: The multi-source data acquisition module includes: The brain wave acquisition unit collects the operator's brain wave signals through a wearable brain wave sensor , using adaptive filtering and wavelet decomposition algorithm to perform multi-level noise reduction on the signal and enhance the signal of key frequency bands, the key frequency bands include α wave and β wave; the adaptive wave is used to eliminate the noise in the EEG signal, using an adaptive filter model; Wavelet decomposition of EEG signals Perform multi-layer wavelet decomposition; The voice fluctuation collection unit collects the operator's voice signal in real time through a wearable microphone, applies a noise suppression algorithm to remove environmental background noise, and extracts and enhances the voice tone and frequency changes through voice spectrum analysis to capture the correlation characteristics between voice fluctuations and brain power fluctuations; The finger muscle activity acquisition unit acquires finger muscle activity data through an electromyographic sensor placed on the operator's hand, uses time domain filtering to remove noise from the electromyographic signal, and uses spectrum analysis to extract key features of finger muscle activity; The multimodal data fusion unit receives brain wave, voice fluctuation and muscle activity data, and performs synchronous fusion processing on various types of data through weighted fusion and feature alignment to generate processed operator mental workload data. The weighted fusion generates mental workload data, and the expression is: ,in, is the integrated mental workload data, , , Respectively represent the processing results of brain waves, speech fluctuations and electromyographic signals; , , They represent the weighting coefficients of brain waves, speech fluctuations and electromyographic signals respectively.

3. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 1 is characterized in that: The load benchmark generation module specifically includes: The data preprocessing unit is used to receive the operator's historical mental workload data and use the normalization method to convert the mental workload data under different time and tasks into Mapped to a uniform numerical range [0,1]; The benchmark and threshold setting unit uses normalized mental workload data , combined with the mean of historical data and standard deviation To determine the operator's individual load baseline and threshold range.

4. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 3 is characterized in that: The operator's personalized load base value And the threshold range is expressed as: ； ； ； in, is the standard deviation, is the average value of historical load data, is the number of historical data points, is a safety factor used to adjust the threshold range.

5. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 3 is characterized in that: The task characteristic-mental workload association module specifically includes: Task feature extraction unit, extracts task feature vector from the current task ,in, is the number of task features, each task feature Represents specific mission parameters, including path planning complexity, number of coordinated UAVs, and multi-target density of the mission, and forms a mission characteristic dataset through numerical quantification of characteristics; Multidimensional regression analysis unit, task characteristic vector and operator historical mental workload data Conduct multidimensional regression analysis, establish a nonlinear relationship model, calculate the linear and nonlinear relationship between task characteristics and load changes, and obtain the predicted mental workload value ; The adaptive deep learning model unit uses the deep neural network model of the deep learning model to model the complex nonlinear relationship between task characteristics and mental workload based on the results of multidimensional regression analysis. The loss function of the deep learning model is to minimize the difference between the predicted workload and the actual workload: ,in, is the loss function of the deep learning model, The model predicts The mental workload value of each task instance, is the actual mental workload value in historical data, is the sample size; The task characteristic contribution recognition unit performs task characteristic sensitivity analysis on the trained model and calculates the contribution of each task characteristic. Contribution to mental workload, i.e. contribution of task characteristics ; Dynamic load forecasting unit, based on the current task characteristic vector The trained model is used to predict the current mental workload change trend in real time and output the real-time workload change prediction value. .

6. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 5 is characterized in that: The nonlinear relationship model is expressed as: ,in, represents the 𝑖th task characteristic parameter, which represents the quantized value of the task-related characteristics. Indicates Task characteristic parameters, is the predicted mental workload value, is the intercept term of the regression model, which indicates the initial mental workload without the influence of task characteristics. For task characteristics The linear regression coefficient of Direct impact on mental workload, is the second-order interaction regression coefficient between task characteristics, which is used to model the nonlinear effects between task characteristics.

7. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 5 is characterized in that: The task characteristic contribution is calculated as: ,in, For task characteristics Predicting mental workload The contribution of Sensitivity to changes in mental workload, Indicates the predicted load Task Characteristics The partial derivative of is used to calculate the task characteristics degree of impact.

8. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 6 is characterized in that: In the dynamic mental workload assessment module, the formula is: , input the task characteristic vector of the current task type , output the operator's current mental workload value ,in, is a function expression of the deep learning model, representing the task feature vector The nonlinear relationship between It is a set of parameters of the deep learning model, including all weights and biases, which predicts the current mental workload. Stored in time series data to update and plot mental workload fluctuation curves in real time , as the task execution time The fluctuation curve is expressed as: ; Get the operator's personalized load baseline value and threshold range , the current mental workload data Compare with the threshold range to determine whether the operator's mental workload exceeds the safe range: like or , it is considered that the operator's mental load exceeds the safety threshold and an overload alarm will be issued; Using volatility curves The load trend and fluctuation frequency are analyzed by using the trend analysis method to determine whether there is an upward trend in load or high-frequency fluctuation.

9. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 8 is characterized in that: The task real-time adaptive adjustment module specifically includes: Task load monitoring unit: receives the output of the dynamic mental load assessment module to determine whether the operator's mental load exceeds the safety threshold range; Adaptive control algorithm unit: based on operator mental workload data When the safety threshold is exceeded, the contribution of task characteristics Determine the optimal adjustment direction and prioritize the task characteristics with high contribution according to the impact of the task characteristics on the load. , to reduce mental workload, adjust the amount The calculation of is as follows: ,in, It is a task feature The adjustment amount reflects the adjustment range of the task characteristics. is the adjustment coefficient, which is used to control the adjustment speed and sensitivity of the task parameters. : Deviation between current load data and reference value; Task parameter adjustment unit, based on the adjustment amount Prioritize adjusting the characteristics of tasks with high contribution ,The task parameters are dynamically adjusted during multi-tasking execution to keep the operator's load within the safety threshold.

10. The operator mental assessment and monitoring system based on manned-unmanned flight collaborative tasks according to claim 9 is characterized in that: The task parameter adjustment unit includes: Adjustment of the number of coordinated drones: If the contribution of the number of drones High, it reduces the number of drones that operators need to manage simultaneously : ; Task complexity adjustment: If the contribution of task complexity If the value is high, the operation steps or automation level in the task scenario will be simplified, and the frequency of information interaction will be reduced; Path optimization adjustment: If the contribution of path planning complexity If the load is high, the path planning scheme with low load will be given priority to reduce the complexity of the path and reduce the load on the operator.

Images