User workload state processing method and apparatus based on multimodal data, and device

Abstract

A user workload state processing method and apparatus based on multimodal data, and a device. The method comprises: acquiring state information of a first user; determining workload information of the first user on the basis of the acquired state information, wherein the state information comprises at least two of the following information: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and feeding back the workload information to a first management account, and formulating a training scheme matched with the workload information. According to the embodiments of the present application, workload of users can be identified on the basis of multimodal data of users.

Description

Personnel workload status processing method, device and equipment based on multimodal data

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 28, 2023, with application number 202311841098.9 and application name “Method, device and equipment for processing personnel workload status based on multimodal data”, the entire contents of which are incorporated by reference into this application.

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 29, 2023, with application number 202311865043.1 and application name “Pilot workload identification method, device and equipment based on human intelligence”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the technical field of physiological signal recognition, and in particular to a method, apparatus and device for processing workload status based on multimodal data. Background Art

[0004] Workload refers to the amount of work the human body endures per unit time. When workload is high, workers' performance levels are elevated, leading to significant physical and mental strain. With less remaining capacity, workers are more likely to be unable to cope with emergencies, leading to various accidents. Therefore, timely identification and adjustment of workers' workload during work is an effective way to ensure workplace safety.

[0005] Summary of the Invention

[0006] The present application provides a method, apparatus, and device for processing workload status based on multimodal data, which can identify a user's workload based on the user's multimodal data.

[0007] In a first aspect, an embodiment of the present application provides a method for processing workload status based on multimodal data, comprising:

[0008] collecting status information of the first user;

[0009] Determining workload information of the first user based on the collected status information, wherein the status information includes at least two of the following: physiological information, eye movement information, electroencephalogram information, brain functional imaging information, motion capture information, spatiotemporal collection information, behavioral collection information, facial expression, and status information;

[0010] The workload information is fed back to the first management account, and a training plan that matches the workload information is formulated.

[0011] The method collects status information of the first user, determines the workload information of the first user based on the status information, and thus can identify the user's workload based on the multimodal status information of the first user; moreover, the workload information is fed back to the first management account and a training plan that matches the workload information is formulated, so that a training plan can be formulated to adapt to the workload information of the first user, thereby ensuring the scientific nature and safety of the first user's training.

[0012] In one possible implementation, when the state information of the first user includes a first electroencephalogram signal and a first near-infrared functional brain imaging signal of the first user, determining the workload information of the first user based on the collected state information includes:

[0013] extracting a first feature of the first electroencephalogram signal and a second feature of the first near-infrared functional brain imaging signal respectively;

[0014] Workload information of the first user is determined based on the first feature and the second feature.

[0015] In one possible implementation, determining the workload information of the first user according to the first feature and the second feature includes:

[0016] Fusing the first feature and the second feature to obtain a first fused feature;

[0017] The first fusion feature is input into a preset workload identification model, and the workload information output by the workload identification model is used as the workload information of the first user.

[0018] In a possible implementation, the first feature includes a first time series feature and a first image feature, and the second feature includes a second time series feature and a second image feature;

[0019] The first feature and the second feature are fused to obtain a first fused feature, including:

[0020] Fusing the first time series feature and the second time series feature to obtain a third time series feature;

[0021] Fusing the first image feature and the second image feature to obtain a third image feature;

[0022] The third time series feature and the third image feature are fused to obtain a first fused feature.

[0023] In a possible implementation, fusing the first time series feature and the second time series feature to obtain a third time series feature includes:

[0024] Converting the first temporal feature into a fourth image feature;

[0025] converting the second time series feature into a fifth image feature;

[0026] The fourth image feature and the fifth image feature are fused to obtain a sixth image feature, and the sixth image feature is used as the third time series feature.

[0027] In a possible implementation, the first time series feature includes: a time domain feature, a frequency domain feature, a time-frequency domain feature, a nonlinear feature and / or a brain functional connectivity feature; and / or,

[0028] The first image feature includes: a 2D atlas and / or a 3D densely connected network; and / or,

[0029] The second time series feature includes: the concentration of oxygenated and deoxygenated hemoglobin, the beta index of a single channel, and / or the Pearson correlation coefficient between channels; and / or,

[0030] The second image features include: a 3D channel activation map, a correlation matrix between channels, and / or a brain network connection map.

[0031] In a possible implementation, the workload information includes: a workload level and / or a workload type.

[0032] In a possible implementation, the first feature includes a first time series feature and a first image feature, and the second feature includes a second time series feature and a second image feature; the workload information includes: a workload level and a workload type;

[0033] Determining workload information of the first user based on the first feature and the second feature includes:

[0034] Fusing the first time series feature and the second time series feature to obtain a third time series feature; identifying the workload level of the first user according to the third time series feature;

[0035] The first image feature and the second image feature are fused to obtain a third image feature; and the workload type of the first user is identified according to the third image feature.

[0036] In one possible implementation, identifying the workload level of the first user according to the third time series feature includes:

[0037] The third time series feature is input into the load level identification model, and the workload level output by the load level identification model is used as the workload level of the first user.

[0038] In one possible implementation, identifying the workload type of the first user according to the third image feature includes:

[0039] The third image feature is input into the load type recognition model, and the workload type output by the load type recognition model is used as the workload type of the first user.

[0040] In a possible implementation, the first time series feature includes: a time domain feature, a frequency domain feature, a time-frequency domain feature, a nonlinear feature and / or a brain functional connectivity feature; and / or,

[0041] The first image feature includes: a brain topography map, and / or a power spectrum topography map; and / or,

[0042] The second time series feature includes: the concentration of oxygenated and deoxygenated hemoglobin, the beta index of a single channel, and / or the Pearson correlation coefficient between channels; and / or,

[0043] The second image features include: 3D channel activation map, complex brain network, default brain network, correlation heat map, and / or brain network connection map.

[0044] In one possible implementation, workload types include: auditory, visual, attentional, executive, and / or planning.

[0045] In one possible implementation, extracting a second image feature of the first near-infrared brain functional imaging signal includes:

[0046] A brain network map with different functions is extracted from the first near-infrared brain functional imaging signal as the second image feature.

[0047] In a possible implementation, before extracting the first feature of the first EEG signal and the second feature of the first near-infrared brain functional imaging signal, the method further includes:

[0048] The first electroencephalogram signal and the first near-infrared functional brain imaging signal are preprocessed respectively.

[0049] In a possible implementation, the method further includes:

[0050] Perform early warning processing of user workload based on workload information.

[0051] In one possible implementation, performing early warning processing of user workload based on workload information includes:

[0052] When the workload level in the workload information is not lower than a preset level and the workload type in the workload information includes a first type, an alarm is issued for the first type of workload type.

[0053] In one possible implementation, performing early warning processing of user workload based on workload information includes:

[0054] When the workload level in the workload information is not lower than the preset level and the workload type in the workload information includes the second type, determine the proportion of the second type in the multiple workload information obtained within the preset first time period, and when the proportion is not lower than the preset proportion threshold, alarm for the second type of workload type.

[0055] In one possible implementation, formulating a training plan that matches the workload information includes:

[0056] A training plan corresponding to the workload information is obtained as the training plan for the first user.

[0057] In one possible implementation, a method for training a workload identification model includes:

[0058] Acquiring a second EEG signal, a second near-infrared functional brain imaging signal, and a workload information tag of a second user; the workload information tag is used to record workload information of the second user when acquiring the second EEG signal and the second near-infrared functional brain imaging signal of the second user;

[0059] extracting a first feature of the second electroencephalogram signal and a second feature of the second near-infrared functional brain imaging signal respectively;

[0060] fusing the first feature of the second EEG signal and the second feature of the second near-infrared functional brain imaging signal to obtain a second fused feature;

[0061] The workload recognition model is trained using the second fusion feature and the workload information label as samples.

[0062] In one possible implementation, the workload identification model includes multiple network layers;

[0063] The workload recognition model is trained using the second fusion feature and the workload information label as samples, including:

[0064] The second fused feature is input into the workload identification model to obtain the output result of each network layer in the forward propagation process;

[0065] Determine the target results for each network layer based on workload information tags;

[0066] For each network layer, the similarity between the output result of the network layer and the target result is calculated, and the weight of the network layer is adjusted according to the similarity.

[0067] In one possible implementation, a method for training the load level identification model includes:

[0068] Acquiring a third EEG signal, a third near-infrared functional brain imaging signal, and a workload level tag of a third user; the workload level tag is used to record the workload level of the third user when the third EEG signal and the third near-infrared functional brain imaging signal of the third user are acquired;

[0069] extracting a fourth time series feature of the third EEG signal, and extracting a fifth time series feature of the third near-infrared functional brain imaging signal;

[0070] fusing the fourth time series feature of the third EEG signal and the fifth time series feature of the third near-infrared functional brain imaging signal to obtain a sixth time series feature;

[0071] The sixth time series feature and workload level label are used as samples to train the workload level recognition model.

[0072] In one possible implementation, a method for training a load type identification model includes:

[0073] Acquiring a third EEG signal, a third near-infrared functional brain imaging signal, and a workload type tag of a third user; the workload type tag is used to record the workload type of the third user when acquiring the third EEG signal and the third near-infrared functional brain imaging signal of the third user;

[0074] extracting a fourth image feature of the third electroencephalogram signal, and extracting a fifth image feature of the third near-infrared functional brain imaging signal;

[0075] fusing the fourth image feature of the third EEG signal and the fifth image feature of the third near-infrared functional brain imaging signal to obtain a sixth image feature;

[0076] The workload type recognition model is trained using the sixth image feature and the workload type label as samples.

[0077] In a second aspect, an embodiment of the present application provides a workload status processing device based on multimodal data, comprising:

[0078] A collection module, configured to collect status information of the first user;

[0079] a determination module, configured to determine workload information of the first user based on the collected status information, wherein the status information includes at least two of the following: physiological information, eye movement information, electroencephalogram information, brain functional imaging information, motion capture information, spatiotemporal collection information, behavioral collection information, facial expression, and status information;

[0080] The processing module is used to feed back the workload information to the first management account and formulate a training plan that matches the workload status level.

[0081] In a possible implementation, the state information of the first user includes: a first electroencephalogram signal and a first near-infrared brain functional imaging signal of the first user; and the determination module may include:

[0082] a feature extraction module, configured to extract a first feature of the first EEG signal and a second feature of the first near-infrared functional brain imaging signal;

[0083] The information determination module is configured to determine workload information of the first user based on the first feature and the second feature.

[0084] In one possible implementation, the information determination module includes:

[0085] A feature fusion module, configured to fuse the first feature and the second feature to obtain a first fused feature;

[0086] The identification module is used to input the first fusion feature into a preset workload identification model, and use the workload information output by the workload identification model as the workload information of the first user.

[0087] In a possible implementation, the method further includes: a first training module, wherein:

[0088] The acquisition module is further configured to: acquire a second EEG signal, a second near-infrared brain function imaging signal, and a workload information tag of the second user; the workload information tag is configured to record workload information of the second user when the second EEG signal and the second near-infrared brain function imaging signal of the second user are acquired;

[0089] The feature extraction module is further used to: extract a first feature of the second EEG signal and a second feature of the second near-infrared functional brain imaging signal respectively;

[0090] The feature fusion module is further used to: fuse the first feature of the second EEG signal and the second feature of the second near-infrared brain functional imaging signal to obtain a second fused feature;

[0091] The first training module is used to train the workload recognition model by using the second fusion feature and the workload information label as samples.

[0092] In one possible implementation, the information determination module includes:

[0093] A first fusion module is used to fuse the first time series feature and the second time series feature to obtain a third time series feature;

[0094] a first identification module, configured to identify a workload level of a first user according to a third time series feature;

[0095] A second fusion module is used to fuse the first image feature and the second image feature to obtain a third image feature;

[0096] The second identification module is configured to identify the workload type of the first user according to the third image feature.

[0097] In a possible implementation, the method further includes: a second training module, wherein:

[0098] The acquisition module is further configured to: acquire a third EEG signal, a third near-infrared functional brain imaging signal, and a workload level tag of a third user; the workload level tag is configured to record the workload level of the third user when the third EEG signal and the third near-infrared functional brain imaging signal of the third user are acquired;

[0099] The feature extraction module is further used to: extract a fourth time series feature of the third EEG signal, and extract a fifth time series feature of the third near-infrared functional brain imaging signal;

[0100] The first fusion module is further configured to: fuse the fourth time series feature of the third EEG signal and the fifth time series feature of the third near-infrared functional brain imaging signal to obtain a sixth time series feature;

[0101] The second training module is used to train the load level recognition model using the sixth time series feature and the workload level label as samples.

[0102] In a possible implementation, the method further includes: a third training module, wherein:

[0103] The acquisition module is further configured to: acquire a third EEG signal, a third near-infrared functional brain imaging signal, and a workload type tag of a third user; the workload type tag is configured to record the workload type of the third user when acquiring the third EEG signal and the third near-infrared functional brain imaging signal of the third user;

[0104] The feature extraction module is further used to: extract a fourth image feature of the third EEG signal, and extract a fifth image feature of the third near-infrared functional brain imaging signal;

[0105] The second fusion module is further configured to: fuse the fourth image feature of the third EEG signal and the fifth image feature of the third near-infrared functional brain imaging signal to obtain a sixth image feature;

[0106] The third training module is used to train the workload type recognition model using the sixth image feature and the workload type label as samples.

[0107] In a third aspect, the present application provides an electronic device in real time, comprising: a processor and a memory; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the electronic device to execute any one of the methods of the first aspect.

[0108] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes any one of the methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0110] FIG1A is a schematic diagram of a system architecture applicable to a method for processing workload status based on multimodal data according to an embodiment of the present application;

[0111] FIG1B is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0112] FIG2 is a flow chart of a method for processing workload status based on multimodal data according to an embodiment of the present application;

[0113] FIG3 is another flowchart of a method for processing workload status based on multimodal data according to an embodiment of the present application;

[0114] FIG4A is a schematic diagram of another flow chart of a method for processing workload status based on multimodal data according to an embodiment of the present application;

[0115] FIG4B is a schematic diagram of a fourth flow chart of a method for processing workload status based on multimodal data provided in an embodiment of the present application;

[0116] FIG5 is a flow chart of a method for training a workload identification model according to an embodiment of the present application;

[0117] FIG6 is a schematic diagram of a fifth flow chart of a method for processing workload status based on multimodal data provided in an embodiment of the present application;

[0118] FIG7 is a flow chart of a pilot workload identification method based on human intelligence according to an embodiment of the present application;

[0119] FIG8 is another flowchart of a pilot workload identification method based on human intelligence provided by an embodiment of the present application;

[0120] FIG9 is a schematic diagram of another flow chart of a pilot workload identification method based on human intelligence according to an embodiment of the present application;

[0121] FIG10 is a schematic diagram of a fourth flow chart of a pilot workload identification method based on human intelligence according to an embodiment of the present application;

[0122] FIG11 is a flow chart of a method for training a load type identification model according to an embodiment of the present application;

[0123] FIG12 is another flow chart of a method for training a load level identification model according to an embodiment of the present application;

[0124] FIG13 is a schematic diagram of a fourth flow chart of a pilot workload identification method based on human intelligence according to an embodiment of the present application;

[0125] FIG14 is a schematic structural diagram of a workload status processing device based on multimodal data provided in an embodiment of the present application;

[0126] FIG15 is another structural diagram of a workload status processing device based on multimodal data provided by an embodiment of the present application;

[0127] FIG16 is a schematic diagram of another structure of a workload status processing device based on multimodal data provided in an embodiment of the present application;

[0128] FIG17 is a schematic structural diagram of a workload identification device provided in an embodiment of the present application;

[0129] FIG18 is another structural diagram of the workload identification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0130] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0131] Workload refers to the amount of work the human body endures per unit time. When workload is high, workers' work capacity is at a high level, resulting in significant physical and mental strain. With less remaining work capacity, workers are more likely to be unable to cope with emergencies, leading to various accidents. Therefore, timely identification of workers' workload during work and adjusting it is an effective way to ensure work safety. This is especially true in scenarios where worker safety requirements are high, such as drivers driving vehicles, pilots flying aircraft, and pilots undergoing flight training. It is even more important to promptly identify workers' workload during work to ensure their work safety.

[0132] EEG data captures electrical activity in the cerebral cortex with millisecond-level temporal resolution, can extract a rich variety of characteristic values, and is highly responsive. Furthermore, with the advancement of sensor technology, the types of portable EEG devices are becoming increasingly diverse, with applications in biofeedback, emotion detection, state recognition, and brain-computer interfaces.

[0133] To this end, an embodiment of the present application provides a workload status processing method based on multimodal data, which utilizes the user's multimodal data, such as the above-mentioned EEG data, to identify the user's workload, thereby improving the safety of the user's work.

[0134] FIG1A is a schematic diagram of a system architecture applicable to a workload state processing method based on multimodal data according to an embodiment of the present application. As shown in FIG1A , the system may include: an electronic device 100, an EEG signal acquisition device 110, and a near-infrared brain function imaging device 120; wherein,

[0135] The EEG signal acquisition device 110 is used to acquire the user's EEG signals and transmit the user's EEG signals to the electronic device 100 .

[0136] The near-infrared brain function imaging device 120 is used to collect the user's near-infrared brain function imaging signals and transmit the user's near-infrared brain function imaging signals to the electronic device 100 .

[0137] In some embodiments, the above-mentioned EEG signal acquisition device 110 and near-infrared brain function imaging device 120 can also be implemented by a multimodal brain function imaging device that has both EEG signal acquisition and near-infrared brain function imaging signal acquisition functions.

[0138] The electronic device 100 is used to execute the workload status processing method based on multimodal data provided in the embodiment of the present application to realize user workload identification.

[0139] In some embodiments, the workload status processing method based on multimodal data in embodiments of the present application can be applied to identifying the workload of working users such as drivers and pilots. Taking the identification of driver workload as an example, the system shown in FIG1A can be, for example, a driver assistance system. Taking the identification of pilot workload as an example, the system shown in FIG1A can be, for example, a flight assistance system or a flight simulation system.

[0140] FIG1B is a schematic diagram of the structure of an electronic device 100 according to an embodiment of the present application. The electronic device 100 includes a processor 110, a memory 120, and the like.

[0141] Optionally, in order to improve the functions of the electronic device, the electronic device may also include one or more devices selected from the group consisting of a display screen, a camera, a speaker, an antenna, a mobile communication module, a wireless communication module, an audio module, a receiver, a microphone, a headphone jack, a charging management module, a power management module, a battery, etc., which are not limited in the embodiments of the present application.

[0142] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0143] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0144] The memory 120 can be used to store computer executable program codes, which include instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, etc.) created during the use of the electronic device 100, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120, and / or instructions stored in a memory provided in the processor.

[0145] It should be noted that, in the embodiment shown in FIG1 , the memory 120 is set in the electronic device 100 as an example. In other embodiments provided in the embodiments of the present application, the above-mentioned memory 120 may not be set in the electronic device 100. In this case, the memory 120 can be connected to the electronic device 100 through the interface provided by the electronic device 100, and then can be connected to the processor 110 in the electronic device 100.

[0146] In some embodiments, the EEG signal acquisition and near-infrared brain function imaging signal acquisition functions can be integrated into the electronic device 100. In this case, the electronic device 100 may further include an EEG signal acquisition component 103 and a near-infrared brain function imaging component 104. The EEG signal acquisition component 103 is used to acquire the user's EEG signals, and the near-infrared brain function imaging component 104 is used to acquire the user's near-infrared brain function imaging information.

[0147] In some embodiments, the above-mentioned EEG signal acquisition component 103 and near-infrared brain function imaging component 104 can also be implemented by a multimodal brain function imaging component that has both EEG signal acquisition and near-infrared brain function imaging signal acquisition functions.

[0148] Hereinafter, the workload status processing method based on multimodal data in an embodiment of the present application will be described in detail in combination with the above-mentioned system structure and the structure of the electronic device.

[0149] FIG2 is a flow chart of a workload status processing method based on multimodal data provided in an embodiment of the present application. The method can be executed by the electronic device shown in FIG1A and FIG1B .

[0150] As shown in FIG2 , the method may include:

[0151] Step 201: Collect status information of the first user.

[0152] Optionally, the above-mentioned status information may include at least two of the following information: physiological information, eye movement information, EEG information, brain functional imaging information, motion capture information, spatiotemporal acquisition information, behavioral acquisition information, facial expressions and status information.

[0153] The physiological information may include: electrocardiogram (ECG) signals and other signals related to the first user's physiology. The above information can be obtained by physiological information detection equipment such as an ECG detector.

[0154] The motion capture information is information obtained by capturing the motion of the first user, and can be obtained by capturing an image of the first user with a camera and detecting the image.

[0155] The spatiotemporal acquisition information can be obtained by encoding the video information, time information, and spatial information of the scene where the first user is located.

[0156] The behavior collection information is information obtained by collecting the behavior of the first user. Specifically, the first user's video image can be captured by a camera, and the first user's behavior can be obtained based on the user's image detection.

[0157] The facial expression and status information is information obtained by collecting the facial expression and status of the first user, and can be obtained by capturing the facial image of the first user with a camera and detecting the facial image.

[0158] In some embodiments, in this step, a first EEG signal and a first near-infrared brain functional imaging signal of the first user may be collected.

[0159] In one instance, if the method shown in Figure 2 is applied to the system shown in Figure 1A, the electronic device can control the EEG signal acquisition device to collect the first EEG signal of the first user, and control the near-infrared brain function imaging device to collect the first near-infrared brain function imaging information of the first user.

[0160] In another example, if the method shown in Figure 2 is applied to the electronic device shown in Figure 1B, the electronic device can control the EEG signal acquisition component to collect the first EEG signal of the first user, and control the near-infrared brain function imaging component to collect the first near-infrared brain function imaging information of the first user.

[0161] Optionally, the first EEG signal may be a whole-brain EEG signal, which refers to an EEG signal obtained by detecting the entire brain.

[0162] Near-infrared brain function imaging is the latest generation of brain function detection technology based on optical principles. The first near-infrared brain function imaging signal in this step is a signal obtained using near-infrared brain function imaging technology. When the brain nerves are active, the brain metabolism changes, and the hemodynamics of the cerebral cortex changes accordingly, which in turn causes changes in oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb) in the brain tissue. Near-infrared light (650-900nm) signals can penetrate human tissue and the skull and reach the cerebral cortex (3cm deep below the scalp). According to the correlation between light attenuation and changes in the concentration of chromophores (HbO and Hb) in the brain, changes in cerebral cortical hemodynamics can be detected, and then the neural activity of the brain can be inferred through the law of neurovascular coupling. The detected signal can be called a near-infrared brain function imaging signal.

[0163] Step 202: Determine the workload information of the first user based on the collected status information.

[0164] Optionally, in this step, state features may be extracted from the collected state information respectively, and the extracted multimodal features may be fused to obtain multimodal fused features, and the workload information of the first user may be determined based on the fused features.

[0165] For example, if the status information of the first user collected in step 201 includes: a first electroencephalogram signal and a first near-infrared brain functional imaging signal, step 202 can be implemented through the embodiment shown in Figure 3, or can be implemented through steps 704 to 707 in the embodiment shown in Figure 7.

[0166] As shown in FIG3 , step 202 may specifically include:

[0167] Step 301: extracting a first feature of a first EEG signal and a second feature of a first near-infrared functional brain imaging signal respectively;

[0168] Step 302: Fusing the first feature and the second feature to obtain a first fused feature;

[0169] Step 303: Input the first fusion feature into a preset workload identification model, and use the workload information output by the workload identification model as the workload information of the first user.

[0170] The following is an exemplary description of possible implementations of the above step 301.

[0171] Optionally, the first feature of the first brain electrical signal may include: a first time sequence feature and a first image feature of the first brain electrical signal.

[0172] The first time series feature of the first EEG signal refers to a time-related feature of the first EEG signal. Optionally, the first time series feature of the first EEG signal may include: a time domain feature, a frequency domain feature, a time-frequency domain feature, a nonlinear feature, and / or a brain functional connectivity feature of the first EEG signal. In some related technologies, the above features may also be referred to as indicators.

[0173] Optionally, the time domain features of the first EEG signal may include: kurtosis, skewness, mean, standard deviation, latency of event-related potential (ERP), peak value of ERP, etc. of the first EEG signal.

[0174] The meaning and representation significance of the above time domain features are shown in Table 1 below.

[0175] Table 1

[0176] Optionally, a time domain analysis method for EEG signals in related technologies may be used to extract the time domain features of the first EEG signal. The time domain analysis method may be, for example, event-related potential (ERP) or spatiotemporal modeling.

[0177] The frequency domain characteristics of the first EEG signal are used to describe the distribution of the energy, phase and other information of the first EEG signal at the frequency points, which can reflect the cognitive state of the user. Optionally, the frequency domain characteristics of the first EEG signal may include: the energy values ​​of the first EEG signal in five frequency bands, such as Delta, Theta, Alpha, Beta, and Gamma, as well as brain cognitive function characteristics such as α / β, θ / β, (α+θ) / β, (α+θ) / (α+β), θ / (α+β), and SMR (db). The above α represents the α wave of the first EEG signal, β represents the β wave of the first EEG signal, θ represents the θ wave of the first EEG signal, and SMR represents the SMR wave of the first EEG signal.

[0178] The meaning and representation significance of the above frequency bands and frequency domain features are shown in Table 2 below.

[0179] Table 2

[0180] The above-mentioned frequency domain features can be obtained by analyzing the first EEG signal through relevant frequency domain analysis methods of EEG signals, such as Fourier transform, periodogram method (welch), multi-window method, and / or autoregressive model. The specific implementation of the embodiments of this application will not be repeated here.

[0181] The Fourier transform is a fundamental frequency domain analysis method used to convert a time domain signal into a frequency domain signal. It decomposes a signal into the amplitude and phase information of its different frequency components. The Fourier transform is typically applied to steady-state signals.

[0182] The periodogram method is a method used to study the periodic components of a signal, including the autocorrelation function, cross-correlation function, and power spectral density function.

[0183] The multi-window method involves using different types of window functions, such as rectangular windows, Hamming windows, and Hanning windows, to analyze the spectral characteristics of a signal. Different window functions are suitable for different applications and signal types.

[0184] An autoregressive model is a method for modeling time series data, typically used to estimate the frequency components of a signal. Autoregressive models include autoregressive models (AR) and autoregressive-moving average (ARMA) models.

[0185] Optionally, the time-frequency domain features of the first EEG signal may include: a power value of the signal at each specific time and frequency point.

[0186] The time-frequency domain characteristics of the first EEG signal are used to indicate whether an increase (ERS) or decrease (ERD) in the signal power of a specific frequency band occurs in the corresponding region of interest (ROI) after a certain stimulus occurs. The time-frequency domain characteristics can be obtained by processing the first EEG signal using a time-frequency domain analysis method for EEG signals. The above-mentioned time-frequency domain analysis method may include: short-time Fourier transform, continuous wavelet transform, etc.

[0187] The nonlinear characteristics of the first EEG signal are used to reflect changes in the dynamic characteristics of the brain. Optionally, the nonlinear characteristics of the first EEG signal may include: the complexity of the first EEG signal, various entropies, and / or Lyapunov exponents, etc. The above nonlinear characteristics can be obtained by analyzing the first EEG signal using nonlinear dynamics.

[0188] The complexity of the first EEG signal can be used to measure the information capacity of the EEG segment, thereby reflecting the potential activity characteristics of neurons and indicating the speed at which new patterns appear in the time series as the sequence length increases. Optionally, the complexity of the first EEG signal may include: Lempel-Ziv complexity, Lempel-Ziv sorting complexity, etc.

[0189] The entropy value of the first EEG signal indicates its level of disorder. Different entropy algorithms describe information capacity from different perspectives. A decrease in entropy indicates a decrease in the ability of information to interact within the brain. Entropy values ​​for the first EEG signal can include Shannon entropy, approximate entropy, sample entropy, and sorting entropy based on the time domain. Entropy values ​​based on time and frequency include wavelet entropy and Hilbert-Huang spectral entropy.

[0190] The maximum Lyapunov exponent of the first EEG signal can quantitatively characterize the average divergence rate of adjacent orbits in the phase space, and how small initial state disturbances in the system gradually increase over time.

[0191] The functional connectivity features of the first EEG signal are used to assess the connectivity between brain regions. Each part of the brain has a unique function in human behavior, and even the simplest tasks require the collaboration of multiple brain regions. Optionally, the functional connectivity features of the first EEG signal may include features based on coherence, features based on phase synchronization, features based on generalized synchronization, and features based on Granger causality.

[0192] The first image feature of the first EEG signal refers to a feature of the first EEG signal that is related to space. Optionally, the first image feature of the first EEG signal may include: a 2D atlas of the first EEG signal, and / or a 3D densely connected network.

[0193] Optionally, the specified time domain features and / or brain functional connectivity features of the first EEG signal can be visualized based on graph theory or the international 10-20 system, and then spatial information analysis can be performed based on the Common Spatial Pattern (CSP) to construct a 2D atlas of the first EEG signal. The specific method can be implemented through relevant technologies and will not be detailed here.

[0194] Optionally, a 3D densely connected network can be constructed based on the time domain features, frequency domain features, and / or time-frequency domain features of the first EEG signal. The specific construction method can be completed by related technologies and will not be described in detail here.

[0195] Optionally, the second feature of the first near-infrared brain function imaging signal may include: a second time sequence feature and a second image feature of the first near-infrared brain function imaging signal.

[0196] The second time series feature of the first near-infrared brain function imaging signal refers to a time-related feature of the first near-infrared brain function imaging signal. Optionally, the second time series feature of the first near-infrared brain function imaging signal may include: the concentration of oxygenated and deoxygenated hemoglobin, the beta index of a single channel, and / or the Pearson correlation coefficient between channels, etc.

[0197] Optionally, the concentrations of oxygenated and deoxygenated hemoglobin of the above-mentioned first near-infrared brain functional imaging signal can be calculated based on the modified Beer-Lambert law, the beta index of a single channel of the above-mentioned first near-infrared brain functional imaging signal can be determined using an activation degree analysis method based on a general linear model (GLM), and the Pearson correlation coefficient between the channels of the above-mentioned first near-infrared brain functional imaging signal can be calculated using a preset brain network.

[0198] Optionally, the second image feature of the first near-infrared brain functional imaging signal may include: a 3D channel activation map, a correlation matrix formed between channels, and / or a brain network connection map.

[0199] Optionally, a 3D channel activation map of the first near-infrared brain functional imaging signal can be determined based on GLM, and a brain network connection map of the first near-infrared brain functional imaging signal can be calculated using methods such as seed correlation analysis, complex brain network connection and / or small-world characteristics.

[0200] The following are illustrative examples of the seed correlation analysis method, the complex brain network connection analysis method, and the small-world feature analysis method.

[0201] Seed correlation analysis: This involves calculating the correlation between a selected brain region of interest and other brain regions based on relevant theories, using this as the strength of functional connectivity. During this analysis, a generalized linear model (GLM) is used to estimate the activation of each channel. Significantly activated channels are identified through a t-test, and these are considered seed points. Correlation analysis is then performed between the seed points and the time series of other channels across the brain. Statistical methods are then used to determine which channels are most closely associated with the seed points, thereby generating a functional connectivity map.

[0202] Complex brain networks: Complex brain networks are an analytical method used to describe the interconnected network structure between multiple brain regions. First, different brain regions are defined as nodes in the network, which can represent specific brain regions or functions. Then, the connection strength between these nodes is calculated, typically using the correlation of time series data to measure the association between nodes. Because the correlation matrix is ​​often very dense, a threshold is set to remove weaker connections to simplify the network structure. Complex brain network analysis helps reveal information transfer, collaborative activities, and functional integration between different brain regions.

[0203] Small-world properties: A hallmark of complex brain networks, small-world properties indicate efficient global information transfer and local interconnectedness. Specifically, small-world networks have a short average shortest path length, indicating efficient information transfer between nodes. Furthermore, small-world networks maintain a high clustering coefficient, reflecting the close connections between nodes. This structure, similar to a "small world," allows for rapid information transmission to distant nodes while maintaining strong local connections, contributing to efficient information processing and functional integration in the brain.

[0204] EEG signals can only analyze information on the cerebral cortex, and there are certain limitations in exploring the activation of specific brain areas based on the distribution of functional connections and energy topography. The embodiment of the present application combines the first near-infrared brain functional imaging signal with the first EEG signal to identify the user's workload, and the first near-infrared brain functional imaging signal can provide more and more accurate spatial information for the activation between brain regions. Based on this information, the workload state processing method based on multimodal data in the embodiment of the present application can make more accurate judgments on the brain areas activated by users, such as pilots, in certain task states.

[0205] The following is an exemplary description of possible implementations of the above step 302.

[0206] Optionally, when the first feature of the first EEG signal includes a first time series feature and a first image feature, and the second feature of the first near-infrared brain function imaging signal includes a second time series feature and a second image feature of the first near-infrared brain function imaging signal, this step may include:

[0207] Fusing the first time series feature and the second time series feature to obtain a third time series feature;

[0208] Fusing the first image feature and the second image feature to obtain a third image feature;

[0209] The third time series feature and the third image feature are fused to obtain a first fused feature.

[0210] Optionally, fusing the first time series feature and the second time series feature to obtain a third time series feature may include:

[0211] Converting the first temporal feature into a fourth image feature;

[0212] converting the second time series feature into a fifth image feature;

[0213] The fourth image feature and the fifth image feature are fused to obtain a sixth image feature, and the sixth image feature is used as the third time series feature.

[0214] The conversion of the first time series feature into the fourth image feature may be achieved by a related conversion method from time series feature to image feature, which is not limited in the embodiment of the present application.

[0215] Optionally, fusing the first image feature and the second image feature to obtain the third image feature may include:

[0216] The pixel values ​​of the pixels corresponding to the first image feature and the second image feature are added or multiplied to obtain the pixel values ​​of the pixels corresponding to the third image feature; or the first image feature and the second image feature are concatenated to obtain the third image feature.

[0217] Optionally, fusing the fourth image feature and the fifth image feature to obtain the sixth image feature may include:

[0218] The pixel values ​​of the pixels corresponding to the fourth image feature and the fifth image feature are added or multiplied to obtain the pixel value of the pixel corresponding to the sixth image feature; or the fourth image feature and the fifth image feature are concatenated to obtain the sixth image feature.

[0219] Optionally, when the third time series feature is realized by the above-mentioned sixth image feature, the above-mentioned fusion processing of the third time series feature and the third image feature to obtain the first fusion feature may include: adding or multiplying the pixel values ​​of the pixels corresponding to the sixth image feature and the third image feature as the pixel values ​​of the pixels corresponding to the first fusion feature; or, splicing the sixth image feature and the third image feature to obtain the seventh image feature as the first fusion feature.

[0220] The following is an exemplary description of possible implementations of the above step 303.

[0221] Optionally, the workload information may include: workload level and / or workload type.

[0222] The workload level is used to represent the level of user workload and may include at least two levels. In one example, the workload level may be divided into three levels: high, medium, and low.

[0223] The workload type is used to characterize the type of the user's workload, and may include, for example, visual, auditory, and / or attention types, etc. Optionally, the attention type may include fatigue, distraction, concentration, etc.

[0224] Taking workload information including workload level and workload type as an example, the workload identification model can output the workload type with relatively high workload at each workload level when outputting the workload level, or it can also output the workload type with relatively high workload at the workload level only when outputting a workload level of a specified level.

[0225] In one instance, assuming that the workload level includes three levels: high, medium, and low, and the workload types include: visual, auditory, and attention types, the workload recognition model outputs the workload level and workload type. For example, if the output is (low, auditory), it can indicate that the workload level is low, and the load of the auditory workload type is relatively high, or it can output (medium, visual), which can indicate that the workload level is medium, and the load of the visual workload type at the medium level is relatively high.

[0226] In another example, assuming that the workload level includes three levels: high, medium, and low, and the workload types include: visual, auditory, and attention types, then when the workload recognition model recognizes that the workload level is low, it can only output the workload level. When it recognizes that the workload level is medium or high, it outputs the workload level and workload type, for example (medium, visual), thereby indicating that the workload level is medium, and the load of the visual workload type at the medium level is relatively high.

[0227] It is understandable that when the workload identification model outputs the workload level and workload type, the number of output workload types may be one or more, which is not limited in the embodiment of the present application.

[0228] Step 203: Feedback is sent to the first management account based on the workload information, and a training plan matching the workload information is formulated.

[0229] Optionally, in this step, the workload information may be directly sent to the first management account, or a report in a preset format may be generated according to the workload information and then sent to the first management account.

[0230] Optionally, in some embodiments, training plans corresponding to different workload information may be preset. In this step, the training plan corresponding to the workload information may be directly obtained, thereby achieving the above-mentioned formulation of a training plan that matches the workload information.

[0231] In other embodiments, if the workload information includes workload levels and workload types, training plans corresponding to different workload levels and workload types can be preset. When developing a training plan that matches the workload information in this step, the training plan corresponding to the workload level and the training plan corresponding to the workload type in the workload information can be obtained, and the two training plans can be combined to obtain a training plan that matches the workload information. The specific method for combining the two training plans can be implemented using relevant methods and is not limited by the embodiments of this application.

[0232] By formulating a training plan that matches the workload information of the first user, the training plan of the first user can be made to use the workload requirements of the first user, thereby improving the training efficiency and safety of the first user.

[0233] In the method shown in Figure 2, the workload information of the first user is determined based on the user's multimodal status information, thereby realizing the identification of the user's workload; moreover, the workload information is fed back to the first management account and a training plan matching the workload information is formulated, so that a training plan can be formulated to adapt to the workload information of the first user, thereby ensuring the scientificity and safety of the first user's training.

[0234] FIG4A is another flowchart of a method for identifying workload information based on multimodal data provided in an embodiment of the present application. As shown in FIG4A , the method may further include the following step 204 after step 202 shown in FIG2 .

[0235] Step 204: Perform early warning processing of the user's workload based on the user's workload information.

[0236] The execution order between step 204 and step 203 is not limited in this embodiment of the present application.

[0237] Optionally, this step may specifically include:

[0238] When the workload level in the workload information is not lower than a preset level and the workload type includes a first type, an alarm is issued for the first type of workload type; and / or,

[0239] When the workload level in the workload information is not lower than the preset level and the workload type includes the second type, the proportion of the second type in the multiple workload information obtained within the preset first time period is determined, and when the proportion is not lower than the preset proportion threshold, an alarm is issued for the second type of workload type.

[0240] For example, the above-mentioned preset level may be medium, and the first type may be distraction. When the workload level in the workload information is medium or high, and the workload type is distraction, an early warning is issued for the above-mentioned distraction workload type.

[0241] For another example, the above-mentioned preset level is intermediate, the second type is fatigue, the first duration is 5s, and the preset proportion threshold is 70%. Then, when multiple workload information of the user is obtained within 5s, if the workload level in more than 70% of the workload information is higher than intermediate and the workload type includes fatigue, an early warning will be issued for the workload type of fatigue.

[0242] It is understandable that the specific warning method mentioned above can be text, sound, and / or light, etc., and is not limited in the embodiment of the present application.

[0243] In the embodiment of the present application, early warning processing can be performed based on different workload information, thereby improving the user's work efficiency and ensuring the user's work safety.

[0244] In another embodiment of the workload information identification method based on multimodal data provided in the present application, a preprocessing step may be included between steps 201 and 202 in the above embodiment. As shown in Figure 4B , the preprocessing step shown in step 401 is performed between steps 201 and 202 in the method shown in Figure 4A as an example.

[0245] Step 401: Preprocess the status information of the first user.

[0246] Accordingly, in step 202 , the workload information of the first user may be determined according to the pre-processed state information.

[0247] The above-mentioned preprocessing may specifically include: noise reduction processing, physiological violation removal, and / or outlier detection, etc.

[0248] Optionally, if the state information of the first user includes a first EEG signal and a first near-infrared brain function imaging signal, this step may specifically include: preprocessing the first EEG signal and the first near-infrared brain function imaging signal respectively.

[0249] Correspondingly, in step 301 , feature extraction is performed on the preprocessed first electroencephalogram signal and the first near-infrared functional brain imaging signal.

[0250] The above-mentioned preprocessing may specifically include: noise reduction processing, physiological violation removal, and / or outlier detection, etc.

[0251] The above-mentioned preprocessing can be specifically achieved through end-to-end denoising and artifact removal algorithms such as frequency domain filtering, ICA analysis, wavelet denoising, motion artifact detection, bandpass filtering, bad segment and bad conductor detection, interpolation of bad conductors, steady-state evoked response (SSR) short separation regression, and / or convolutional neural networks.

[0252] EEG data is greatly affected by noise and artifacts. Through the above preprocessing, the first EEG signal and the first near-infrared brain functional imaging signal obtained after processing can be made more accurate, thereby making the subsequent identification of the user's workload results more accurate.

[0253] The following is an exemplary description of the training method of the workload identification model in the above embodiment with reference to FIG5 . As shown in FIG5 , the training method of the workload identification model may include:

[0254] Step 501: Acquire a second EEG signal, a second near-infrared functional brain imaging signal, and a workload information tag of a second user.

[0255] The second user can be any user, rather than a specific user. The first user can also be the second user in this step.

[0256] Optionally, the workload information tag is used to record the workload information of the second user when collecting the second EEG signal and the second near-infrared brain functional imaging signal of the second user.

[0257] Step 502: extracting a first feature of the second EEG signal and a second feature of the second near-infrared functional brain imaging signal respectively.

[0258] The implementation of this step can refer to step 301, the main difference is that the first EEG signal is replaced by the second EEG signal in this step, and the first near-infrared brain function imaging signal is replaced by the second near-infrared brain function imaging signal in this step.

[0259] Optionally, steps 501 and 502 may also include steps of preprocessing the second EEG signal and the second near-infrared brain function imaging signal respectively. The implementation of the preprocessing step can refer to step 401. The main difference is that the first EEG signal is replaced by the second EEG signal, and the first near-infrared brain function imaging signal is replaced by the second near-infrared brain function imaging signal. It will not be repeated here.

[0260] Step 503: Fusing the first feature of the second EEG signal and the second feature of the second near-infrared functional brain imaging signal to obtain a second fused feature.

[0261] The implementation of this step can refer to step 302, the main difference is that the first feature of the first EEG signal is replaced by the first feature of the second EEG signal in this step, and the second feature of the first near-infrared brain functional imaging signal is replaced by the second feature of the second near-infrared brain functional imaging signal in this step.

[0262] Step 504: Use the second fusion feature and the workload information label as samples to train the workload recognition model.

[0263] The workload identification model in this step is an untrained workload identification model.

[0264] Optionally, the workload identification model may be implemented by a neural network. Specifically, the workload identification model may include multiple network layers. In this step, using the second fusion feature and the workload information label as samples to train the workload identification model may specifically include:

[0265] The second fused feature is input into the workload identification model to obtain the output result of each network layer in the forward propagation process;

[0266] Determine the target results for each network layer based on workload information tags;

[0267] For each network layer, the similarity between the output result of the network layer and the target result is calculated, and the weight of the network layer is adjusted according to the similarity.

[0268] Among them, the above-mentioned step of determining the target result of each network layer according to the workload information label can be implemented using relevant technologies, which will not be repeated here.

[0269] Among them, the above steps of calculating the similarity between the output result of the network layer and the target result and adjusting the weight of the network layer according to the similarity can be implemented using relevant technologies, which will not be repeated here.

[0270] Step 505: Determine whether the workload recognition model has been trained. If so, stop training. Otherwise, return to step 501 and continue to obtain the second EEG signal, second near-infrared brain functional imaging signal and workload information label of the second user or other users to train the workload recognition model.

[0271] In this step, a condition for workload identification model training completion can be preset. The setting of this condition can be implemented with reference to related technologies. The implementation of determining whether the workload identification model training is complete based on this condition in this step can also be implemented with reference to related technologies, and is not limited in the present embodiment. It is understood that when it is determined that the workload identification model training is complete, the trained workload identification model can be obtained and used as the model for identifying the user's workload information in step 303.

[0272] The workload status processing method based on multimodal data provided in the embodiment of the present application can also be executed by an electronic device having the structure shown in Figure 1B. The workload identification model training method provided in the embodiment of the present application can be executed by an electronic device having the structure shown in Figure 1B. It can be understood that the electronic device that executes the workload status processing method based on multimodal data and the electronic device that executes the workload identification model training method can be the same electronic device or different electronic devices.

[0273] The following is an example of the workload information identification method based on multimodal data according to an embodiment of the present application, using FIG6 . As shown in FIG6 , the method includes:

[0274] Collecting a user's first electroencephalogram (EEG) signal and a first near-infrared functional brain imaging (fNIRS) signal;

[0275] Preprocessing the first EEG signal; extracting features from the preprocessed first EEG signal to obtain a first time series feature and a first image feature of the first EEG signal;

[0276] Preprocessing the first near-infrared functional brain imaging signal fNIRS; extracting features from the preprocessed first near-infrared functional brain imaging signal fNIRS to obtain a second time series feature and a second image feature of the first near-infrared functional brain imaging signal fNIRS;

[0277] fusing the first time series feature of the first electroencephalogram (EEG) signal and the second time series feature of the first near-infrared functional brain imaging (fNIRS) signal to obtain a third time series feature;

[0278] fusing the first image feature of the first electroencephalogram (EEG) signal and the second image feature of the first near-infrared functional brain imaging signal (fNIRS) to obtain a third image feature;

[0279] Fusing the third time series feature and the third image feature to obtain a first fused feature;

[0280] The first fusion feature is input into a preset workload recognition model to obtain the user's workload information.

[0281] In the following embodiments, a pilot workload identification method based on human intelligence is provided.

[0282] It is understandable that the pilot in the following embodiments can also be extended to users in other work scenarios, such as drivers, etc.

[0283] In some embodiments, the following pilot workload identification method based on human intelligence can be used as a possible implementation of the above-mentioned workload status processing method based on multimodal data.

[0284] Human intelligence combines human factors engineering with artificial intelligence (AI) technology. Human factors engineering applies human psychological and physiological principles to the engineering and design of products, processes, and systems. In other words, it is the technology of designing and improving the human-machine-environment system according to human characteristics. Combining this technology with artificial intelligence can enable machines to interact with humans more intelligently.

[0285] The pilot workload identification method based on human factors intelligence provided in the embodiment of the present application utilizes the above-mentioned characteristics of EEG data to identify the user's workload based on the user's EEG data, thereby improving the safety of the user's work.

[0286] FIG7 is a flow chart of a pilot workload identification method based on human intelligence according to an embodiment of the present application. The method can be executed by the electronic device shown in FIG1A and FIG1B .

[0287] As shown in FIG7 , the method may include:

[0288] Step 701: Collect a first EEG signal and a first near-infrared functional brain imaging signal of a first user.

[0289] In one instance, if the method shown in Figure 7 is applied to the system shown in Figure 1A, the electronic device can control the EEG signal acquisition device to collect the first EEG signal of the first user, and control the near-infrared brain function imaging device to collect the first near-infrared brain function imaging signal of the first user.

[0290] In another example, if the method shown in FIG7 is applied to the electronic device shown in FIG1B, the electronic device can control the EEG signal acquisition component to acquire the first EEG signal of the first user, and control the near-infrared brain function imaging component to acquire the first near-infrared brain function imaging signal of the first user.

[0291] Optionally, the first EEG signal may be a whole-brain EEG signal, which refers to an EEG signal obtained by detecting the entire brain.

[0292] Near-infrared brain function imaging is a latest generation brain function detection technology based on optical principles. The first near-infrared brain function imaging signal in this step is a signal obtained using near-infrared brain function imaging technology.

[0293] The first EEG signal collected in this step has a high temporal resolution, and the first near-infrared brain functional imaging signal has a high spatial resolution. The former is electrical activity, and the latter is hemodynamic changes. The two are complementary to a certain extent. The above two signals are collected in the following steps for joint analysis in subsequent steps, which helps to improve the accuracy of user workload detection.

[0294] Step 702: extracting a first time series feature and a first image feature of a first EEG signal.

[0295] The first time series feature of the first EEG signal refers to a time-related feature of the first EEG signal. Optionally, the first time series feature of the first EEG signal may include: a time domain feature, a frequency domain feature, a time-frequency domain feature, a nonlinear feature, and / or a brain functional connectivity feature of the first EEG signal. In some related technologies, the above features may also be referred to as indicators.

[0296] For the specific implementation of the time domain features, frequency domain features, time-frequency domain features, nonlinear features and / or brain functional connection features of the first EEG signal, please refer to the corresponding description in the aforementioned step 301, which will not be repeated here.

[0297] The first image feature of the first EEG signal refers to a feature of the first EEG signal that is related to space. Optionally, the first image feature of the first EEG signal may include: a brain topography map and / or a power spectrum topography map.

[0298] Optionally, the above-mentioned brain topography map and / or power spectrum topography map can be obtained using a feature extraction method of relevant brain function images, which is not limited in the embodiments of the present application.

[0299] Step 703: extracting the second time series features and the second image features of the first near-infrared brain functional imaging signal.

[0300] The second time series feature of the first near-infrared brain function imaging signal refers to a time-related feature of the first near-infrared brain function imaging signal. Optionally, the second time series feature of the first near-infrared brain function imaging signal may include: the concentration of oxygenated and deoxygenated hemoglobin, the beta index of a single channel, and / or the Pearson correlation coefficient between channels, etc.

[0301] Optionally, the concentrations of oxygenated and deoxygenated hemoglobin of the above-mentioned first near-infrared brain functional imaging signal can be calculated based on the modified Beer-Lambert law, the beta index of a single channel of the above-mentioned first near-infrared brain functional imaging signal can be determined using an activation degree analysis method based on a general linear model (GLM), and the Pearson correlation coefficient between the channels of the above-mentioned first near-infrared brain functional imaging signal can be calculated using a preset brain network.

[0302] In some embodiments, in this step, brain network maps with different functions can be extracted from the first near-infrared brain functional imaging signal as the second image feature.

[0303] The human brain contains multiple different brain networks, such as the dorsal attention network (DAN), the default mode network (DMN), the sensorimotor network (SMN), the DAN-SMN, the DAN-VAN, the ventral attention network (VAN), the auditory attention network (AAN), the prefrontal cortex network (FPN), and the social network (SN). The following provides examples of each.

[0304] DAN: Top-down attention control that helps the brain direct attention externally, such as motor control or visual search.

[0305] DMN: Default mode network, related to introspection, self-cognition, individual cognition and social cognition, is active in the resting state and weakened when performing tasks.

[0306] SMN: sensorimotor network, which processes information from sensory organs and controls muscle movements, enabling the body to perceive and respond to the environment.

[0307] DAN-SMN: Predicts performance during multitasking.

[0308] DAN-VAN: Spatial recognition,spatial attention allocation.

[0309] VAN: The ventral attention network plays an important role in allocating and controlling attention in the external environment, handling goal-oriented tasks, inhibiting interference, and processing unexpected events. It is a bottom-up attention network.

[0310] AAN: The auditory attention network plays an important role in processing auditory information and allocating attention to auditory stimuli.

[0311] FPN: The prefrontal network, including the prefrontal cortex and parietal lobe, is associated with higher cognitive functions and decision-making.

[0312] SN: Social networks involve social interactions, emotional processing, and the perception of social emotions. They play a key role in understanding and processing social information, emotional expressions, and social emotions.

[0313] In this step, the image features of the above-mentioned brain network (i.e., the brain network map) can be extracted from the first near-infrared brain functional imaging signal, so that in subsequent steps, the user's workload type can be obtained according to the brain network and the functional connection analysis between brain networks, thereby improving the detection accuracy of the workload type.

[0314] Optionally, the second image feature of the first near-infrared brain functional imaging signal may specifically include: a 3D channel activation map, a complex brain network, a default brain network, a correlation heat map, and / or a brain network connection map, etc.

[0315] Optionally, a 3D channel activation map of the first near-infrared brain functional imaging signal can be determined based on GLM, and a brain network connection map of the first near-infrared brain functional imaging signal can be calculated using methods such as seed correlation analysis, complex brain network connection and / or small-world characteristics.

[0316] The execution order between step 702 and step 703 is not limited in this embodiment of the application.

[0317] Step 704: Fusing the first time series feature and the second time series feature to obtain a third time series feature.

[0318] Optionally, the fusion processing in this step can be implemented using a related time series feature fusion method, such as a time series feature weighted fusion method, etc., which is not limited in the embodiments of the present application.

[0319] Step 705: Identify the workload level of the first user based on the third time series feature.

[0320] The workload level is used to characterize the workload of a user and is a measure of the overall workload of the user. The workload level may include at least two levels. In one example, the workload level may be divided into three levels: high, medium, and low.

[0321] In some embodiments, a load level recognition model can be pre-trained. The model's inputs can be user time series features and workload levels. The model's training method is described in FIG7 and is not further described here. This step can specifically include inputting the third time series features into the load level recognition model and using the workload level output by the load level recognition model as the first user's workload level.

[0322] Step 706: Fusing the first image feature and the second image feature to obtain a third image feature.

[0323] Optionally, the fusion processing in this step can be implemented using relevant fusion methods, such as pixel-level fusion methods, feature-level fusion methods (for example, weighted averaging method, Bayesian estimation method, cluster analysis method, etc.) or decision-level fusion methods, etc., which are not limited in the embodiments of the present application.

[0324] Step 707: Identify the workload type of the first user according to the third image feature.

[0325] The workload type is used to characterize the type of the user's workload, and may include, for example, vision, hearing, attention, execution, and / or planning.

[0326] In some scenarios, workers' work is more complicated. Take pilots flying airplanes or conducting flight simulations as an example. Flying an airplane is essentially a complex continuous tracking task that requires a bottom-up process to perform multi-sensory integration of information from the external environment, and requires top-down regulatory influences, strategies, and current intentions based on the pilot's internal goals. Therefore, in complex work scenarios such as pilots flying airplanes or conducting flight simulations, predicting the pilot's workload is of great significance and value.

[0327] In an embodiment of the present application, the workload type of a user at work (for example, a pilot flying an airplane or performing a flight simulation) can be predicted based on the relationship between the different networks in the above-mentioned brain and different workload types such as vision, hearing, attention, execution, and / or planning.

[0328] In some embodiments, a load type recognition model can be pre-trained. The model's inputs can be user image features and workload types. The model's training method is described in FIG6 and is not further described here. This step can specifically include inputting the third image feature into the load type recognition model and using the workload type output by the load type recognition model as the first user's workload type.

[0329] It is understandable that when the load type identification model outputs the workload type, the number of output workload types may be one or more, which is not limited in the embodiment of the present application.

[0330] As shown in FIG. 7 , there is no restriction on the execution order of steps 704 to 705 and steps 706 to 707 .

[0331] EEG signals can be used to analyze information about the cerebral cortex. However, there are certain limitations in exploring the activation of specific brain areas based on the distribution of functional connections and energy topography. In the embodiment of the present application, based on the first EEG signal and combined with the first near-infrared brain functional imaging signal, the user's workload type is identified. The first near-infrared brain functional imaging signal can provide more and more accurate spatial information for the activation of brain regions and the activation between brain regions. Based on this information, the pilot workload identification method based on human intelligence in the embodiment of the present application can make more accurate judgments on the brain regions activated by users, such as pilots, in certain task states, and thus can more accurately judge the workload type of users, such as pilots.

[0332] In the method shown in Figure 7, the first time series feature and the first image feature of the first EEG signal of the first user are extracted respectively, and the second time series feature and the second image feature of the first near-infrared brain functional imaging signal are extracted. The workload level of the first user is identified based on the third time series feature obtained by fusing the first time series feature and the second time series feature. The workload type of the first user is identified based on the third image feature obtained by fusing the first image feature and the second image feature, thereby realizing the identification of the user workload.

[0333] In some embodiments, before extracting the first time series feature and the first image feature in step 702 , the first EEG signal may be preprocessed.

[0334] Accordingly, in step 702 , the first time series feature and the first image feature may be extracted from the preprocessed first EEG signal.

[0335] In some embodiments, before extracting the second time series features and the second image features in step 703 , the first near-infrared brain functional imaging signal may be preprocessed.

[0336] Accordingly, in step 703 , the second time series feature and the second image feature may be extracted from the preprocessed first near-infrared brain functional imaging signal.

[0337] For example, FIG8 takes the method shown in FIG7 as an example of adding step 801 before step 702 and step 703.

[0338] Step 801: Preprocessing a first EEG signal and a first near-infrared brain functional imaging signal.

[0339] The above-mentioned preprocessing may specifically include: noise reduction processing, physiological violation removal, and / or outlier detection, etc.

[0340] The above-mentioned preprocessing can be specifically achieved through end-to-end denoising and artifact removal algorithms such as frequency domain filtering, ICA analysis, wavelet denoising, motion artifact detection, bandpass filtering, bad segment and bad conductor detection, interpolation of bad conductors, steady-state evoked response (SSR) short separation regression, and / or convolutional neural networks.

[0341] EEG data is greatly affected by noise and artifacts. Through the above preprocessing, the first EEG signal and the first near-infrared brain functional imaging signal obtained after processing can be made more accurate, thereby making the subsequent identification of the user's workload results more accurate.

[0342] In some embodiments, an early warning processing step may be included after step 705 and step 707 of the above embodiment.

[0343] FIG9 is another flowchart of a pilot workload identification method based on human intelligence provided in an embodiment of the present application, taking the following step 901 as an example, which is added after step 705 and step 707 shown in FIG8 .

[0344] Step 901: Perform early warning processing of the user's workload according to the user's workload type and workload level.

[0345] Optionally, this step may specifically include:

[0346] When the workload level is not lower than a preset level and the workload type includes the first type, an alarm is issued for the first type of workload type; and / or,

[0347] When the workload level is not lower than the preset level and the workload type includes the second type, the proportion of the second type in the multiple workload types obtained within the preset first time period is obtained, and when the proportion is not lower than the preset proportion threshold, an alarm is issued for the second type of workload type.

[0348] For example, the preset level may be medium, and the first type may be visual. When the workload level is medium or high, and the workload type is visual, an early warning is issued for the visual workload type.

[0349] For another example, the above preset level is intermediate, the second type is visual, the first duration is 5s, and the preset proportion threshold is 70%. Then, when multiple workload types of the user are obtained within 5s, if the workload levels of more than 70% of the workload types are higher than intermediate and the workload types include vision, an early warning will be issued for the visual workload type.

[0350] It is understandable that the specific warning method mentioned above can be text, sound, and / or light, etc., and is not limited in the embodiment of the present application.

[0351] The embodiments of the present application can be applied to scenarios where a pilot is flying an airplane or simulated flight, and can provide an early warning when the pilot has a high workload of a certain type, so as to ensure the pilot's work safety.

[0352] In the embodiment of the present application, early warning processing can be performed based on different workload types and workload levels, thereby improving the user's work efficiency and ensuring the user's work safety.

[0353] In some embodiments, the step of formulating a training plan may be further included after step 705 and step 707 of the above embodiment.

[0354] FIG10 is another flowchart of a pilot workload identification method based on human intelligence provided in an embodiment of the present application. As shown in FIG10 , the method takes the example of adding step 1001 after step 705 and step 707 shown in FIG8 .

[0355] Step 1001: Develop a training plan for a first user based on workload type and workload level.

[0356] Optionally, in some embodiments, training plans corresponding to different combinations of workload types and workload levels can be preset. In this step, the training plans corresponding to the workload types and workload levels can be directly obtained, thereby formulating a training plan that matches the workload types and workload levels.

[0357] In other embodiments, training plans corresponding to different workload levels and different workload types can be preset. In this step, the training plan corresponding to the workload level and the training plan corresponding to the workload type of the first user can be obtained, and the two training plans can be combined to obtain a training plan that matches the workload type and workload level. The specific method of combining the above two training plans can be implemented using relevant methods and is not limited by the embodiments of this application.

[0358] By developing a training plan that matches the workload type and workload level of the first user, the first user's training plan can be adapted to the first user's workload requirements, thereby improving the first user's training efficiency and safety. For example, embodiments of the present application can be used in simulated flight scenarios for pilots, thereby enabling training plans to be developed based on the pilot's workload level and type during actual flight or simulated flight, thereby improving the pilot's training effectiveness and safety.

[0359] In this method, a training plan for the first user is formulated according to the workload type and workload level, so that the training plan can be adapted to the workload type and workload level of the first user, ensuring the scientificity and safety of the first user's training.

[0360] In some embodiments, other workload-based processing may be performed based on the user's workload type and workload level, such as cockpit optimization, equipment evaluation, etc., which are not listed one by one in the embodiments of this application.

[0361] The following is an exemplary description of the training method of the load type identification model in the above embodiment with reference to FIG11. As shown in FIG11, the training method of the load type identification model may include:

[0362] Step 1101: Acquire a third EEG signal, a third near-infrared functional brain imaging signal, and a workload type label of a third user.

[0363] The third user can be any user, not a specific user. The first user and the second user can also be the third user in this step.

[0364] Optionally, the workload type tag is used to record the workload type of the third user when collecting the third EEG signal and the third near-infrared brain functional imaging signal of the third user.

[0365] Step 1102: extracting the fourth image feature of the third EEG signal and the fifth image feature of the third near-infrared functional brain imaging signal respectively.

[0366] The implementation of this step can refer to step 702 and step 703, the main difference is that the first EEG signal is replaced by the third EEG signal in this step, and the first near-infrared brain function imaging signal is replaced by the third near-infrared brain function imaging signal in this step.

[0367] Optionally, a step of preprocessing the third EEG signal and / or the third near-infrared brain function imaging signal may also be included between step 1101 and step 1102. The implementation of the preprocessing step can refer to the corresponding description of the preprocessing step mentioned above. The main difference is that the first EEG signal is replaced by the third EEG signal, and the first near-infrared brain function imaging signal is replaced by the third near-infrared brain function imaging signal, which will not be repeated here.

[0368] Step 1103: fusing the fourth image feature of the third EEG signal and the fifth image feature of the third near-infrared functional brain imaging signal to obtain a sixth image feature.

[0369] The implementation of this step can refer to step 706. The main difference is that the first image feature of the first EEG signal is replaced by the fourth image feature of the third EEG signal in this step, and the second image feature of the first near-infrared brain functional imaging signal is replaced by the fifth image feature of the third near-infrared brain functional imaging signal in this step.

[0370] Step 1104: Use the sixth image feature and the workload type label as samples to train the workload type recognition model.

[0371] The load type identification model in this step is an untrained load type identification model.

[0372] Optionally, the load type recognition model may be implemented by a neural network. Specifically, the load type recognition model may include multiple network layers. In this step, using the sixth image feature and the workload type label as samples to train the load type recognition model may specifically include:

[0373] Inputting the sixth image feature into the load type recognition model to obtain the output result of each network layer in the forward propagation process;

[0374] Determine the target outcome for each network layer based on the workload type label;

[0375] For each network layer, the similarity between the output result of the network layer and the target result is calculated, and the weight of the network layer is adjusted according to the similarity.

[0376] Among them, the above step of determining the target result of each network layer according to the workload type label can be implemented using relevant technologies, which will not be repeated here.

[0377] Among them, the above steps of calculating the similarity between the output result of the network layer and the target result and adjusting the weight of the network layer according to the similarity can be implemented using relevant technologies, which will not be repeated here.

[0378] Step 1105: Determine whether the load type identification model has been trained. If so, stop training. Otherwise, return to step 1101 and continue to obtain the third EEG signal, third near-infrared brain functional imaging signal and workload type label of the third user or other users to train the load type identification model.

[0379] In this step, a condition for completing the training of the load type identification model can be preset. The setting of this condition can be implemented with reference to related technologies. The implementation of determining whether the training of the load type identification model is complete based on this condition in this step can also be implemented with reference to related technologies, and is not limited in the embodiments of this application. It is understood that when it is determined that the training of the load type identification model is complete, the trained load type identification model can be obtained and used as the model for identifying the user's workload type in step 707.

[0380] The pilot workload identification method based on human intelligence provided in the embodiments of the present application can be executed by an electronic device having the structure shown in FIG1B . The load type identification model training method provided in the embodiments of the present application can be executed by an electronic device having the structure shown in FIG1B . It is understood that the electronic device that executes the pilot workload identification method based on human intelligence and the electronic device that executes the load type identification model training method can be the same electronic device or different electronic devices.

[0381] The following is an exemplary description of the training method of the load level identification model in the above embodiment with reference to FIG12. As shown in FIG12, the training method of the load level identification model may include:

[0382] Step 1201: Acquire a third EEG signal, a third near-infrared functional brain imaging signal, and a workload level label of a third user.

[0383] The third user can be any user, not a specific user. The first user and the second user can also be the third user in this step.

[0384] Optionally, the workload level tag is used to record the workload level of the third user when collecting the third EEG signal and the third near-infrared brain functional imaging signal of the third user.

[0385] Step 1202: extracting the fourth time series feature of the third EEG signal and the fifth time series feature of the third near-infrared functional brain imaging signal respectively.

[0386] The implementation of this step can refer to step 702 and step 703, the main difference is that the first EEG signal is replaced by the third EEG signal in this step, and the first near-infrared brain function imaging signal is replaced by the third near-infrared brain function imaging signal in this step.

[0387] Optionally, a step of preprocessing the third EEG signal and / or the third near-infrared brain function imaging signal may also be included between step 1201 and step 1202. The implementation of the preprocessing step can refer to the corresponding description of the preprocessing step mentioned above. The main difference is that the first EEG signal is replaced by the third EEG signal, and the first near-infrared brain function imaging signal is replaced by the third near-infrared brain function imaging signal, which will not be repeated here.

[0388] Step 1203: fusing the fourth time series feature of the third EEG signal and the fifth time series feature of the third near-infrared functional brain imaging signal to obtain a sixth time series feature.

[0389] The implementation of this step can refer to step 704. The main difference is that the first timing feature of the first EEG signal is replaced by the fourth timing feature of the third EEG signal in this step, and the second timing feature of the first near-infrared brain function imaging signal is replaced by the fifth timing feature of the third near-infrared brain function imaging signal in this step.

[0390] Step 1204: Use the sixth time series feature and the workload level label as samples to train the workload level recognition model.

[0391] The load level recognition model in this step is an untrained load level recognition model.

[0392] Optionally, the load level recognition model may be implemented by a neural network. Specifically, the load level recognition model may include multiple network layers. In this step, the sixth time series feature and the workload level label are used as samples to train the load level recognition model, which may specifically include:

[0393] Input the sixth time series feature into the load level recognition model to obtain the output results of each network layer in the forward propagation process;

[0394] Determine target outcomes for each network layer based on workload level labels;

[0395] For each network layer, the similarity between the output result of the network layer and the target result is calculated, and the weight of the network layer is adjusted according to the similarity.

[0396] The above-mentioned step of determining the target result of each network layer according to the workload level label can be implemented using relevant technologies, which will not be described in detail here.

[0397] Among them, the above steps of calculating the similarity between the output result of the network layer and the target result and adjusting the weight of the network layer according to the similarity can be implemented using relevant technologies, which will not be repeated here.

[0398] Step 1205: Determine whether the load level recognition model has been trained. If so, stop training. Otherwise, return to step 1201 and continue to obtain the third EEG signal, third near-infrared brain functional imaging signal and workload level label of the third user or other users to train the load level recognition model.

[0399] In this step, a condition for completing the training of the load level identification model can be preset. The setting of this condition can be implemented with reference to related technologies. The implementation of determining whether the training of the load level identification model is complete based on this condition in this step can also be implemented with reference to related technologies, and is not limited in the embodiments of this application. It is understood that when it is determined that the training of the load level identification model is complete, the trained load level identification model can be obtained and used as the model for identifying the user's workload level in step 1205.

[0400] The pilot workload identification method based on human intelligence provided in the embodiments of the present application can be executed by an electronic device having the structure shown in FIG1B . The load level identification model training method provided in the embodiments of the present application can be executed by an electronic device having the structure shown in FIG1B . It is understood that the electronic device that executes the pilot workload identification method based on human intelligence and the electronic device that executes the load level identification model training method can be the same electronic device or different electronic devices.

[0401] The pilot workload identification method based on human intelligence according to an embodiment of the present application is illustrated below with reference to FIG13 . As shown in FIG6 , it includes:

[0402] Collecting a user's first electroencephalogram (EEG) signal and a first near-infrared functional brain imaging (fNIRS) signal;

[0403] Preprocessing the first EEG signal; extracting features from the preprocessed first EEG signal to obtain a first time series feature and a first image feature of the first EEG signal;

[0404] Preprocessing the first near-infrared functional brain imaging signal fNIRS; extracting features from the preprocessed first near-infrared functional brain imaging signal fNIRS to obtain a second time series feature and a second image feature of the first near-infrared functional brain imaging signal fNIRS;

[0405] A first time series feature of the first EEG signal and a second time series feature of the first fNIRS signal are fused to obtain a third time series feature; the third time series feature is input into a load level recognition model to obtain a user's workload level;

[0406] A first image feature of the first electroencephalogram (EEG) signal and a second image feature of the first near-infrared functional brain imaging signal (fNIRS) are fused to obtain a third image feature; the third image feature is input into a load type recognition model to obtain the user's workload type.

[0407] Afterwards, early warning processing, training plan designation, and other processing can be performed based on the user's workload level and workload type.

[0408] FIG14 is a schematic diagram of a structure of a workload information identification device based on multimodal data provided by an embodiment of the present application. As shown in FIG14 , the device 700 may include:

[0409] A collection module 710 is configured to collect status information of a first user;

[0410] a determination module 720 configured to determine workload information of the first user based on the collected state information, wherein the state information includes at least two of the following: physiological information, eye movement information, electroencephalogram information, brain functional imaging information, motion capture information, spatiotemporal acquisition information, behavioral acquisition information, facial expression, and state information;

[0411] The processing module 730 is configured to feed back the workload information to the first management account and formulate a training plan that matches the workload status level.

[0412] Optionally, when the state information of the first user collected by the collection module 710 includes: a first electroencephalogram signal and a first near-infrared brain functional imaging signal of the first user, as shown in FIG7 , the determination module 720 may include:

[0413] A feature extraction module 721 is used to extract a first feature of the first EEG signal and a second feature of the first near-infrared functional brain imaging signal;

[0414] An information determination module (not shown in FIG14 ) is configured to determine workload information of the first user based on the first feature and the second feature.

[0415] In some embodiments, as shown in FIG14 , the information determination module may include:

[0416] A fusion module 722 is configured to fuse the first feature and the second feature to obtain a first fused feature;

[0417] The identification module 723 is configured to input the first fusion feature into a preset workload identification model, and use the workload information output by the workload identification model as the user's workload information.

[0418] In other embodiments, the information determination module may include the first fusion module 940, the first identification module 950, the second fusion module 960, and the second identification module 970 in the following embodiments.

[0419] Optionally, as shown in FIG14 , the feature extraction module 721 may specifically include:

[0420] an EEG feature extraction module 7211, configured to extract a first feature of a first EEG signal;

[0421] The near-infrared feature extraction module 7212 is used to extract the second feature of the first near-infrared brain functional imaging signal.

[0422] Optionally, as shown in FIG15 , the apparatus 700 may further include a pre-processing module 740 , which may be disposed between the acquisition module 710 and the determination module 720 ;

[0423] The pre-processing module 750 is used to pre-process the status information of the first user.

[0424] Specifically, when the status information of the first user collected by the acquisition module 710 includes: the first EEG signal and the first near-infrared brain function imaging signal of the first user, the preprocessing module 750 can preprocess the first EEG signal and the first near-infrared brain function imaging signal respectively, and send the preprocessed first EEG signal and the first near-infrared brain function imaging signal to the feature extraction module 721.

[0425] Optionally, as shown in FIG15 , the apparatus 700 may further include an early warning module 750 ; the early warning module 750 is configured to perform early warning processing on the workload of the first user according to the workload information of the first user determined by the determination module 720 .

[0426] In one example, if the workload identification model is also trained by the workload identification device, the determination module 720 may further include a first training module 724. As shown in FIG16 , taking the workload identification device shown in FIG15 including the first training module 724 as an example, at this time,

[0427] The acquisition module 710 may also be used to: acquire a second EEG signal, a second near-infrared brain functional imaging signal, and a workload information tag of a second user;

[0428] The preprocessing module 740 is further configured to preprocess the second EEG signal and the second near-infrared brain function imaging signal, respectively, and input the preprocessed second EEG signal and the second near-infrared brain function imaging signal into the feature extraction module 720;

[0429] The feature extraction module 721 is further configured to extract a first feature of the second EEG signal and a second feature of the second near-infrared functional brain imaging signal respectively;

[0430] The fusion module 722 is further configured to: perform fusion processing on the first feature of the second EEG signal and the second feature of the second near-infrared brain functional imaging signal to obtain a second fusion feature;

[0431] The first training module 724 is used to train the workload recognition model using the second fusion feature and the workload information label as samples, and send the trained workload recognition model to the recognition module 723 .

[0432] It can be seen that in the workload identification device shown in Figure 16, the workload identification model can be first trained by the acquisition module 710, the preprocessing module 740, the feature extraction module 721, the fusion module 722 and the first training module 724 to obtain a trained workload identification model. The first training module 724 transmits the trained workload identification model to the identification module 723; thereafter, the acquisition module 710, the preprocessing module 740, the feature extraction module 721, the fusion module 722 and the identification module 723 realize the identification of user workload information.

[0433] FIG17 is a schematic diagram of a workload identification device according to an embodiment of the present application. As shown in FIG17 , the device 900 may include:

[0434] An acquisition module 910 is configured to acquire a first EEG signal and a first near-infrared functional brain imaging signal of a first user;

[0435] A first feature extraction module 920 is used to extract a first time series feature and a first image feature of the first EEG signal;

[0436] A second feature extraction module 930 is used to extract second time series features and second image features of the first near-infrared brain functional imaging signal;

[0437] A first fusion module 940 is configured to fuse the first time series feature and the second time series feature to obtain a third time series feature;

[0438] A first identification module 950 is configured to identify a workload level of the first user according to the third time series feature;

[0439] A second fusion module 960 is configured to fuse the first image feature and the second image feature to obtain a third image feature;

[0440] The second identification module 970 is configured to identify the workload type of the first user according to the third image feature.

[0441] In some embodiments, as shown in FIG10 , the apparatus 900 may further include: a second training module 980 , wherein:

[0442] The acquisition module 910 is further configured to: acquire a second EEG signal, a second near-infrared brain function imaging signal, and a workload level tag of a second user; the workload level tag is configured to record the workload level of the second user when the second EEG signal and the second near-infrared brain function imaging signal of the second user are acquired;

[0443] The first feature extraction module 920 is further configured to: extract a fourth time series feature of the second EEG signal;

[0444] The second feature extraction module 930 is further configured to: extract a fifth time series feature of the second near-infrared brain functional imaging signal;

[0445] The first fusion module 940 is further configured to: fuse the fourth time series feature of the second EEG signal and the fifth time series feature of the second near-infrared brain functional imaging signal to obtain a sixth time series feature;

[0446] The second training module 980 is used to train the load level recognition model using the sixth time series feature and the workload level label as samples.

[0447] Optionally, the second training module may send the trained load level recognition model to the first recognition module 950 , so that the first recognition module 950 may recognize the user's workload level according to the load level recognition model.

[0448] In some embodiments, as shown in FIG18 , the apparatus 900 may further include: a third training module 990 , wherein:

[0449] The acquisition module 910 is further configured to: acquire a second EEG signal, a second near-infrared brain function imaging signal, and a workload type tag of a second user; the workload type tag is configured to record the workload type of the second user when acquiring the second EEG signal and the second near-infrared brain function imaging signal of the second user;

[0450] The first feature extraction module 920 is further configured to: extract a fourth image feature of the second EEG signal;

[0451] The second feature extraction module 930 is further configured to: extract a fifth image feature of the second near-infrared brain functional imaging signal;

[0452] The second fusion module 960 is further configured to: fuse the fourth image feature of the second EEG signal and the fifth image feature of the second near-infrared functional brain imaging signal to obtain a sixth image feature;

[0453] The third training module 990 is configured to train the workload type recognition model using the sixth image feature and the workload type label as samples.

[0454] Optionally, the third training module may send the trained load type identification model to the second identification module 970 , so that the second identification module 970 may identify the user workload level according to the load type identification model.

[0455] The devices provided in the embodiments shown in Figures 14 to 18 can be used to implement the technical solutions of the method embodiments of the present application. The implementation principles and technical effects thereof can be further referred to the relevant descriptions in the method embodiments.

[0456] It should be understood that the division of the various modules of the devices shown in Figures 14 to 18 above is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or it can be integrated into a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. During the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0457] An embodiment of the present application also provides a workload identification system, including the device shown in any of the above embodiments.

[0458] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is used to implement the method provided in the embodiment of the present application.

[0459] An embodiment of the present application also provides a workload identification system, including an electronic device, an electroencephalogram signal acquisition device, and a near-infrared brain function imaging device, and the electronic device is used to implement the method provided in the embodiment of the present application.

[0460] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the method provided by the embodiment of the present application.

[0461] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program product is run on a computer, it enables the computer to execute the method provided by the embodiment of the present application.

[0462] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0463] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0464] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0465] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disk, and other media that can store program code.

[0466] The above description is merely a specific embodiment of the present application. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. The scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing the workload status of personnel based on multimodal data, characterized in that, Including: Collecting the status information of the first user; Determining the workload information of the first user according to the collected status information; wherein, the status information includes at least two of the following information: physiological information, eye movement information, electroencephalogram information, brain functional imaging information, motion capture information, spatio-temporal acquisition information, behavior acquisition information, facial expression and status information; Feeding back the first management account according to the workload information and formulating a training plan matching the workload information.

2. The method according to claim 1, characterized in that, When the status information of the first user includes the first electroencephalogram signal and the first near-infrared brain functional imaging signal of the first user, the determining the workload information of the first user according to the collected status information includes: Respectively extracting the first feature of the first electroencephalogram signal and the second feature of the first near-infrared brain functional imaging signal; Determining the workload information of the first user according to the first feature and the second feature.

3. The method according to claim 2, wherein The determining the workload information of the first user according to the first feature and the second feature includes: Performing a fusion process on the first feature and the second feature to obtain a first fusion feature; Inputting the first fusion feature into a preset workload recognition model, and taking the workload information output by the workload recognition model as the workload information of the first user.

4. The method according to claim 3, characterized in that, The first feature includes a first time series feature and a first image feature, and the second feature includes a second time series feature and a second image feature; The performing a fusion process on the first feature and the second feature to obtain a first fusion feature includes: Performing a fusion process on the first time series feature and the second time series feature to obtain a third time series feature; Performing a fusion process on the first image feature and the second image feature to obtain a third image feature; Performing a fusion process on the third time series feature and the third image feature to obtain the first fusion feature.

5. The method according to claim 4, wherein The performing a fusion process on the first time series feature and the second time series feature to obtain a third time series feature includes: Converting the first time series feature into a fourth image feature; Converting the second time series feature into a fifth image feature; Performing a fusion process on the fourth image feature and the fifth image feature to obtain a sixth image feature, and taking the sixth image feature as the third time series feature.

6. The method according to claim 4 or 5, characterized in that, The first time series feature includes: time domain feature, frequency domain feature, time-frequency domain feature, non-linear feature and / or brain functional connection feature; and / or, The first image feature includes: 2D atlas and / or 3D densely connected network; and / or, The second time series feature includes: concentration of oxygenated and deoxygenated hemoglobin, beta index of a single channel, and / or Pearson correlation coefficient between channels; and / or, The second image feature includes: 3D channel activation map, correlation matrix formed between channels, and / or brain network connection map.

7. The method according to any one of claims 1 to 6, characterized in that, The workload information includes: workload level and / or workload type.

8. The method according to claim 2, wherein The first feature includes a first timing feature and a first image feature, and the second feature includes a second timing feature and a second image feature; the workload information includes: a workload level and a workload type; Determining the workload information of the first user according to the first feature and the second feature includes: Performing a fusion process on the first timing feature and the second timing feature to obtain a third timing feature; identifying the workload level of the first user according to the third timing feature; Performing a fusion process on the first image feature and the second image feature to obtain a third image feature; identifying the workload type of the first user according to the third image feature.

9. The method according to claim 8, characterized in that Identifying the workload level of the first user according to the third timing feature includes: Inputting the third timing feature into a workload level identification model, and taking the workload level output by the workload level identification model as the workload level of the first user.

10. The method according to claim 8 or 9, characterized in that Identifying the workload type of the first user according to the third image feature includes: Inputting the third image feature into a workload type identification model, and taking the workload type output by the workload type identification model as the workload type of the first user.

11. The method according to any one of claims 8 to 10, characterized in that, The first timing feature includes: time domain feature, frequency domain feature, time-frequency domain feature, non-linear feature, and / or brain functional connectivity feature; and / or, The first image feature includes: electroencephalogram topographic map, and / or power spectrum topographic map; and / or, The second timing feature includes: concentrations of oxygenated and deoxygenated hemoglobin, beta index of a single channel, and / or Pearson correlation coefficient between channels; and / or, The second image feature includes: 3D channel activation map, complex brain network, default brain network, correlation heat map, and / or brain network connectivity map.

12. The method according to any one of claims 7 to 11, characterized in that, The workload type includes: auditory, visual, attention, execution, and / or planning.

13. The method according to any one of claims 4 to 12, characterized in that, Extracting the second image feature of the first near-infrared brain functional imaging signal includes: Extracting brain network diagrams with different functions from the first near-infrared brain functional imaging signal as the second image feature.

14. The method according to any one of claims 7 to 13, characterized in that It further includes: Performing a warning process on the user's workload according to the workload information.

15. The method according to claim 14, wherein Performing a warning process on the user's workload according to the workload information includes: When the workload level in the workload information is not lower than a preset level and the workload type in the workload information includes a first type, issuing an alarm for the first type of workload.

16. The method according to claim 14, wherein Performing a warning process on the user's workload according to the workload information includes: When the workload level in the workload information is not lower than a preset level and the workload type in the workload information includes a second type, determining the proportion of the second type in multiple workload information obtained within a preset first duration, and when the proportion is not lower than a preset proportion threshold, issuing an alarm for the second type of workload.

17. The method according to any one of claims 1 to 16, characterized in that, Formulating a training plan matching the workload information includes: Obtaining the training plan corresponding to the workload information as the training plan of the first user.

18. A personnel workload status processing device based on multimodal data, characterized in that, Comprising: A collection module, configured to collect the status information of a first user; A determination module, configured to determine the workload information of the first user according to the collected status information; wherein, the status information includes at least two of the following information: physiological information, eye movement information, electroencephalogram information, brain functional imaging information, motion capture information, spatio-temporal acquisition information, behavior acquisition information, facial expression and status information; A processing module, configured to feedback a first management account according to the workload information and formulate a training plan matching the workload status level.

19. A personnel workload status processing system based on multimodal data, characterized in that The system includes the device described in claim 18.

20. An electronic device, characterized in that, Comprising: A processor, a memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the processor, cause the electronic device to execute the method described in any one of claims 1 to 17.

21. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when it runs on a computer, it causes the computer to execute the method described in any one of claims 1 to 17.

Images