Controller working status detection method based on multimodal cognitive data fusion

Through multimodal data fusion and deep learning analysis, the working status of air traffic controllers is accurately detected, solving the problem of inaccurate monitoring in the existing technology, and improving monitoring efficiency and safety.

CN118761035BActive Publication Date: 2024-11-15SICHUAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411245080.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-15
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

In the prior art, the fatigue status monitoring of air traffic controllers is inaccurate due to inaccurate image and sound information, which affects monitoring efficiency. A method that can accurately detect the working status of the controllers is urgently needed.

Method used

The controller's working status detection method based on multimodal cognitive data fusion is adopted. By collecting voice data, eye movement data, complete head image data and physiological data, a multimodal decision fusion model is constructed, and analyzed in combination with machine deep learning. Real-time detection of the controller's working status.

Benefits of technology

Accurate detection of the working status of the controller is achieved, the work efficiency of the air traffic controller is improved, and the risk of unsafe accidents caused by fatigue is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761035B_ABST
    Figure CN118761035B_ABST
Patent Text Reader

Abstract

The present application relates to a method for detecting the working status of a controller based on multimodal cognitive data fusion. The method comprises: collecting multimodal data of the controller in the actual control work process, and performing data preprocessing to obtain a multimodal controller working status data set; constructing a controller working status detection model based on multimodal decision fusion, the controller working status detection model includes a single-modal decision module and a decision-level fusion module; training the controller working status detection model based on the multimodal controller working status data set; inputting the real-time controller multimodal data into the trained controller working status detection model to obtain the target working status. When detecting the working status of the controller, multimodal data is collected, and the data from multiple aspects and angles is analyzed and monitored in combination with machine deep learning to obtain the probability distribution of the controller's working status, and the controller's working status is determined in real time, so that the controller's working status can be accurately detected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of air traffic controller working status analysis, and in particular to a method for detecting the working status of air traffic controllers based on multimodal cognitive data fusion. Background Art

[0002] Air traffic controllers, or air traffic controllers, are key players in ensuring the safe operation of aircraft in the air. Their main responsibilities include commanding, monitoring and managing the take-off and landing and flight processes of aircraft to prevent collisions and improve flight efficiency. Their every move is related to the safety of modern air traffic and the safety of people's lives and property. Due to the high intensity and high pressure of the work of controllers, fatigue management has become an important part of it. For example, the continuous working time of controllers on duty in the tower, approach and regional control rooms shall not exceed 6 hours, and the continuous working time of controllers directly engaged in radar control shall not exceed 2 hours, and the time interval between two working hours shall not be less than 30 minutes. However, in the actual command process of controllers, unsafe accidents caused by fatigue still occur frequently. Therefore, real-time monitoring of the fatigue status of controllers has become particularly necessary and important.

[0003] In the existing technology, image and sound information monitoring is usually used, combined with artificial intelligence AI facial recognition algorithms, big data analysis, machine deep learning, etc. to conduct in-depth analysis to obtain the working status information of the controller and issue relevant fatigue status warnings. However, this method may lead to inaccurate analysis results from the source due to inaccurate image and sound information, which greatly affects the judgment and warning of fatigue status and reduces the efficiency of monitoring.

[0004] Therefore, in the relevant technology, there is an urgent need for a method that can accurately detect the working status of the controller to improve the work efficiency of the air traffic controller. Summary of the invention

[0005] Based on this, it is necessary to provide a controller working status detection method based on multimodal cognitive data fusion, which can accurately detect the working status of controllers to improve the work efficiency of air traffic controllers in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting the working status of a controller based on multimodal cognitive data fusion. The method comprises:

[0007] Collect multimodal data of controllers in the real control process, and perform data preprocessing to obtain a multimodal controller working status dataset;

[0008] Constructing a controller working state detection model based on multimodal decision fusion, the controller working state detection model includes a single-modal decision module and a decision-level fusion module, wherein the single-modal decision module is used to obtain the corresponding working state based on single-modal data, and the decision-level fusion module is used to fuse the working states of each single-modal decision module to obtain the final working state;

[0009] Training the controller working status detection model based on the multimodal controller working status data set;

[0010] The real-time controller multimodal data is input into the trained controller working state detection model to obtain the target working state.

[0011] Optionally, in one embodiment of the present application, the data preprocessing includes:

[0012] Collect voice data, eye movement data, complete head image data, and physiological data during real control work;

[0013] A working status tag is added according to a timestamp based on the voice data, eye movement data, complete head image data, and physiological data.

[0014] Optionally, in one embodiment of the present application, the unimodal decision module includes a voice data decision module, an eye movement data decision module, an image data decision module and a physiological data decision module, wherein the voice data decision module is used to obtain the corresponding working state based on the voice data, the eye movement data decision module is used to obtain the corresponding working state based on the eye movement data, the image data decision module is used to obtain the corresponding working state based on the complete head image data, and the physiological data decision module is used to obtain the corresponding working state based on the physiological data.

[0015] Optionally, in one embodiment of the present application, the speech data decision module includes an audio pre-training module, a decoder module and a decision module, wherein the audio pre-training module is used to determine the feature representation vector of the speech frame, the decoder module is used to parse the feature representation vector to obtain the recognition text of the speech signal and the corresponding emotion probability distribution, and the decision module is used to determine the working status of the controller.

[0016] Optionally, in one embodiment of the present application, the eye movement data decision module includes an image encoding module, an input embedding module and a decoder, wherein the image encoding module is used to determine the gaze point distribution characteristics, the input embedding module is used to determine the eye muscle movement characteristics, and the decoder is used to determine the controller's working status.

[0017] Optionally, in one embodiment of the present application, the image data decision module includes an image encoding module, a classification module and a decoder, wherein the image encoding module is used to determine the image representation, the classification module is used to determine the probability distribution of facial expression labels and head posture labels corresponding to the image signal, and the decoder is used to determine the working status of the controller.

[0018] Optionally, in one embodiment of the present application, the physiological data decision module includes an input embedding module and a timing modeling module, wherein the input embedding module is used to determine physiological information characteristics, and the timing modeling module is used to determine the working status of the controller.

[0019] Optionally, in one embodiment of the present application, the decision-level fusion module performs linear weighted fusion based on expert knowledge and the decision accuracy of the single-modal decision fusion module to obtain the final working state of the controller's working state.

[0020] Optionally, in one embodiment of the present application, the process of linear weighted fusion is:

[0021]

[0022] in, For the The decision probability distribution output by a single-modal decision module, is the decision probability distribution matrix, is the weight vector of decision probability distribution, is the final decision probability distribution, and Softmax is the activation function.

[0023] Optionally, in one embodiment of the present application, the training of the controller working status detection model based on the multimodal controller working status data set includes:

[0024] Input the speech data, eye movement data, complete head image data and physiological data into the speech data decision module, eye movement data decision module, image data decision module and physiological data decision module respectively, and output a predicted working state label;

[0025] Determine a cross entropy loss based on the predicted working state label and the working state labels corresponding to the speech data, eye movement data, complete head image data, and physiological data;

[0026] The parameters of the speech data decision module, the eye movement data decision module, the image data decision module and the physiological data decision module are updated based on the cross entropy loss.

[0027] The above-mentioned method for detecting the working state of a controller based on multimodal cognitive data fusion, firstly, collects the multimodal data of the controller in the real control process, and performs data preprocessing to obtain a multimodal controller working state data set; then, constructs a controller working state detection model based on multimodal decision fusion, the controller working state detection model includes a single-modal decision module and a decision-level fusion module, wherein the single-modal decision module is used to obtain the corresponding working state based on the single-modal data, and the decision-level fusion module is used to fuse the working states of each single-modal decision module to obtain the final working state; then, the controller working state detection model is trained based on the multimodal controller working state data set; finally, the real-time controller multimodal data is input into the trained controller working state detection model to obtain the target working state. That is to say, when detecting the working state of the controller, multimodal data is collected, and the data from multiple aspects and angles is analyzed and monitored in combination with machine deep learning to obtain the probability distribution of the working state of the controller, and the working state of the controller is determined in real time, which can accurately detect the working state of the controller, facilitate relevant fatigue state warnings, and further improve the work efficiency of air traffic controllers. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an application environment diagram of a method for detecting the working status of a controller based on multimodal cognitive data fusion in one embodiment;

[0029] Figure 2 1 is a flow chart of a method for detecting the working status of a controller based on multimodal cognitive data fusion in one embodiment;

[0030] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The controller working status detection method based on multimodal cognitive data fusion provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 101 communicates with the server 102 through the network. The data storage system can store the data that the server 102 needs to process. The data storage system can be integrated on the server 102, or it can be placed on the cloud or other network servers. Among them, the terminal 101 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 102 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0033] In one embodiment, Figure 2 As shown in the figure, a method for detecting the working status of controllers based on multimodal cognitive data fusion is provided. Figure 1 The server in the example is used to illustrate the following steps:

[0034] S201: Collect multimodal data of controllers during real control work, and perform data preprocessing to obtain a multimodal controller work status data set.

[0035] In the embodiment of the present application, firstly, the multimodal data of the controller in the real control process is collected by voice acquisition equipment, video acquisition equipment, eye tracker and physiological monitoring system installed on the controller's seat, and the data is preprocessed to obtain a multimodal controller working status data set. Among them, the data preprocessing includes manual labeling of relevant labels, data cleaning, and establishing data pairing operations by timestamp.

[0036] S203: Construct a controller working status detection model based on multimodal decision fusion, wherein the controller working status detection model includes a single-modal decision module and a decision-level fusion module, wherein the single-modal decision module is used to obtain the corresponding working status based on single-modal data, and the decision-level fusion module is used to fuse the working status of each single-modal decision module to obtain the final working status.

[0037] In an embodiment of the present application, after obtaining a multimodal controller working status data set, a controller working status detection model based on multimodal decision fusion is constructed. Specifically, the controller working status detection model includes a unimodal decision module and a decision-level fusion module, wherein the unimodal decision module is used to obtain the corresponding working status based on unimodal data, and the unimodal data refers to voice data, eye movement data, image data, physiological data, etc. The decision-level fusion module is used to fuse the decision results of each unimodal decision module to obtain the final decision result.

[0038] S205: Training the controller working status detection model based on the multimodal controller working status data set.

[0039] In the embodiment of the present application, a controller working status detection model based on multimodal decision fusion is constructed by training a multimodal controller working status data set. Specifically, the multimodal controller working status data set is divided into a training set, a test set, and a validation set, wherein the training set is used for the preliminary training model, the test set is used to evaluate the training effect of the model, and the validation set is used to evaluate the generalization ability of the model. During the training process, the corresponding parameters of each module of the model are updated by calculating the cross entropy loss and combining the cross entropy loss for error back propagation.

[0040] S207: Inputting the real-time controller multimodal data into the trained controller working status detection model to obtain the target working status.

[0041] In an embodiment of the present application, after the model training is completed, the controller's multimodal data is collected in real time through voice acquisition equipment, video acquisition equipment, eye tracker, physiological monitoring system and other equipment installed on the controller's seat, and input into the controller's working status detection model based on multimodal decision fusion. The model outputs the final decision, that is, the probability distribution of the controller's real-time working status, and the working status specifically includes normal, mild fatigue, moderate fatigue and abnormal.

[0042] In the above-mentioned method for detecting the working state of the controller based on multimodal cognitive data fusion, first, the multimodal data of the controller in the real control process is collected, and the data is preprocessed to obtain a multimodal controller working state data set; then, a controller working state detection model based on multimodal decision fusion is constructed, and the controller working state detection model includes a single-modal decision module and a decision-level fusion module; then, the controller working state detection model is trained based on the multimodal controller working state data set; finally, the real-time controller multimodal data is input into the trained controller working state detection model to obtain the target working state. That is to say, when detecting the working state of the controller, multimodal data is collected, and the data from multiple aspects and angles is analyzed and monitored in combination with machine deep learning to obtain the probability distribution of the working state of the controller, and the working state of the controller is determined in real time, which can accurately detect the working state of the controller, facilitate relevant fatigue state warnings, and further improve the work efficiency of air traffic controllers.

[0043] In one embodiment of the present application, the data preprocessing includes:

[0044] S301: Collect voice data, eye movement data, complete head image data, and physiological data during the actual control work process.

[0045] S303: Adding a working status tag according to the timestamp based on the voice data, eye movement data, complete head image data, and physiological data.

[0046] In one embodiment of the present application, first, the voice data, eye movement data, complete head image data and physiological data of the controller in the actual control work process are collected, wherein the voice data is manually annotated to obtain the text label corresponding to the voice data, as well as the emotional label; the eye movement data includes data such as blinking frequency, blinking duration, pupil diameter and gaze point distribution map; the complete head image data refers to the real-time image data covering the complete head of the controller and the corresponding head posture label, and is manually annotated to obtain the emotional label of the image data; the physiological data includes data such as skin resistance, heart rate and heart rate variability. Afterwards, the collected voice data, eye movement data, complete head image data, and physiological data are cleaned, and the corresponding working status label is added according to the timestamp of each data to establish a data pair, that is, the data pair contains a time length of The speech, eye movement, image, physiological sequence data and corresponding work status labels include normal, mild fatigue, moderate fatigue and abnormal.

[0047] In an embodiment of the present application, by collecting voice data, eye movement data, complete head image data, and physiological data during the actual control work process, and adding work status labels according to timestamps based on the voice data, eye movement data, complete head image data, and physiological data, a real, complete, and uniformly formatted data set can be obtained for subsequent model training.

[0048] In one embodiment of the present application, the unimodal decision module includes a voice data decision module, an eye movement data decision module, an image data decision module and a physiological data decision module, wherein the voice data decision module is used to obtain the corresponding working state based on the voice data, the eye movement data decision module is used to obtain the corresponding working state based on the eye movement data, the image data decision module is used to obtain the corresponding working state based on the complete head image data, and the physiological data decision module is used to obtain the corresponding working state based on the physiological data.

[0049] In one embodiment of the present application, the unimodal decision module includes a voice data decision module, an eye movement data decision module, an image data decision module and a physiological data decision module, each of which analyzes the probability distribution of the working state of the controller according to the time-series unimodal data, and the probability distribution of the working state of the controller is called a decision. Specifically, the voice data decision module is used to obtain the corresponding working state based on the voice data, the eye movement data decision module is used to obtain the corresponding working state based on the eye movement data, the image data decision module is used to obtain the corresponding working state based on the complete head image data, and the physiological data decision module is used to obtain the corresponding working state based on the physiological data.

[0050] In one embodiment of the present application, the speech data decision module includes an audio pre-training module, a decoder module and a decision module, wherein the audio pre-training module is used to determine the feature representation vector of the speech frame, the decoder module is used to parse the feature representation vector to obtain the recognition text of the speech signal and the corresponding emotion probability distribution, and the decision module is used to determine the working status of the controller.

[0051] In one embodiment of the present application, the speech data decision module includes an audio pre-training module, a decoder module and a decision module, wherein the audio pre-training module uses the speech signal as input to obtain a feature representation vector of the speech frame; thereafter, the feature representation vector of the speech frame is input into the decoder module to obtain the recognition text of the speech signal and the corresponding emotion probability distribution.

[0052] The audio pre-training module consists of a convolution module, a Transformer encoder, a quantization representation module and a context representation module; the convolution module consists of multiple one-dimensional convolution modules, which include a one-dimensional convolution layer, a group normalization layer and a GELU activation function to complete the downsampling operation of the speech signal and obtain the downsampled feature representation; the Transformer encoder performs feature mining on the downsampled feature representation to obtain the speech coding representation; the quantization representation module consists of a quantization representation tensor, which is a three-dimensional tensor with dimensions of codebook, codeword and feature respectively. The quantization representation module accepts the probability distribution of the quantization representation and outputs the quantization representation according to the maximum likelihood estimation. The quantization representation tensor participates in parameter updating during the pre-training process; the context representation module consists of multiple linear mapping layers to learn the probability distribution of the quantization representation from the speech coding representation.

[0053] The decoder module consists of a classification vector, a Transformer encoder and two classification modules; the classification vector is a one-dimensional tensor, which participates in parameter update during fine-tuning, and its corresponding output in the Transformer encoder will be used as the implicit feature of the speech emotion label; the Transformer encoder performs feature mining on the classification vector and the speech coding representation output by the audio pre-training module, and outputs the corresponding implicit features; the two classification modules are composed of multiple linear mapping layers and RELU activation functions, and the last activation function is the Softmax activation function. The first classification module completes the probability distribution estimation of speech emotion and uses maximum likelihood estimation to obtain the final speech emotion judgment. The second classification module completes the probability distribution estimation of the recognized text and decodes it through the greedy search or beam search algorithm to obtain a readable text sequence.

[0054] The decision module will count the three key indicators in the readable text sequence, including the number of times similar flight numbers were not reminded, the number of times the "correction" text appeared, and the speaking speed. It will use expert knowledge to give the normal, slightly abnormal, moderately abnormal, and abnormal intervals of the three key indicators and speech emotions, and classify the three key indicators and speech emotions into intervals; use the interval labels of the three key indicators and speech emotions to build a decision classifier and train it; the decision classifier is composed of multiple linear mapping layers and Sigmoid activation functions, and the last activation function is the Softmax activation function. The input sequence is a 16-dimensional vector. Each 4-dimensional vector in the input sequence corresponds to one of the three key indicators and speech emotions. If the element is 1, it means that the indicator is in the The output sequence is a 4-dimensional vector, which represents the probability of the controller's working status being normal, slightly fatigued, moderately fatigued, and abnormal.

[0055] In one embodiment of the present application, the eye movement data decision module includes an image encoding module, an input embedding module and a decoder, wherein the image encoding module is used to determine the gaze point distribution characteristics, the input embedding module is used to determine the eye muscle movement characteristics, and the decoder is used to determine the controller's working status.

[0056] In one embodiment of the present application, the eye movement data decision module is composed of an image encoding module, an input embedding module and a decoder. The image encoding module uses a single-channel gaze point distribution map as input to obtain gaze point distribution features; the input embedding module uses a low-dimensional vector composed of blink frequency, blink duration, and pupil diameter as input to obtain eye muscle movement features; the decoder uses the cascaded gaze point distribution features and eye muscle movement features as input to output the probability distribution of the controller's work status label.

[0057] The image encoding module is composed of multiple VGG convolution blocks, which completes the feature mining and data dimension reduction of image signals; the VGG convolution block is composed of multiple two-dimensional convolution blocks, RELU activation functions and maximum pooling layers. The input embedding module is composed of multiple linear mapping layers and RELU activation functions, which embeds the multivariate time series composed of blink frequency, blink duration and pupil diameter to obtain the eye muscle movement characteristics. The decoder is composed of classification vectors, Transformer encoders and classification modules; the classification vector is a one-dimensional tensor, which participates in parameter update during the training process, and its corresponding output in the Transformer encoder will be used as the implicit feature of the controller's work status label; the Transformer encoder performs feature mining on the cascaded gaze point distribution features, eye muscle movement features and classification vectors, and outputs the corresponding implicit features; the classification module is composed of multiple linear mapping layers and RELU activation functions, and the last activation function is the Softmax activation function, which completes the probability distribution estimation of the controller's work status label.

[0058] In one embodiment of the present application, the image data decision module includes an image encoding module, a classification module and a decoder, wherein the image encoding module is used to determine the image representation, the classification module is used to determine the probability distribution of the facial expression label and the head posture label corresponding to the image signal, and the decoder is used to determine the working status of the controller.

[0059] In one embodiment of the present application, the image data decision module is composed of an image encoding module, two classification modules, and a decoder. The image encoding module uses the image signal as input, performs feature mining on the image signal, and obtains an image representation; then, the image representation is input into the two classification modules to obtain the probability distribution of the facial expression label and the head posture label corresponding to the image signal; then, the decoder uses the probability distribution of the facial expression label and the time series of the head posture label as input, and outputs the probability distribution of the controller's working status label.

[0060] The image encoding module is composed of multiple VGG convolution blocks, which complete the feature mining and data dimension reduction of image signals; the VGG convolution block is composed of multiple two-dimensional convolution blocks, RELU activation functions and maximum pooling layers. The two classification modules are composed of multiple linear mapping layers and RELU activation functions, and the last activation function of the first classification module is the Softmax activation function, and the last activation function of the second classification module is the identity function; the first classification module completes the probability distribution estimation of facial expression labels (establishing expression labels and classifying them by yourself; focusing attention), and the second classification module completes the estimation of head posture labels. The decoder consists of a classification vector, a Transformer encoder and a classification module; the classification vector is a one-dimensional tensor, which participates in parameter updating during the training process, and its corresponding output in the Transformer encoder will be used as the implicit feature of the controller's work status label; the Transformer encoder performs feature mining on the probability distribution of facial expression labels, the concatenated vector of head posture labels and the classification vector, and outputs the corresponding implicit features; the classification module is composed of multiple linear mapping layers and RELU activation functions, and the last activation function is the Softmax activation function, which completes the probability distribution estimation of the controller's work status label.

[0061] In one embodiment of the present application, the physiological data decision module includes an input embedding module and a timing modeling module, wherein the input embedding module is used to determine physiological information characteristics, and the timing modeling module is used to determine the working status of the controller.

[0062] In one embodiment of the present application, the physiological data decision module is composed of an input embedding module and a time series modeling module. The input embedding module uses a low-dimensional vector composed of skin resistance, heart rate and heart rate variability as input to obtain physiological information features; the time series modeling module uses the physiological information features as input, outputs corresponding implicit features, and completes the probability distribution estimation of the controller's work status label.

[0063] The input embedding module is composed of multiple linear mapping layers and RELU activation functions, which embed the low-dimensional vectors composed of skin resistance, heart rate and heart rate variability to obtain physiological information features. The time series modeling module is composed of classification vectors, Transformer encoders and classification modules; the classification vector is a one-dimensional tensor, which participates in parameter update during the training process, and its corresponding output in the Transformer encoder will be used as the implicit feature of the controller's work status label; the Transformer encoder performs feature mining on the physiological information features and classification vectors, and outputs the corresponding implicit features; the classification module is composed of multiple linear mapping layers and RELU activation functions, and the last activation function is the Softmax activation function, which completes the probability distribution estimation of the controller's work status label.

[0064] In one embodiment of the present application, the decision-level fusion module performs linear weighted fusion based on expert knowledge and the decision accuracy of the single-modal decision fusion module to obtain the final working state of the controller's working state.

[0065] In one embodiment of the present application, the decision-level fusion module fuses the decisions made by the four unimodal decision modules, that is, linear weighted fusion is performed based on expert knowledge and the decision accuracy of the unimodal decision fusion module, and the final working state of the controller is output. Specifically, the expert uses a 1-9 scale to score the contribution of voice, eye movement, image, and physiological data to the detection of the controller's working state, and obtains a contrast matrix of multimodal data, as shown in the following formula.

[0066]

[0067] in, is the contrast matrix of multimodal data, Indicates Single modal data for The relative importance of single-modal data satisfies the following rules:

[0068] (1) As the relative importance increases, Starting from 1, odd numbers increase to 9. It means that both have the same importance. It means that the before and after are extremely important compared to the latter.

[0069] (2) When the metric granularity in (1) is too large, an even number between two odd numbers can be used as an intermediate value of the adjacent relative importance;

[0070] (3) The elements in ;

[0071] Afterwards, four unimodal decision accuracy pairs were used Correction is made to satisfy the following formula:

[0072]

[0073] in, For the A unimodal decision module The accuracy is the working status label of all test samples in the first The front of a unimodal decision module The ratio of the number of decision probabilities to the number of all test samples, for Parameters in accuracy.

[0074] Afterwards, Normalize the column vector to get :

[0075]

[0076] in, for elements.

[0077] Afterwards, The weight vector is obtained by summing and normalizing the row vectors, as shown in the following formula:

[0078]

[0079] in, is the weight vector; For the The weights of the unimodal decision modules, for The sum of all elements.

[0080] Afterwards, Perform consistency check, as shown below:

[0081]

[0082] in, is the weight vector after linear transformation, for Elements of is the maximum eigenvalue, is the test value, is the average random consistency index when the matrix order is 4, if , then it is considered that the construction If a logical error occurs, the expert scoring will be repeated; if , then it is believed that Has good consistency, can be used as the final weight vector.

[0083] Afterwards, linear weighted fusion is performed based on the weight vector to obtain the final decision probability distribution, and the final working state is obtained using maximum likelihood estimation.

[0084] In an embodiment of the present application, the decision-level fusion module performs linear weighted fusion based on expert knowledge and the decision accuracy of the single-modal decision fusion module to obtain the final working state of the controller, and can accurately detect the working state of the controller.

[0085] Specifically, in one embodiment of the present application, the process of linear weighted fusion is:

[0086]

[0087] in, For the The decision probability distribution output by a single-modal decision module, is the decision probability distribution matrix, is the weight vector of decision probability distribution, is the final decision probability distribution, and Softmax is the activation function.

[0088] In one embodiment of the present application, the training of the controller working status detection model based on the multimodal controller working status data set includes:

[0089] S401: Input the voice data, eye movement data, complete head image data and physiological data into the voice data decision module, eye movement data decision module, image data decision module and physiological data decision module respectively, and output a predicted working status label.

[0090] S403: Determine a cross entropy loss based on the predicted working state label and the working state labels corresponding to the speech data, eye movement data, complete head image data, and physiological data.

[0091] S405: Update the parameters of the speech data decision module, the eye movement data decision module, the image data decision module and the physiological data decision module based on the cross entropy loss.

[0092] In one embodiment of the present application, the training process of the speech data decision module is divided into two steps. The first step is: first, the existing publicly available pre-trained network parameters are loaded into the audio pre-training module of the speech data decision module, and then the speech sequence data is used as input to obtain the probability distribution estimate of the recognized text and speech emotion, and finally the cross-entropy loss is calculated together with the text label and the emotion label corresponding to the speech sequence data, and the error back propagation is performed to update the network parameters of the decoder module; the second step is: fix the network parameters of the audio pre-training module and the decoder module of the speech data decision module, use the speech sequence data as input, obtain the probability distribution estimate of the speech emotion and the recognized text, and then use expert knowledge to perform interval classification and use it as the input of the decision module of the speech data decision module to obtain the probability distribution estimate of the controller's working status label, calculate the cross-entropy loss together with the controller's working status label corresponding to the speech sequence data, and perform error back propagation to update the network parameters of the decision module.

[0093] The training process of the eye movement data decision module is as follows: taking the eye movement sequence data as input, obtaining the probability distribution estimate of the controller's working status label, calculating the cross entropy loss together with the controller's working status label corresponding to the eye movement sequence data, and performing error back propagation to update the network parameters of all modules.

[0094] The training process of the image data decision module is divided into two steps. The first step is to take the image sequence data as input, obtain the probability distribution estimate of the facial expression label and the head posture label corresponding to the image sequence data, calculate the cross entropy loss together with the facial expression label and the head posture label corresponding to the image sequence data, and perform error back propagation to update the network parameters except the decoder module; the second step is to fix the network parameters except the decoder module in the image data decision module, take the image sequence data as input, obtain the probability distribution estimate of the facial expression label and the head posture label corresponding to the image sequence data and input them into the decoder module of the image data decision module, obtain the probability distribution estimate of the controller's working status label, calculate the cross entropy loss together with the controller's working status label corresponding to the image sequence data, and perform error back propagation to update the network parameters of the decoder module.

[0095] The training process of the physiological data decision module is as follows: taking the physiological sequence data as input, obtaining the probability distribution estimate of the controller's working status label, calculating the cross entropy loss together with the controller's working status label corresponding to the physiological sequence data, and performing error back propagation to update the network parameters of all modules.

[0096] In an embodiment of the present application, the voice data, eye movement data, complete head image data and physiological data are respectively input into the voice data decision module, the eye movement data decision module, the image data decision module and the physiological data decision module to output a predicted working state label. The cross entropy loss is determined based on the predicted working state label and the working state label corresponding to the voice data, eye movement data, complete head image data and physiological data. The parameters of the voice data decision module, the eye movement data decision module, the image data decision module and the physiological data decision module are updated based on the cross entropy loss. The detection effect of the controller working state detection model based on multimodal decision fusion obtained by training can be more accurate and the generalization effect can be better.

[0097] The following is a specific example to illustrate the specific implementation steps of the controller working status detection method based on multimodal cognitive data fusion of the present application. First, S501, collect the multimodal data of the controller in the actual control work process, and perform data preprocessing to obtain a multimodal controller working status data set. Specifically, S503-S505, collect voice data, eye movement data, complete head image data, and physiological data in the actual control work process; based on the voice data, eye movement data, complete head image data, and physiological data, add a working status label according to the timestamp.

[0098] Afterwards, S507, a controller working state detection model based on multimodal decision fusion is constructed, wherein the controller working state detection model includes a unimodal decision module and a decision-level fusion module, wherein the unimodal decision module is used to obtain the corresponding working state based on unimodal data, and the decision-level fusion module is used to fuse the working states of each unimodal decision module to obtain the final working state. Specifically, S509, the unimodal decision module includes a voice data decision module, an eye movement data decision module, an image data decision module and a physiological data decision module, wherein the voice data decision module is used to obtain the corresponding working state based on voice data, the eye movement data decision module is used to obtain the corresponding working state based on eye movement data, the image data decision module is used to obtain the corresponding working state based on complete head image data, and the physiological data decision module is used to obtain the corresponding working state based on physiological data. S511-S517, the speech data decision module includes an audio pre-training module, a decoder module and a decision module, wherein the audio pre-training module is used to determine the feature representation vector of the speech frame, the decoder module is used to parse the feature representation vector to obtain the recognition text of the speech signal and the corresponding emotion probability distribution, and the decision module is used to determine the working status of the controller. The eye movement data decision module includes an image encoding module, an input embedding module and a decoder, wherein the image encoding module is used to determine the gaze point distribution characteristics, the input embedding module is used to determine the eye muscle movement characteristics, and the decoder is used to determine the working status of the controller. The image data decision module includes an image encoding module, a classification module and a decoder, wherein the image encoding module is used to determine the image representation, the classification module is used to determine the probability distribution of the facial expression label and the head posture label corresponding to the image signal, and the decoder is used to determine the working status of the controller. The physiological data decision module includes an input embedding module and a time series modeling module, wherein the input embedding module is used to determine the physiological information characteristics, and the time series modeling module is used to determine the working status of the controller. S519, the decision-level fusion module performs linear weighted fusion based on expert knowledge and the decision accuracy of the single-mode decision fusion module to obtain the final working state of the controller. Among them, S521, the process of linear weighted fusion is:

[0099]

[0100] in, For the The decision probability distribution output by a single-modal decision module, is the decision probability distribution matrix, is the weight vector of decision probability distribution, is the final decision probability distribution, and Softmax is the activation function.

[0101] Afterwards, S523, the controller working state detection model is trained based on the multimodal controller working state data set. Specifically, S525-S529, the voice data, eye movement data, complete head image data and physiological data are respectively input into the voice data decision module, eye movement data decision module, image data decision module and physiological data decision module, and a predicted working state label is output; a cross entropy loss is determined based on the predicted working state label and the working state label corresponding to the voice data, eye movement data, complete head image data and physiological data; and the parameters of the voice data decision module, eye movement data decision module, image data decision module and physiological data decision module are updated based on the cross entropy loss.

[0102] Finally, S531, the real-time controller multimodal data is input into the trained controller working status detection model to obtain the target working status.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting the working status of a controller based on multimodal cognitive data fusion is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the shell of the computer device, or an external keyboard, touchpad or mouse.

[0105] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0108] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting the working status of a controller based on multimodal cognitive data fusion, characterized in that: The method comprises: Collect multimodal data of controllers in the real control process, and perform data preprocessing to obtain a multimodal controller working status dataset; Constructing a controller working state detection model based on multimodal decision fusion, the controller working state detection model includes a single-modal decision module and a decision-level fusion module, wherein the single-modal decision module is used to obtain the corresponding working state based on single-modal data, and the decision-level fusion module is used to fuse the working states of each single-modal decision module to obtain the final working state; Training the controller working status detection model based on the multimodal controller working status data set; Inputting the real-time controller multimodal data into the trained controller working state detection model to obtain the target working state; The data preprocessing includes: Collect voice data, eye movement data, complete head image data, and physiological data during real control work; Adding a working status tag according to the timestamp based on the voice data, eye movement data, complete head image data, and physiological data; The single-modal decision module includes a voice data decision module, an eye movement data decision module, an image data decision module and a physiological data decision module, wherein the voice data decision module is used to obtain a corresponding working state based on voice data, the eye movement data decision module is used to obtain a corresponding working state based on eye movement data, the image data decision module is used to obtain a corresponding working state based on complete head image data, and the physiological data decision module is used to obtain a corresponding working state based on physiological data; The voice data decision module includes an audio pre-training module, a decoder module and a decision module, wherein the audio pre-training module is used to determine the feature representation vector of the voice frame, the decoder module is used to parse the feature representation vector to obtain the recognition text of the voice signal and the corresponding emotion probability distribution, and the decision module is used to determine the working status of the controller; The eye movement data decision module includes an image encoding module, an input embedding module and a decoder, wherein the image encoding module is used to determine the gaze point distribution characteristics, the input embedding module is used to determine the eye muscle movement characteristics, and the decoder is used to determine the controller's working status; The image data decision module includes an image encoding module, a classification module and a decoder, wherein the image encoding module is used to determine the image representation, the classification module is used to determine the probability distribution of the facial expression label and the head posture label corresponding to the image signal, and the decoder is used to determine the working status of the controller; The physiological data decision module includes an input embedding module and a timing modeling module, wherein the input embedding module is used to determine the physiological information characteristics, and the timing modeling module is used to determine the working status of the controller.

2. The method for detecting the working status of a controller based on multimodal cognitive data fusion according to claim 1 is characterized in that: The decision-level fusion module performs linear weighted fusion based on expert knowledge and the decision accuracy of the single-mode decision fusion module to obtain the final working state of the controller's working state.

3. The method for detecting the working status of a controller based on multimodal cognitive data fusion according to claim 2 is characterized in that: The process of linear weighted fusion is as follows: in, For the The decision probability distribution output by a single-modal decision module, is the decision probability distribution matrix, is the weight vector of decision probability distribution, is the final decision probability distribution, and Softmax is the activation function.

4. The method for detecting the working status of a controller based on multimodal cognitive data fusion according to claim 1, characterized in that: The training of the controller working status detection model based on the multimodal controller working status data set includes: Input the speech data, eye movement data, complete head image data and physiological data into the speech data decision module, eye movement data decision module, image data decision module and physiological data decision module respectively, and output a predicted working state label; Determine a cross entropy loss based on the predicted working state label and the working state labels corresponding to the speech data, eye movement data, complete head image data, and physiological data; The parameters of the speech data decision module, the eye movement data decision module, the image data decision module and the physiological data decision module are updated based on the cross entropy loss.