Emotion recognition and feedback method and system based on artificial intelligence
Through the emotional recognition and feedback method based on artificial intelligence, facial expressions and voice data are used to evaluate the emotional state and work efficiency of crew members, solving the problem of difficult monitoring of crew members in maritime working environments, and achieving emotional monitoring and work efficiency improvement.
Patent Information
- Application Number
- CN202411564595.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-14
AI Technical Summary
In a high-pressure maritime working environment, the emotional state of crew members is difficult to effectively monitor and analyze, affecting their work efficiency and mental health.
Using an emotional recognition and feedback method based on artificial intelligence, we collect facial expression data and voice data of crew members, calculate facial emotion index, voice emotion index and comprehensive emotion index, and then evaluate work efficiency and provide corresponding suggestions.
Real-time monitoring of crew members' emotional status, accurately assess work efficiency, provide personalized suggestions, improve work efficiency and mental health, and reduce safety hazards.
Smart Images

Figure CN119445632A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer system engineering, and in particular relates to an emotion recognition and feedback method and system based on artificial intelligence. Background Art
[0002] In modern society, especially in high-pressure working environments, the relationship between crew members' work efficiency and emotional state has received increasing attention. Crew members face multiple challenges when performing tasks at sea, including high-intensity workloads, harsh environmental conditions, and long periods of loneliness. These factors not only affect the physical health of crew members, but are also directly related to their mental state and work efficiency. Therefore, how to monitor and analyze the emotional state of crew members through scientific means to improve their work efficiency has become an important research topic. Summary of the invention
[0003] In view of the defects existing in the above-mentioned prior art, the present invention provides an emotion recognition and feedback method based on artificial intelligence, comprising the following steps:
[0004] Step S101, collecting facial expression data and voice data of crew members;
[0005] Step S103: Obtain facial emotion index E based on facial expression data f ;
[0006] Step S105: Obtain the voice emotion index E based on the voice data v ;
[0007] Step S107, obtaining a comprehensive emotion index E based on the facial emotion score and the voice emotion index;
[0008] Step S109: Calculate the crew work efficiency based on the comprehensive emotion index E and provide corresponding suggestions, wherein the crew work efficiency is calculated by the following formula:
[0009] Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, and β is the adjustment coefficient.
[0010] Wherein, the step S103 includes: using the following formula to obtain the facial emotion index E f ,
[0011] Where t is the current time, T i is the acquisition time of the i-th frame image, δ tis the time standard deviation, I i is the i-th frame image, η i is the weight of the i-th image, Relu(x)=max(0,x) is the activation function, N f is the total number of frames of facial images.
[0012] Among them, the δ t The time standard deviation is obtained by the following steps:
[0013] When capturing each frame of the image, record the corresponding timestamp;
[0014] Assume there are M timestamps T1, T2, ...T M , which can be expressed as T i =t i -t0, where t i is the acquisition time of the i-th frame image, and t0 is the starting time;
[0015] The mean μ of the timestamp t The following formula is used to express
[0016] The time standard deviation
[0017] Wherein, the step S105 includes: using the following formula to obtain the voice emotion index E v ,
[0018] Among them, P j is the weight of the jth speech segment, Q j is the jth speech segment, T j is the time feature, φ is the sentiment classification function, F(Q j ) is the Fourier transform of the jth speech segment, |F(Q j )| 2 is the energy spectrum in the frequency domain, F max is the maximum frequency.
[0019] Wherein, the sentiment classification function is represented by a support vector machine model.
[0020] Wherein, the sentiment classification function is expressed as follows:
[0021] φ(Q j )=sign(W Q T ·F(Q j )+b), where φ(Q j ) represents the sentiment classification result of the jth speech segment, W Q is the weight vector, representing the normal vector of the hyperplane, W Q TRepresents the weight vector W Q The transpose of F(Q j ) is the feature vector extracted from the jth speech segment, b is the bias term, and sign(·) is the sign function.
[0022] The following formula is used in step S107 to obtain the comprehensive emotion index E:
[0023] Among them, α and β are E f , E v The weight coefficient of , N is the normalization coefficient.
[0024] Wherein, the maximum value normalization is used to obtain N, N = max(E f , E v ).
[0025] The following formula is used to obtain the workload L in step S109:
[0026] L=w1·T c +w2·T p +w3·T N , where T c Score the task complexity, T p Rating time pressure, T N is the environmental noise score, w1, w2, w3 are the weight coefficients of each factor.
[0027] The present invention also proposes an emotion recognition and feedback system based on artificial intelligence, comprising:
[0028] A facial data collection module, which is used to collect facial expression data of crew members;
[0029] A voice data collection module, which is used to collect the voice data of the crew;
[0030] The facial emotion index calculation module is used to obtain the facial emotion index E based on the facial expression data. f ;
[0031] The voice emotion index calculation module is used to obtain the voice emotion index E based on the voice data. v ;
[0032] A comprehensive emotion index calculation module, which is used to obtain a comprehensive emotion index E based on the facial emotion score and the voice emotion index;
[0033] A work efficiency calculation module is used to calculate the crew work efficiency based on the comprehensive emotion index E, wherein the crew work efficiency is calculated by the following formula:
[0034] Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, β is the adjustment coefficient;
[0035] The suggestion making module is used to provide corresponding suggestions based on the work efficiency calculation results.
[0036] Compared with the prior art, the present invention has the following advantages:
[0037] By monitoring the emotional state of crew members in real time, changes in their work efficiency can be identified in a timely manner. The comprehensive calculation of the emotional index can provide managers with an accurate work efficiency assessment, and then improve overall work efficiency by adjusting work tasks and arrangements.
[0038] Emotional state directly affects the decision-making ability and reaction speed of crew members. By implementing an emotional monitoring system, an alarm can be issued in time when the crew members are depressed or under too much pressure, reminding them to take rest or adjustment measures, thereby reducing safety hazards caused by bad emotions.
[0039] Continuous high-pressure working environment can easily lead to mental health problems for crew members. Through scientific emotional assessment and feedback mechanisms, we can help crew members recognize their own emotional changes, provide necessary psychological support and intervention, and promote mental health.
[0040] Emotional analysis based on facial expressions and voice data can provide each crew member with personalized emotional status reports and work suggestions. This personalized management can tailor work arrangements for each crew member based on their emotional characteristics and workload, thereby improving satisfaction and work enthusiasm.
[0041] Through the long-term collection and analysis of emotional data, a rich database can be formed to provide data-driven decision support for management. Analyzing historical data can identify the correlation between emotional state and work efficiency, helping to optimize workflow and team management.
[0042] Transparency in emotional states helps to enhance understanding and support among team members. When team members can understand each other’s emotional states, they can communicate and collaborate more effectively, enhancing team cohesion.
[0043] The present invention combines advanced emotion recognition technologies (such as computer vision and speech processing) to promote the modernization and intelligence of crew management technology, enabling crew members to better adapt to changes brought about by technological development.
[0044] Through the emotion monitoring system, the interaction between the crew and the ship management system can be more intelligent and humane. The system can automatically adjust the environmental settings (such as lighting and noise level) according to the emotional state of the crew to provide more comfortable working conditions.
[0045] During the training process, by monitoring the emotional state of trainees in real time, trainers can adjust the content and rhythm of training according to the emotional changes of trainees, thereby improving the effectiveness of training and the learning experience of trainees. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:
[0047] Figure 1 The figure is a flow chart showing an emotion recognition and feedback method based on artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.
[0050] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....
[0051] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0052] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0053] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.
[0054] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0055] Embodiment 1
[0056] like Figure 1 As shown, the present invention discloses an emotion recognition and feedback evaluation method based on artificial intelligence, comprising the following steps:
[0057] Step S101, collecting facial expression data and voice data of crew members;
[0058] Step S103: Obtain facial emotion index E based on facial expression data f ;
[0059] Step S105: Obtain the voice emotion index E based on the voice data v ;
[0060] Step S107, obtaining a comprehensive emotion index E based on the facial emotion score and the voice emotion index;
[0061] Step S109: Calculate the crew work efficiency based on the comprehensive emotion index E and provide corresponding suggestions, wherein the crew work efficiency is calculated by the following formula:
[0062] Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, and β is the adjustment coefficient.
[0063] Embodiment 2
[0064] The present invention proposes an emotion recognition and feedback method based on artificial intelligence, comprising the following steps:
[0065] Step S101, collecting facial expression data and voice data of crew members;
[0066] Step S103: Obtain facial emotion index E based on facial expression data f ;
[0067] Step S105: Obtain the voice emotion index E based on the voice data v ;
[0068] Step S107, obtaining a comprehensive emotion index E based on the facial emotion score and the voice emotion index;
[0069] Step S109: Calculate the crew work efficiency based on the comprehensive emotion index E and provide corresponding suggestions, wherein the crew work efficiency is calculated by the following formula:
[0070] Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, and β is the adjustment coefficient.
[0071] δ can be derived through experimental research or data analysis, usually through regression analysis to determine the specific impact of emotions on work efficiency.
[0072] Among them, Cov(E,W) is the covariance between the comprehensive emotional state index E and work efficiency W; Var(E) is the variance of the comprehensive emotional state index E. L is the mean value of workload, σ L The standard deviation of the workload is obtained and δ t The calculation for the time standard deviation and mean is similar and will not be repeated here. The adjustment coefficient β can be determined by experiments or data analysis, usually based on regression analysis of the interactive effects of emotional state and workload on work efficiency.
[0073] Wherein, the step S103 includes: using the following formula to obtain the facial emotion index E f ,
[0074] Where t is the current time, T i is the acquisition time of the i-th frame image, δ t is the time standard deviation, Ii is the i-th frame image, usually a two-dimensional array (matrix), containing the pixel values of the image, representing the facial expressions of the crew members; η i is the weight of the i-th image, which is used to measure the contribution of the frame image to the emotion score. The weight can be obtained through machine learning model training and is usually a scalar value; Relu(x)=max(0,x) is the activation function, N f is the total number of frames of facial images.
[0075] Among them, the δ t The time standard deviation is obtained by the following steps:
[0076] When capturing each frame of the image, record the corresponding timestamp;
[0077] Assume there are M timestamps T1, T2, ...T M , which can be expressed as T i =t i -t0, where t i is the acquisition time of the i-th frame image, and t0 is the starting time;
[0078] The mean μ of the timestamp t The following formula is used to express
[0079] The time standard deviation
[0080] Wherein, the step S105 includes: using the following formula to obtain the voice emotion index E v ,
[0081] Among them, P j is the weight of the jth speech segment, which is obtained through machine learning model training. The importance of the speech segment is usually evaluated based on its characteristics. For example, the output of a regression model or a neural network can be used as the weight; Q j is the jth speech segment, T j is the time feature, which is obtained by calculating the start and end time of the speech segment or the duration of the speech segment, φ is the emotion classification function, F(Q j )=FFT(Q j ) is the Fourier transform of the jth speech segment, is the fast Fourier transform function, is the energy spectrum in the frequency domain, |F(Q j ) f | is the Fourier transform amplitude of frequency f, F max is the maximum frequency, usually based on the sampling rate f s Calculated using the Nyquist theorem, where f sis the signal sampling rate.
[0082] Wherein, the sentiment classification function is represented by a support vector machine model.
[0083] Wherein, the sentiment classification function is expressed as follows:
[0084] φ(Q j )=sign(W Q T ·F(Q j )+b), where φ(Q j ) represents the sentiment classification result of the jth speech segment, W Q is a weight vector, representing the normal vector of the hyperplane, which is obtained by optimizing the training data and reflects the influence of each feature on the classification result. Q T Represents the weight vector W Q The transpose of F(Q j ) is the feature vector extracted from the jth speech segment, including frequency features, time domain features, etc., b is the bias term, and sign(·) is the sign function.
[0085] The symbol function sign is used to determine the classification result. If (W Q T ·F(Q j )+b)>0, it is classified as positive (for example, positive emotion); if (W Q T ·F(Q j )+b)<0, it is classified as a negative class (e.g., negative emotions). The bias term b is used to adjust the position of the hyperplane to ensure the flexibility of classification.
[0086] The following formula is used in step S107 to obtain the comprehensive emotion index E:
[0087] Among them, α and β are E f , E v The weight coefficient of , N is the normalization coefficient.
[0088] Wherein, the maximum value normalization is used to obtain N, N = max(E f , E v ).
[0089] The following formula is used to obtain the workload L in step S109:
[0090] L=w1·T c +w2·T p +w3·T N , where T c Score the task complexity, Tp Rating time pressure, T N is the environmental noise score, w1, w2, w3 are the weight coefficients of each factor, which are usually obtained through expert evaluation or machine learning model training.
[0091] If the value of W is high (for example, greater than a preset threshold), it means that the crew's work efficiency is good, and the system can suggest continuing the current working status.
[0092] If the value of W is low (for example, less than a preset threshold), it may indicate that the crew member is in a bad emotional state or is overloaded with work, and the system needs to provide adjustment suggestions.
[0093] Suggestions for adjusting emotional state and workload include:
[0094] 1. Mood-enhancing suggestions:
[0095] Relaxation Activities: Crew members are advised to take short breaks to engage in deep breathing, meditation or a short walk to improve their emotional state.
[0096] Social Interaction: Encourage relaxed communication with colleagues and increase positive social interactions.
[0097] 2. Workload adjustment:
[0098] Task reallocation: If L is too high, it is recommended to assign some tasks to other colleagues to reduce the workload.
[0099] Task priority adjustment: It is recommended that crew members prioritize important tasks, postpone non-urgent work, and arrange working hours reasonably.
[0100] Embodiment 3
[0101] The present invention also proposes an emotion recognition and feedback system based on artificial intelligence, which includes:
[0102] A facial data collection module, which is used to collect facial expression data of crew members;
[0103] A voice data collection module, which is used to collect the voice data of the crew;
[0104] The facial emotion index calculation module is used to obtain the facial emotion index E based on the facial expression data. f ;
[0105] The voice emotion index calculation module is used to obtain the voice emotion index E based on the voice data. v ;
[0106] A comprehensive emotion index calculation module, which is used to obtain a comprehensive emotion index E based on the facial emotion score and the voice emotion index;
[0107] A work efficiency calculation module is used to calculate the crew work efficiency based on the comprehensive emotion index E, wherein the crew work efficiency is calculated by the following formula:
[0108] Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, β is the adjustment coefficient;
[0109] The suggestion making module is used to provide corresponding suggestions based on the work efficiency calculation results.
[0110] Embodiment 4:
[0111] An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.
[0112] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0114] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (AN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0116] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.
[0117] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.
Claims
1. An emotion recognition and feedback method based on artificial intelligence, characterized in that: The following steps are involved: Step S101, collecting facial expression data and voice data of crew members; Step S103: Obtain facial emotion index E based on facial expression data f ; Step S105: Obtain the voice emotion index E based on the voice data v ; Step S107, obtaining a comprehensive emotion index E based on the facial emotion score and the voice emotion index; Step S109: Calculate the crew work efficiency based on the comprehensive emotion index E and provide corresponding suggestions, wherein the crew work efficiency is calculated by the following formula: Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, and β is the adjustment coefficient.
2. The method according to claim 1, characterized in that: The step S103 includes: obtaining the facial emotion index E by using the following formula: f , Where t is the current time, T i is the acquisition time of the i-th frame image, δ t is the time standard deviation, I i is the i-th frame image, η i is the weight of the i-th image, Relu(x)=max(0,x) is the activation function, N f is the total number of frames of facial images.
3. The method according to claim 2, characterized in that: The δ t The time standard deviation is obtained by the following steps: When capturing each frame of the image, record the corresponding timestamp; Assume there are M timestamps T1, T2, ...T M , which can be expressed as T i =t i -t0, where t i is the acquisition time of the i-th frame image, and t0 is the starting time; The mean μ of the timestamp t The following formula is used to express The time standard deviation 4. The method according to claim 1, characterized in that: The step S105 includes: obtaining the speech emotion index E by using the following formula: v , Among them, P j is the weight of the jth speech segment, Q j is the jth speech segment, T j is the time feature, φ is the sentiment classification function, F(Q j ) is the Fourier transform of the jth speech segment, |F(Q j )| 2 is the energy spectrum in the frequency domain, F max is the maximum frequency.
5. The method according to claim 4, characterized in that: The sentiment classification function is represented by a support vector machine model.
6. The method according to claim 5, characterized in that The sentiment classification function is expressed as follows: φ(Q j )=sign(W Q T ·F(Q j )+b), where φ(Q j ) represents the sentiment classification result of the jth speech segment, W Q is the weight vector, representing the normal vector of the hyperplane, W Q T Represents the weight vector W Q The transpose of F(Q j ) is the feature vector extracted from the jth speech segment, b is the bias term, and sign(·) is the sign function.
7. The method according to claim 6, characterized in that: In step S107, the following formula is used to obtain the comprehensive emotion index E: Among them, α and β are E f , E v The weight coefficient of , N is the normalization coefficient.
8. The method according to claim 7, characterized in that: The maximum value normalization is used to obtain N, N = max(E f , E v ).
9. The method according to claim 1, characterized in that: In step S109, the workload L is obtained by using the following formula: L=w1·T c +w2·T p +w3·T N , where T c Score the task complexity, T p Rating time pressure, T N is the environmental noise score, w1, w2, w3 are the weight coefficients of each factor.
10. An emotion recognition and feedback system based on artificial intelligence, comprising: A facial data collection module, which is used to collect facial expression data of crew members; A voice data collection module, which is used to collect the voice data of the crew; The facial emotion index calculation module is used to obtain the facial emotion index E based on the facial expression data. f ; The voice emotion index calculation module is used to obtain the voice emotion index E based on the voice data. v ; A comprehensive emotion index calculation module, which is used to obtain a comprehensive emotion index E based on the facial emotion score and the voice emotion index; A work efficiency calculation module is used to calculate the crew work efficiency based on the comprehensive emotion index E, wherein the crew work efficiency is calculated by the following formula: Among them, W is the work efficiency index, E is the comprehensive emotional state index, L is the current workload, δ is the influence coefficient of emotion on work efficiency, μ L is the mean value of workload, σ L is the standard deviation of workload, β is the adjustment coefficient; The suggestion making module is used to provide corresponding suggestions based on the work efficiency calculation results.