Passenger stress emotion recognition and hierarchical treatment method and system based on dual-mode

Abstract

The application discloses a passenger stress emotion recognition and hierarchical treatment method and system based on a dual mode, and belongs to the technical field of intelligent customer service. The method part comprises the following steps: S1, dual mode data acquisition; S2, mode preprocessing and quality evaluation; S3, civil aviation scene recognition; S4, dual mode feature extraction and deep fusion; S5, emotion recognition and grade determination; S6, emotion trend prediction; S7, dynamic treatment strategy generation; S8, dual channel strategy execution; S9, effect evaluation and model iteration. The system comprises a text-voice dual mode data acquisition and preprocessing module, a civil aviation scene dual mode emotion recognition engine module and a hierarchical dynamic treatment decision module. The application solves the problems of single mode information loss and airport noise robustness, and the problems of too passive intervention, inaccurate passenger emotion calming time node grasping and insufficient service investment precision.

Description

Technical Field

[0001] This invention belongs to the field of intelligent customer service technology, specifically relating to a method and system for identifying and classifying passenger stress emotions based on a dual-modality approach. Background Technology

[0002] With the rapid development of my country's civil aviation industry, air transport has become the core mode of travel for medium and long-distance passengers, and passenger traffic has remained at a high level. Against the backdrop of large-scale and high-density air transport, passenger stress incidents caused by factors unique to civil aviation, such as flight delays / cancellations, lost / damaged baggage, overbooking, strict security procedures, gate changes, long-haul flight fatigue, and the enclosed environment at high altitudes, have shown an increasing trend year by year.

[0003] Emotional outbursts in the civil aviation sector not only cause widespread flight delays and huge economic losses, but also directly threaten aviation safety and cause adverse social impacts. Therefore, the ability of civil aviation customer service to accurately assess and identify passengers' stressful emotions and take appropriate measures is crucial for maintaining the smooth operation of airlines and airports.

[0004] Currently, civil aviation customer service primarily identifies passengers' stressful emotions from the following aspects: Based on unimodal text recognition, the system determines the passenger's inquiry intent from the plain text information of the passenger's inquiry and identifies whether the passenger is experiencing stress. Based on single-modal speech recognition, key information is extracted from the interactive voice dialogue between passengers and customer service, and stress emotions are identified by combining the passenger's tone of voice. Text combined with speech serial recognition: first, speech recognition is performed to obtain text, and then sentiment analysis is performed on the text; Current passenger stress emotion recognition technologies still have the following technical shortcomings: 1) Text-speech modal separation leads to loss of emotional information. Existing systems mostly adopt a serial process of "speech-text-semantic understanding". After the speech signal is converted into text by ASR, the original acoustic features such as speech rate, tone, and pause patterns are discarded. Studies have shown that acoustic features contribute up to 55% in the expression of anger, while text semantics only account for 45%. Existing technologies have not established a dual-modal parallel fusion mechanism, resulting in the loss of key emotional cues. 2) Speech emotion recognition fails in complex acoustic environments at airports. Airport scenarios are characterized by high-intensity environmental noise. Existing voice emotion recognition models experience a sharp drop in accuracy to below 40% when the signal-to-noise ratio is below 10dB, which fails to meet the needs of on-site services. 3) Lack of a contextualized emotion labeling system for civil aviation General emotion recognition only distinguishes between "positive / negative / neutral" emotions and does not establish fine-grained emotion labels for civil aviation business scenarios, such as "flight delay anxiety", "overbooking and refusal of boarding anger", and "baggage damage claim". This results in the system being unable to accurately match professional civil aviation service strategies, and the response content is mismatched with the actual needs of passengers. 4) Passive responses lead to escalation of emotions. The existing system can only respond after passengers make an inquiry, and cannot proactively intervene in the early stages of passenger emotions. Data shows that in flight delay scenarios, if passengers do not receive effective information within 15 minutes after the delay notice is issued, the probability of their emotions escalating increases by 65%, and the cost of customer service intervention at this time increases significantly. 5) Rigidity of the emotion-coping strategy mapping The existing system's emotional handling strategies are simple, such as "transfer to human agent" or "continue automatic response," without establishing a dynamic matching mechanism between emotional level and service resources; this results in mildly anxious passengers being over-served, while severely angry passengers are not handled in a timely manner. 6) Lack of bimodal complementarity mechanism When the speech modality fails due to environmental noise, the system cannot automatically increase the weight of the text modality; when the passenger inputs very short text, the system cannot use speech features to help understand the true emotions; and there is a lack of dynamic complementarity and confidence calibration mechanisms between modalities. Summary of the Invention

[0005] This invention provides a method and system for passenger stress emotion recognition and graded handling based on bimodality. By deeply integrating text semantic features and speech acoustic features, it constructs a civil aviation scenario-based emotion recognition model and establishes a dynamic matching mechanism between emotion level and service resources, realizing a service mode shift from passive response to proactive intervention. This invention solves the problems of information loss and airport noise robustness in current single-modality systems, as well as overly passive intervention, which leads to inaccurate timing of passenger emotion reassurance and insufficient precision in service input.

[0006] To achieve the above-mentioned technical objectives, the present invention is implemented through the following technical solution: A bimodal approach to passenger stress identification and tiered management includes: S1: Dual-modal data acquisition, simultaneously activating the voice and text channels to collect passengers' interactive text data and voice data; S2: Modal preprocessing and quality assessment, including noise reduction, ASR recognition, and acoustic feature extraction of speech data; Entity recognition is performed on text data, and syntactic analysis is performed based on words and sentences. Evaluate the quality of both modes and label the low-quality mode; S3: Civil aviation scene recognition. Based on text entities and voice keywords, determine the current business scene, such as delay / baggage / security check, etc., construct a civil aviation business scene knowledge graph, load the corresponding scene knowledge graph based on the currently determined business scene, and obtain the graph relationship of emotion association. S4: Dual-modal feature extraction and deep fusion, utilizing text encoder and speech encoder to extract features in parallel, performing dual-modal feature fusion through cross-attention mechanism, and dynamically adjusting weights based on quality assessment; S5: Emotion recognition and level determination, outputting fine-grained emotion labels including basic emotion level, civil aviation scenario emotion type, confidence level, dominant modality, and key evidence; S6: Emotion trend prediction, predicting the direction of emotion evolution based on temporal memory, and determining whether intervention is needed based on the emotion development trend; S7: Dynamic handling strategy generation, query the sentiment-resource matching matrix, and generate a targeted and personalized handling plan based on passenger value, scenario urgency, system load, and channel type; S8: Dual-channel strategy execution, executing the strategy through both voice and text channels; S9: Effectiveness evaluation and model iteration. Real-time monitoring of passenger sentiment changes after the implementation of the response plan, evaluation of the effectiveness of the response plan, recording of data in the plan and effect data, and using them to optimize the model. If necessary, upgrade the response level and return to step S7.

[0007] Preferably, the voice channel includes at least: a telephone customer service channel (PSTN / VoIP) and an airport on-site service terminal microphone; The text channel includes at least: an interactive terminal interface input box and an ASR (Automatic Speech Recognition) system.

[0008] Preferably, the speech data denoising is performed using a denoising model, based on a deep learning-based speech enhancement algorithm, and a dedicated denoising model is trained for the unique noises in airports, including at least broadcast sounds, crowd sounds, and baggage cart sounds. This invention provides two denoising model structures, Conv-TasNet or DCCRN, and the denoising model is trained based on these two model structures. The acoustic feature extraction is divided into three levels: Low-level features: speech rate (words / second), average fundamental frequency (F0), fundamental frequency variation range, short-time energy, zero-crossing rate; High-level features: Deep acoustic representations are extracted based on Wav2Vec 2.0 or HuberT pre-trained models; Prosodic features: number / duration of pauses, detection of sighs, detection of laughter, and placement of accents; Entity recognition in said text data includes at least: flight number, date, cabin class, three-letter airport code, and service type; service types include, for example: meal / baggage / seat / upgrade; Said syntactic analysis analyzes the causal relationship and hypothetical relationship in text sentences, for example: "claim compensation because the flight is delayed", "I would not have missed the meeting if you had not canceled the flight", etc.; For sentence fragmentation caused by emotional agitation, a large language model is used for semantic completion; the real intention representation of ironic sentences is identified; for example: "What on earth do you want! My luggage!" is completed as "What on earth are you going to do! My luggage is lost, can you get it back!"; while the real intention of "Great, I've been sitting on the plane for another three hours" is expressed as the passenger's dissatisfaction with the long delay; Said bimodal quality assessment method is: Establish a unified timestamp mechanism to align voice segments and corresponding text sentences according to time windows; said time window is set to 3 to 5 seconds; Real-time assessment of modal quality: Voice quality indicators: signal-to-noise ratio (SNR), automatic speech recognition confidence, proportion of effective speech duration; Text quality indicators: text length, completeness of entity recognition, grammatical completeness; Modal missing early warning: when the voice SNR is less than 5dB or the ASR confidence is less than 0.6, the voice modality is marked as low quality, and the weight of the text modality is triggered to increase; when the text length is less than 5 words and there is no semantic entity, the text modality is marked as low quality, and judgment depends on the voice modality.

[0009] Preferably, the construction process of said knowledge graph for civil aviation business scenarios is as follows: Scenario node definition: flight delay (1 to 2 hours / 2 to 4 hours / more than 4 hours), flight cancellation, overbooking and denial of boarding, baggage delay, damaged / lost baggage, security check conflict (items confiscated / personal inspection dispute), gate change, insufficient transit time (<MCT), cabin downgrade, flight diversion; Scenario-emotion association edges: each scenario is associated with a specific emotion distribution probability; for example, "flight delay of 2 to 4 hours" is associated with "anxiety (50%)", "anger (30%)", "helplessness (20%)"; Emotion-demand association edges: the severity of stress emotion maps to the probability of a passenger's demand; for example: "delay-anxiety type" is associated with "accurate departure time information (60%)", "rebooking options (30%)", "compensation standard (10%)"; "delay-anger type" is associated with "cash compensation (50%)", "complaint channel (50%)", "express rebooking (50%)".

[0010] Preferably, the dual-modal feature fusion model adopts a Cross-Modal Transformer architecture and designs a text-speech cross-attention mechanism; The text encoder extracts text semantic vectors based on BERT or a finely tuned large model in the civil aviation field. The speech encoder is based on Wav2Vec 2.0 or Whisper encoder to extract speech acoustic vectors; The bimodal fusion layer in the bimodal fusion model implements bimodal fusion according to the following process: Early fusion: Concatenating speech vectors and text vectors at the feature layer; Cross-attention: Design a bidirectional attention mechanism between text and speech, and speech and text, to capture cross-modal associations such as "angry words + high speech rate" and "anxious words + many pauses"; Adaptive weighting: The fusion weights of text and speech are dynamically adjusted based on the modal quality assessment results; for example, when the speech signal-to-noise ratio (SNR) is low, the text weight is increased to 0.7. Scene adapter: Loads the corresponding scene fusion parameters based on the identified civil aviation scene.

[0011] Preferably, the basic emotional level includes five levels, in the following order: calm, mild anxiety, obvious anxiety, anger, and severe anger; The emotions in civil aviation scenarios are divided into ten categories: delay anxiety, delay anger, cancellation helplessness, cancellation anger, baggage anxiety, baggage claim, security check conflict, transit panic, overbooking anger, and diversion anxiety. The confidence level is the speech confidence score obtained from the speech quality index assessment; The dominant modality is based on the contribution of the speech modality to indicate whether there are any abnormalities in the passenger's tone of voice; The key evidence consists of keywords that immediately reflect a passenger's emotional fluctuations, such as "complaint". 。

[0012] Preferably, the process of predicting the direction of emotion evolution based on temporal memory is as follows: Emotional state memory: Establish a sequence of emotional states for each passenger during the interaction process and record the changes in emotional level in each round of dialogue; The trend prediction model, using BiLSTM or temporal Transformer, predicts the evolutionary direction of the next round of emotional escalation / maintenance / ease based on the emotional trajectory of 3-5 rounds of historical dialogue. Pre-intervention trigger: When it is predicted that the emotion will escalate within 30 seconds, such as from anxiety to anger, and the confidence level is >0.8, the active intervention mechanism is triggered.

[0013] Preferably, the emotion-resource matching matrix is ​​established according to the mapping rules between emotion level and treatment strategy, and supports dynamic threshold adjustment; The threshold adjustment is based on the following considerations: Passenger value classification: Thresholds are adjusted based on membership level (standard / silver / gold / platinum), ticket price, and historical complaint records; high-value passengers (platinum card) will be on standby for human assistance when their emotional level is L2, while ordinary passengers will be on standby at L3. Scenario urgency adjustment: For emergency scenarios within 2 hours before flight departure, all thresholds are reduced by one level, such as L2 being treated as L3; System load adaptation: During peak customer service periods, such as when large-scale delays occur, the automatic handling rate increases by 20%, but L5 still requires manual intervention; Channel characteristic adaptation: The telephone channel focuses on voice emotion recognition, while the APP channel focuses on text semantic analysis. Different weight configurations are preloaded according to the channel type.

[0014] Preferably, the voice channel processing strategy is executed as follows: Adjust the parameters of the synthesized speech (TTS): reduce the speech rate by 20% and lower the pitch to appear calmer when anxious; insert pauses and avoid provocative words when angry; Play pre-recorded reassuring audio, such as "We completely understand how you feel"; Background noise reduction: Real-time noise reduction at the agent's end to ensure clear calls when a human operator is contacting the agent; The text channel handling strategy is executed as follows: Fonts and colors for interactive terminal interfaces: Use blue calming tones when anxious, and avoid red stimuli when angry; Information structure: Bold key information, shorten paragraphs, and use emojis to ease the atmosphere; Interaction method: Provides shortcut buttons such as "one-click rescheduling" and "quick compensation" to reduce the burden of input; Set up a proactive service channel to execute the following handling strategies: Trigger proactive flight status notifications, which include delay information, reasons, and estimated time. Automatically send a link to rebooking options, which pre-fills passenger information to avoid requiring passengers to fill it out again and exacerbating their emotional state. The compensation voucher will be automatically issued to your account and you will be notified via SMS.

[0015] Preferably, in the effect evaluation, after taking action against the passenger, the emotional changes in the subsequent 1-2 rounds of dialogue are continuously monitored; if the emotion is relieved (the level decreases), the successful strategy is recorded; if the emotion is escalated, the handling level is automatically increased and the strategy is switched, and the passenger is marked as a training sample. The data on "identification-processing-effect" is summarized and fine-tuned using a 24-hour update cycle, and the dual-modal fusion model and strategy matrix are adjusted accordingly.

[0016] Another objective of this invention is to provide a bimodal passenger stress emotion recognition and tiered management system, comprising: The text-speech dual-modal data acquisition and preprocessing module includes a speech modal acquisition and airport environment adaptive enhancement unit, which is used to remove noise interference data in the extracted raw speech data and extract acoustic features. The text modality acquisition and civil aviation semantic parsing unit is used to identify entities in the civil aviation field from text data; perform syntactic analysis, complete fragmented sentences, and clarify the true semantics expressed. The dual-modal spatiotemporal alignment and quality assessment unit evaluates the quality of both speech and text modalities; based on the assessment results, it provides early warnings for cases where there are missing elements in the modalities. The civil aviation scenario-based dual-modal emotion recognition engine module includes a civil aviation stress scenario knowledge graph construction unit, which establishes scenario-emotion association edges and emotion-demand association edges based on scenario node definitions; and constructs a civil aviation stress scenario knowledge graph. A text-speech bimodal deep fusion model is proposed, which adopts a Cross-Modal Transformer architecture and designs a text-speech cross-attention mechanism to perform bimodal fusion of speech and text. The fine-grained emotion tag output unit sets up a basic emotion layer and emotion types for civil aviation scenarios, and outputs fine-grained emotion tags for emotion level, scene emotion type, confidence score, dominant modality indication, and key evidence. The emotion timing tracking and trend prediction unit establishes a memory sequence of passenger emotions and uses BiLSTM or temporal Transformer to predict the subsequent emotional evolution trend of passengers based on 3 to 5 rounds of dialogue; based on the emotional evolution trend, it determines whether to trigger a pre-intervention mechanism. The tiered dynamic response decision-making module constructs a dynamic matching matrix of emotions and service resources to establish mapping rules between five levels of emotions and response strategies, and supports dynamic threshold adjustment. Based on four dimensions of reference—passenger value classification, scenario urgency adjustment, system load self-adaptation, and channel characteristic adaptation—the trigger threshold is dynamically adjusted. The dual-channel collaborative handling and execution unit adopts corresponding reassurance and execution strategies based on the different media characteristics of the voice and text channels; and a proactive service channel is set up to proactively provide passengers with necessary information push and convenient operations; The real-time feedback and closed-loop optimization unit for handling effects is used to monitor the effects after the strategy is implemented and to determine whether the reassurance strategy needs to be deepened based on the monitoring results; the "identification-handling-effect" data is summarized daily to fine-tune the dual-modal fusion model and strategy matrix.

[0017] The beneficial effects of this invention are: Dual-modal fusion significantly improves the accuracy of emotion recognition, avoiding issues such as information loss in single-modal data and robustness to airport noise; it also improves the integrity and accuracy of modal data, thereby enhancing the precision of emotion recognition. By combining unique emotion tags with civil aviation business scenarios, a knowledge graph of civil aviation business scenarios is constructed to achieve accurate mapping between business scenarios and emotions and demands; Establish a dynamic matching matrix between emotions and service resources to achieve dynamic matching and pre-intervention mechanisms between emotion levels and service resources, thereby transforming from passive response to proactive service. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system framework for implementing the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 S1: Dual-modal data acquisition, simultaneously activating the voice and text channels to collect passengers' interactive text data and voice data; The voice channels used for voice data collection include: telephone customer service channels (PSTN / VoIP) and microphones at airport on-site service terminals; Text channels for collecting text data include: text input boxes on various interactive terminal interfaces such as APP, WeChat, and SMS; and speech recognition systems (ASR) that can be used to convert speech into text data; S2: Modal preprocessing and quality assessment, denoising the speech data, using a deep learning-based speech enhancement algorithm, and training a dedicated denoising model for specific noises such as airport announcements, crowd noise, and baggage cart noise; this embodiment provides two structures that can be used as denoising models: Conv-TasNet or DCCRN. ASR recognition maps speech signals into text sequences through feature extraction, pattern matching, and model training. Acoustic feature extraction: This embodiment sets up a three-level acoustic feature extraction process, namely: Low-level features: speech rate (words / second), average fundamental frequency (F0), fundamental frequency variation range, short-time energy, zero-crossing rate; High-level features: Deep acoustic representations are extracted based on Wav2Vec 2.0 or HuberT pre-trained models; Prosodic features: number / duration of pauses, detection of sighs, detection of laughter, and placement of accents; Entity recognition is performed on text data. The civil aviation entities identified include: flight number, date, cabin class, airport code, and service type, where service type refers to meals / baggage / seat / upgrade. Based on word and sentence grammatical analysis, this embodiment analyzes the causal and hypothetical relationships in the text sentences, such as: "I demand compensation because of the flight delay", "I would not have missed the meeting if you hadn't canceled my flight"; these are sentences with causal meaning. To address the issue of fragmented speech or text sentences caused by passengers' emotional distress, a large language model is used for semantic completion; for example, completing "What do you want from me! My luggage!" into "What are you going to do! My luggage is lost, can you find it!"; In addition, for some ironic statements, identify their true meaning; for example, "Great, another three hours on the plane" may actually mean that the passenger is already dissatisfied with the long delay. Evaluate the quality of both modes and label the low-quality mode; Establish a unified timestamp mechanism to align audio segments with corresponding text statements according to a time window; the time window is set to 3-5 seconds. The real-time modal quality assessment method is as follows: The metrics for evaluating speech quality are: signal-to-noise ratio (SNR), speech recognition confidence, and percentage of effective speech duration. The metrics for evaluating text quality are: text length, entity recognition completeness, and syntactic completeness. Evaluate the above indicators: when the speech signal-to-noise ratio (SNR) < 5dB or the Automatic Speech Recognition (ASR) confidence < 0.6, mark the speech modality as low-quality and trigger the weight increase of the text modality; when the text length is less than 5 words and there is no semantic entity, mark the text modality as low-quality, and the judgment shall depend on the speech modality; S3: Civil aviation scenario recognition: determine the current service scenario based on text entities and speech keywords, such as flight delay, baggage, security check, etc., and construct a knowledge graph for civil aviation service scenarios. The process is as follows: Scenario node definition: flight delay (1~2 hours / 2~4 hours / more than 4 hours), flight cancellation, overbooking and denial of boarding, baggage delay, baggage damage / loss, security check conflict (items confiscated / personal inspection dispute), gate change, insufficient transfer time (<MCT), cabin downgrade, flight diversion; Scene-emotion association edges: each scenario is associated with a specific emotion distribution probability; for example, "flight delay of 2~4 hours" is associated with "anxiety (50%)", "anger (30%)", and "helplessness (20%)"; Emotion-demand association edges: the severity of stress emotion maps to the probability of passengers' demands; for example, "delay-anxiety type" is associated with "accurate departure time information (60%)", "rebooking options (30%)", and "compensation standard (10%)"; "delay-anger type" is associated with "cash compensation (50%)", "complaint channel (50%)", and "express rebooking (50%)"; Load the corresponding scenario knowledge graph based on the currently determined service scenario to obtain the emotion-related graph relationships; S4: Bi-modal feature extraction and deep fusion: use a text encoder and a speech encoder to extract features in parallel, perform bi-modal feature fusion through a cross-attention mechanism, and dynamically adjust weights according to quality assessment; The bi-modal feature fusion model adopts the Cross-Modal Transformer architecture, and a text-speech cross-attention mechanism is designed; The text encoder extracts text semantic vectors based on BERT or a large model fine-tuned for the civil aviation field; The speech encoder is based on Wav2Vec 2.0 or Whisper encoder to extract speech acoustic vectors; The bi-modal fusion layer in the above bi-modal feature fusion model implements bi-modal fusion according to the following process: Early fusion: concatenate speech vectors and text vectors at the feature layer; Cross-attention: a bidirectional attention mechanism of text-to-speech and speech-to-text is designed to capture cross-modal associations such as "angry words + fast speech rate" and "anxious words + multiple pauses"; Adaptive weighting: The fusion weights of text and speech are dynamically adjusted based on the modal quality assessment results; for example, when the speech signal-to-noise ratio (SNR) is low, the text weight is increased to 0.7. Scene adapter: Loads the corresponding scene fusion parameters based on the identified civil aviation scene; S5: Emotion recognition and level determination, outputting fine-grained emotion labels including basic emotion level, civil aviation scenario emotion type, confidence level, dominant modality, and key evidence; This embodiment sets five basic emotional levels, in the following order: calm, mild anxiety, obvious anxiety, anger, and severe anger; The emotions in civil aviation scenarios are divided into ten categories: delay anxiety, delay anger, cancellation helplessness, cancellation anger, baggage anxiety, baggage claim, security check conflict, transit panic, overbooking anger, and diversion anxiety. Confidence score is the speech confidence score obtained from the speech quality index assessment. The dominant modality is based on the contribution of the speech modality to indicate whether there are any abnormalities in the passenger's tone of voice; Key evidence consists of identified keywords that immediately reflect passenger emotional fluctuations, such as "complaint"; S6: Emotion trend prediction, based on temporal memory to predict the direction of emotion evolution, the specific process is as follows: Emotional state memory: Establish a sequence of emotional states for each passenger during the interaction process and record the changes in emotional level in each round of dialogue; The trend prediction model, using BiLSTM or temporal Transformer, predicts the evolutionary direction of the next round of emotional escalation / maintenance / ease based on the emotional trajectory of 3-5 rounds of historical dialogue. When it is predicted that the emotion will escalate within 30 seconds, such as from anxiety to anger, and the confidence level is >0.8, the proactive intervention mechanism is triggered, and customer service will take the initiative to handle the necessary follow-up work and reassurance measures for the passenger. S7: Dynamic handling strategy generation, query the sentiment-resource matching matrix, and generate a targeted and personalized handling plan based on passenger value, scenario urgency, system load, and channel type; The aforementioned emotion-resource matching matrix is ​​established according to the mapping rules between emotion level and treatment strategy, and supports dynamic threshold adjustment; Threshold adjustments are implemented based on the following dimensions: Passenger value classification: Thresholds are adjusted based on membership level (standard / silver / gold / platinum), ticket price, and historical complaint records; high-value passengers (platinum card) will be on standby for human assistance when their emotional level is L2, while ordinary passengers will be on standby at L3. Scenario urgency adjustment: For emergency scenarios within 2 hours before flight departure, all thresholds are reduced by one level, such as L2 being treated as L3; System load adaptation: During peak customer service periods, such as when large-scale delays occur, the automatic handling rate increases by 20%, but L5 still requires manual intervention; Channel feature adaptation: Telephone channels focus on voice emotion recognition, while APP channels focus on text semantic analysis. Different weight configurations are preloaded according to the channel type. S8: Dual-channel strategy execution, executing the strategy through both voice and text channels; The voice channel handling strategy is implemented as follows: Adjust the parameters of the synthesized speech (TTS): reduce the speech rate by 20% and lower the pitch to appear calmer when anxious; insert pauses and avoid provocative words when angry; Play pre-recorded reassuring audio, such as "We completely understand how you feel"; Background noise reduction: Real-time noise reduction at the agent's end to ensure clear calls when a human operator is contacting the agent; The text channel handling strategy is implemented as follows: Fonts and colors for interactive terminal interfaces: Use blue calming tones when anxious, and avoid red stimuli when angry; Information structure: Bold key information, shorten paragraphs, and use emojis to ease the atmosphere; Interaction method: Provides shortcut buttons such as "one-click rescheduling" and "quick compensation" to reduce the burden of input; In addition to the interactive data channel, this embodiment also sets up an active service channel to execute the following processing strategies: Trigger proactive flight status notifications, which include delay information, reasons, and estimated time. Automatically send a link to rebooking options, which pre-fills passenger information to avoid requiring passengers to fill it out again and exacerbating their emotional state. Compensation vouchers will be automatically issued to your account and you will be notified via SMS. S9: Effectiveness evaluation and model iteration. Real-time monitoring of passenger emotional changes after the implementation of the handling plan, and evaluation of the handling effect. The effectiveness evaluation method is as follows: after the handling plan is implemented, the emotional changes of passengers in the subsequent 1-2 rounds of dialogue are continuously monitored. If the emotional level decreases and is alleviated, it is recorded as a successful strategy; if the emotion escalates after the handling, the handling level is automatically increased and the handling strategy is switched, and it is marked as a training sample. Record the data and effect data in the plan, and use them to optimize the model. If necessary, upgrade the treatment level and return to step S7. According to the 24-hour update cycle, summarize the "identification-treatment-effect" data, fine-tune the dual-modal fusion model and strategy matrix, and realize closed-loop optimization of emotion recognition and treatment strategy execution.

[0022] Example 2 A bimodal passenger stress emotion recognition and tiered management system includes: The text-speech dual-modal data acquisition and preprocessing module includes a speech modal acquisition and airport environment adaptive enhancement unit, which is used to remove noise interference data in the extracted raw speech data and extract acoustic features. The text modality acquisition and civil aviation semantic parsing unit is used to identify entities in the civil aviation field from text data; perform syntactic analysis, complete fragmented sentences, and clarify the true semantics expressed. The dual-modal spatiotemporal alignment and quality assessment unit evaluates the quality of both speech and text modalities; based on the assessment results, it provides early warnings for cases where there are missing elements in the modalities. The civil aviation scenario-based dual-modal emotion recognition engine module includes a civil aviation stress scenario knowledge graph construction unit, which establishes scenario-emotion association edges and emotion-demand association edges based on scenario node definitions; and constructs a civil aviation stress scenario knowledge graph. A text-speech bimodal deep fusion model is proposed, which adopts a Cross-Modal Transformer architecture and designs a text-speech cross-attention mechanism to perform bimodal fusion of speech and text. The fine-grained emotion tag output unit sets up a basic emotion layer and emotion types for civil aviation scenarios, and outputs fine-grained emotion tags for emotion level, scene emotion type, confidence score, dominant modality indication, and key evidence. The emotion timing tracking and trend prediction unit establishes a memory sequence of passenger emotions and uses BiLSTM or temporal Transformer to predict the subsequent emotional evolution trend of passengers based on 3 to 5 rounds of dialogue; based on the emotional evolution trend, it determines whether to trigger a pre-intervention mechanism. The tiered dynamic response decision-making module constructs a dynamic matching matrix of emotions and service resources to establish mapping rules between five levels of emotions and response strategies, and supports dynamic threshold adjustment. Based on four dimensions of reference—passenger value classification, scenario urgency adjustment, system load self-adaptation, and channel characteristic adaptation—the trigger threshold is dynamically adjusted. The dual-channel collaborative handling and execution unit adopts corresponding reassurance and execution strategies based on the different media characteristics of the voice and text channels; and a proactive service channel is set up to proactively provide passengers with necessary information push and convenient operations; The real-time feedback and closed-loop optimization unit for handling effects is used to monitor the effects after the strategy is implemented and to determine whether the reassurance strategy needs to be deepened based on the monitoring results; the "identification-handling-effect" data is summarized daily to fine-tune the dual-modal fusion model and strategy matrix.

Claims

1. A method for identifying and classifying passenger stress emotions based on a dual-modal approach, characterized in that, include: S1: Dual-modal data acquisition, simultaneously activating the voice and text channels to collect passengers' interactive text data and voice data; S2: Modal preprocessing and quality assessment, including noise reduction, ASR recognition, and acoustic feature extraction of speech data; Entity recognition is performed on text data, and syntactic analysis is performed based on words and sentences. Evaluate the quality of both modes and label the low-quality mode; S3: Civil aviation scene recognition. Based on text entities and voice keywords, determine the current business scene, construct a civil aviation business scene knowledge graph, load the corresponding scene knowledge graph based on the currently determined business scene, and obtain the graph relationship of emotion association. S4: Dual-modal feature extraction and deep fusion, utilizing text encoder and speech encoder to extract features in parallel, performing dual-modal feature fusion through cross-attention mechanism, and dynamically adjusting weights based on quality assessment; S5: Emotion recognition and level determination, outputting fine-grained emotion labels including basic emotion level, civil aviation scenario emotion type, confidence level, dominant modality, and key evidence; S6: Emotion trend prediction, predicting the direction of emotion evolution based on temporal memory, and determining whether intervention is needed based on the emotion development trend; S7: Dynamic handling strategy generation, query the sentiment-resource matching matrix, and generate a targeted and personalized handling plan based on passenger value, scenario urgency, system load, and channel type; S8: Dual-channel strategy execution, executing the strategy through both voice and text channels; S9: Effectiveness evaluation and model iteration. Real-time monitoring of passenger sentiment changes after the implementation of the response plan, evaluation of the effectiveness of the response plan, recording of data in the plan and effect data, and using them to optimize the model. If necessary, upgrade the response level and return to step S7.

2. The method for passenger stress emotion recognition and graded treatment based on bimodality according to claim 1, characterized in that, The speech data denoising is processed using a denoising model, which is based on a deep learning speech enhancement algorithm and is trained with a dedicated denoising model for the unique noise in airports. The acoustic feature extraction is divided into three levels: Low-level features: Speech rate (words / second), average fundamental frequency (F0), fundamental frequency variation range, short-time energy, zero-crossing rate; High-level features: Deep acoustic representations are extracted based on Wav2Vec 2.0 or HuberT pre-trained models; Prosodic features: number / duration of pauses, detection of sighs, detection of laughter, and placement of accents; Entity recognition in the text data includes at least: flight number, date, cabin class, airport code, and service type; The syntactic analysis focuses on analyzing causal and hypothetical relationships within textual statements; To address the fragmentation of sentences generated during emotional excitement, a large language model is used for semantic completion; the true intention behind ironic statements is identified. The two-modal quality assessment method is as follows: Establish a unified timestamp mechanism to align audio segments with corresponding text statements according to a time window; the time window is set to 3-5 seconds. Real-time modal quality assessment: Voice quality metrics: SNR, speech recognition confidence, percentage of effective speech duration; Text quality metrics: text length, entity recognition completeness, and grammatical completeness; Modality missing warning: When the speech SNR < 5dB or ASR confidence < 0.6, the speech modality is marked as low quality, triggering text modality weighting; when the text length is < 5 characters and there are no semantic entities, the text modality is marked as low quality, relying on the speech modality for judgment.

3. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The process of constructing the knowledge graph for civil aviation business scenarios is as follows: Scenario node definition: flight delay, flight cancellation, overbooking and refusal of boarding, baggage delay, damaged / lost baggage, security check conflict, gate change, insufficient transfer time, cabin class downgrade, flight diversion; Scene-emotion association edges: Each scene is associated with a specific emotion distribution probability; Emotion-Demand Association Edge: The probability of a passenger making a demand, which is mapped to the severity of the stressful emotion.

4. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The dual-modal feature fusion model adopts a Cross-Modal Transformer architecture and designs a text-speech cross-attention mechanism; The text encoder extracts text semantic vectors based on BERT or a finely tuned large model in the civil aviation field. The speech encoder is based on Wav2Vec 2.0 or Whisper encoder to extract speech acoustic vectors; The bimodal fusion layer in the bimodal fusion model implements bimodal fusion according to the following process: Early fusion: Concatenating speech vectors and text vectors at the feature layer; Cross-attention: Design a bidirectional attention mechanism between text and speech, and speech and text, to capture cross-modal associations such as "angry words + high speech rate" and "anxious words + many pauses"; Adaptive weights: Dynamically adjust the fusion weights of text and speech based on the modal quality assessment results; Scene adapter: Loads the corresponding scene fusion parameters based on the identified civil aviation scene.

5. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The basic emotional level includes five levels, in order: calm, mild anxiety, obvious anxiety, anger, and severe anger; The emotions in civil aviation scenarios are divided into ten categories: delay anxiety, delay anger, cancellation helplessness, cancellation anger, baggage anxiety, baggage claim, security check conflict, transit panic, overbooking anger, and diversion anxiety. The confidence level is the speech confidence score obtained from the speech quality index assessment; The dominant modality is based on the contribution of the speech modality to indicate whether there are any abnormalities in the passenger's tone of voice; The key evidence is keywords that can immediately reflect passengers' emotional fluctuations.

6. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The process of predicting the direction of emotion evolution based on temporal memory is as follows: Emotional state memory: Establish a sequence of emotional states for each passenger during the interaction process and record the changes in emotional level in each round of dialogue; The trend prediction model, using BiLSTM or temporal Transformer, predicts the evolutionary direction of the next round of emotional escalation / maintenance / ease based on the emotional trajectory of 3-5 rounds of historical dialogue. Pre-intervention trigger: When it is predicted that the emotion will escalate within 30 seconds, the proactive intervention mechanism is triggered.

7. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The emotion-resource matching matrix is ​​established according to the mapping rules between emotion level and treatment strategy, and supports dynamic threshold adjustment; The threshold adjustment is based on the following considerations: Passenger value classification: Thresholds are adjusted based on membership level, ticket price, and historical complaint records; high-value passengers are triggered to be on standby manually when their emotional level is L2, while ordinary passengers are triggered at L3. Scenario urgency adjustment: For emergency scenarios within 2 hours before flight departure, all thresholds are reduced by one level; System load adaptation: During peak customer service periods, the automatic handling rate increases by 20%, but L5 still requires manual intervention; Channel characteristic adaptation: The telephone channel focuses on voice emotion recognition, while the APP channel focuses on text semantic analysis. Different weight configurations are preloaded according to the channel type.

8. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, The voice channel handling strategy is executed as follows: Adjust the parameters of the synthesized speech: reduce the speech rate by 20% and lower the pitch to appear calmer when anxious; insert pauses and avoid provocative words when angry; Play pre-recorded comforting audio; Background noise reduction: Real-time noise reduction at the agent's end to ensure clear calls when a human operator is contacting the agent; The text channel handling strategy is executed as follows: Fonts and colors for interactive terminal interfaces: Use blue calming tones when anxious, and avoid red stimuli when angry; Information structure: Bold key information, shorten paragraphs, and use emojis to ease the atmosphere; Interaction method: Provide shortcut buttons to reduce the burden of input; Set up a proactive service channel to execute the following handling strategies: Trigger proactive flight status notifications, which include delay information, reasons, and estimated time. Automatically send a link to rebooking options, where passenger information is pre-filled; The compensation voucher will be automatically issued to your account and you will be notified via SMS.

9. The method for passenger stress emotion recognition and graded handling based on bimodality according to claim 1, characterized in that, In the effectiveness evaluation, after taking action against the passenger, the emotional changes in the subsequent 1-2 rounds of dialogue should be continuously monitored; if the emotions are relieved, the successful strategy should be recorded. If the emotion escalates, the response level will be automatically increased and the strategy will be switched, while the sample will be marked as a training sample. The data on "identification-processing-effect" is summarized and fine-tuned using a 24-hour update cycle, and the dual-modal fusion model and strategy matrix are adjusted accordingly.

10. A passenger stress emotion recognition and hierarchical handling system based on dual-modality, characterized in that, include: The text-speech dual-modal data acquisition and preprocessing module includes a speech modal acquisition and airport environment adaptive enhancement unit, which is used to remove noise interference data in the extracted raw speech data and extract acoustic features. The text modality acquisition and civil aviation semantic parsing unit is used to identify entities in the civil aviation field from text data; perform syntactic analysis, complete fragmented sentences, and clarify the true semantics expressed. The dual-modal spatiotemporal alignment and quality assessment unit evaluates the quality of both speech and text modalities; based on the assessment results, it provides early warnings for cases where there are missing elements in the modalities. The civil aviation scenario-based dual-modal emotion recognition engine module includes a civil aviation stress scenario knowledge graph construction unit, which establishes scenario-emotion association edges and emotion-demand association edges based on scenario node definitions; and constructs a civil aviation stress scenario knowledge graph. A text-speech bimodal deep fusion model is proposed, which adopts a Cross-Modal Transformer architecture and designs a text-speech cross-attention mechanism to perform bimodal fusion of speech and text. The fine-grained emotion tag output unit sets up a basic emotion layer and emotion types for civil aviation scenarios, and outputs fine-grained emotion tags for emotion level, scene emotion type, confidence score, dominant modality indication, and key evidence. The emotion timing tracking and trend prediction unit establishes a memory sequence of passenger emotions and uses BiLSTM or temporal Transformer to predict the subsequent emotional evolution trend of passengers based on 3 to 5 rounds of dialogue; based on the emotional evolution trend, it determines whether to trigger a pre-intervention mechanism. The tiered dynamic response decision-making module constructs a dynamic matching matrix of emotions and service resources to establish mapping rules between five levels of emotions and response strategies, and supports dynamic threshold adjustment. Based on four dimensions of reference—passenger value classification, scenario urgency adjustment, system load self-adaptation, and channel characteristic adaptation—the trigger threshold is dynamically adjusted. The dual-channel collaborative handling and execution unit adopts corresponding reassurance and execution strategies based on the different media characteristics of the voice and text channels; and a proactive service channel is set up to proactively provide passengers with necessary information push and convenient operations; The real-time feedback and closed-loop optimization unit for handling effects is used to monitor the effects after the strategy is implemented and to determine whether the reassurance strategy needs to be deepened based on the monitoring results; the "identification-handling-effect" data is summarized daily to fine-tune the dual-modal fusion model and strategy matrix.

Images