Multi-modal digital employee reception system and method based on edge calculation

Abstract

The invention relates to a multi-modal digital employee reception system and method based on edge calculation, and relates to the technical field of artificial intelligence, and the system comprises a multi-modal data collection module, an edge calculation module, a data fusion and decision module, a trigger activation module, an emotion recognition module, an interaction response module and a behavior prediction module. The method comprises the following steps: acquiring voice, image and action information of a user through a high-resolution camera, a microphone array and an infrared sensor; locally processing data by using an edge calculation module, and extracting voice, vision and thermal imaging features; a Transformer model is dynamically optimized through Test Time Training (TTT), multi-modal data are dynamically fused in combination with a multi-head attention mechanism, a comprehensive decision vector is generated, the system can accurately judge the emotional state of a user, user behaviors are predicted based on a fused Token sequence and historical records, personalized services are actively provided, and by dynamically adjusting a trigger threshold value and a weight distribution mechanism, the emotion state of the user can be accurately judged. The false triggering rate and the missed triggering rate are reduced, and the stability and the robustness of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to a multimodal digital employee reception system and method based on edge computing, and in particular to a multimodal digital employee reception system and method based on edge computing applied in the field of artificial intelligence technology. Background Art

[0002] With the rapid development of artificial intelligence technology, digital human technology has gradually been applied to scenarios such as smart reception and online customer service. This type of technology provides services to users through multimodal interaction of voice, text and images. However, existing technologies still have some shortcomings in data fusion, real-time response and situational adaptability, which affects user experience and application effects.

[0003] Chinese invention patent CN117828065B discloses a digital human customer service method, system, device and storage medium, which realizes intelligent customer service function by processing user input (voice, text, image) and generating personalized answers. Although the patent provides customized service capabilities, it only processes single-modal data in terms of data fusion and lacks dynamic optimization strategies for complex multi-modal interaction scenarios. In addition, the technology relies on cloud computing, which has response delays and user privacy and security risks.

[0004] Chinese invention patent CN118485447B discloses a customer service processing method for a heating digital human, which improves user satisfaction with problem solving through question-answer matching accuracy and emotion recognition. Although the patent effectively combines emotion recognition and problem handling processes, its functions are limited to specific heating scenarios and it is difficult to meet the needs of other complex interactive scenarios. In addition, emotion recognition mainly relies on voice signals and lacks comprehensive analysis of multimodal data, which may lead to inaccurate emotion judgments.

[0005] The above design realizes the basic functions of digital human customer service through single-mode or simple multi-modal analysis, but it still has the following limitations: Insufficient data fusion capability: The existing technology lacks comprehensive fusion and dynamic weight adjustment of multi-modal data (voice, image, action, body temperature, etc.), and it is difficult to cope with complex environments and the diversity of user behaviors. Weak real-time and privacy protection: Reliance on cloud computing to process data leads to response delays, and there is a risk of user privacy leakage during data transmission. Insufficient adaptability and robustness: In environments with low light, high noise or dynamic occlusion, the system performance may drop significantly, affecting the interaction effect. Application scenario limitations: Existing technologies mostly focus on a single scenario (such as heating customer service), lack versatility and scalability, and are difficult to meet diverse smart reception needs. Summary of the invention

[0006] To solve the above problems, the present invention provides a multimodal digital employee reception system and method based on edge computing, including:

[0007] The multimodal data collection module is used to collect the user's voice, image and motion information, including:

[0008] A high-resolution camera to capture image information of the user and the surrounding environment;

[0009] Microphone array, used to collect user's voice signal;

[0010] Infrared sensor, used to sense the user's presence, movement trajectory and body temperature;

[0011] Edge computing module, used for local processing of multimodal data, including:

[0012] A speech processing unit, used to extract semantic features and voiceprint information from speech signals;

[0013] An image processing unit, used to recognize the user's facial expressions, movement trajectories, and thermal imaging data;

[0014] Infrared data processing unit, used to process infrared signals to detect user's body temperature and movement trajectory;

[0015] The data fusion and decision module dynamically fuses and weights multimodal data through Test Time Training (TTT) and Transformer architecture, including:

[0016] Test Time Training (TTT) mechanism:

[0017] Dynamically fine-tune the parameters of the Transformer model based on real-time collected voice, image, and infrared data; dynamically adjust the feature extraction weights of each modal data in response to environmental changes (such as light intensity, background noise, or changes in user voice tone) to improve the real-time adaptability of the model;

[0018] Transformer Architecture:

[0019] The multimodal data (speech features, facial expressions, motion trajectories, thermal imaging data) are divided into multiple token sequences according to time periods. The multi-head attention mechanism is used to focus on the features of different modal data. The first attention head: analyzes the semantics and voiceprint of the speech; the second attention head: analyzes facial expressions and motion trajectories; the third attention head: analyzes the temperature distribution of thermal imaging data.

[0020] Add time dimension information to each token through position encoding, and use a multi-layer self-attention network to integrate token sequences of multiple time periods;

[0021] Generate the final comprehensive decision vector for subsequent interactive response and behavior prediction;

[0022] The trigger activation module is used to determine the user's interaction intention based on multimodal data and trigger the system response;

[0023] Emotion recognition module, which is used to judge the user's emotional state by combining facial expressions, movement trajectories, voice features and thermal imaging data;

[0024] The interactive response module generates voice, visual or action interactive feedback based on the fused data;

[0025] The behavior prediction module is used to predict user behavior based on multimodal data and historical interaction records, and actively trigger interactions.

[0026] In the above-mentioned multimodal digital employee reception system and method based on edge computing, by integrating high-resolution cameras, microphone arrays and infrared sensors, the user's voice, image and motion information are collected in real time, Test Time Training (TTT) is used to dynamically optimize model parameters, and the multi-head attention mechanism of the Transformer architecture is combined to dynamically adjust the weights and deeply fuse the voice, image and infrared data to ensure efficient interaction capabilities in complex environments. Edge computing technology is introduced to complete the processing and decision-making of multimodal data locally, reduce dependence on the cloud, avoid response delays and protect user privacy. By dynamically adjusting the trigger threshold and modal data weight, the present invention can maintain system stability and interaction accuracy in weak light, high noise or occlusion conditions, comprehensively analyze the user's expression, voice intonation, motion trajectory and body temperature data, accurately judge the user's emotional state, predict the user's potential needs based on the fused Token sequence and historical behavior data, and actively provide personalized services, support multi-scenario applications, including smart reception, service robots, medical assistance and public services, and can flexibly expand sensor equipment to meet different needs.

[0027] As a further improvement of the present application, the trigger activation module includes:

[0028] Multimodal trigger mechanism: triggers system response based on one or a combination of the following conditions:

[0029] Detection of the user entering the camera's field of view and facing the camera;

[0030] The infrared sensor detects when the user approaches the designated interaction area or performs a specific action (such as waving);

[0031] The microphone array collects voice commands or voiceprint signals that meet preset characteristics;

[0032] Trigger judgment rules: Dynamically adjust the trigger judgment threshold by combining the matching degree of multi-modal data to reduce the possibility of false triggering;

[0033] Activation signal confirmation mechanism: After being triggered, the user's intention is confirmed through voice prompts or visual signals to avoid accidental activation of non-target objects.

[0034] As a further improvement of the present application, the emotion recognition module includes:

[0035] Expression feature analysis unit: captures the user's facial image through a high-resolution camera, extracts facial expression features using a deep learning model, and generates a preliminary emotional state;

[0036] Action trajectory analysis unit: monitors the user's action trajectory (such as gestures and head movements) through cameras and infrared sensors, and identifies the user's current emotional tendency based on historical action patterns;

[0037] Voice feature emotion judgment unit: judges the emotional information contained in the voice based on the tone, volume and voiceprint changes of the user's voice signal;

[0038] Thermal imaging emotion analysis unit: uses an infrared camera to obtain the user's facial thermal image, extracts temperature distribution features, and combines deep learning models to infer the user's emotional state;

[0039] Emotion fusion unit: Based on the Transformer architecture, the above multimodal emotion data is converted into a token sequence, and the weights of different modal data are dynamically allocated through the multi-head attention mechanism to generate the final emotion judgment result.

[0040] As a further improvement of this application, the behavior prediction module includes:

[0041] Long-term memory network based on multimodal token sequence, used to analyze user behavior patterns;

[0042] Generate high-confidence predictions of potential demand when users stand for long periods of time or perform repeated actions (such as waving their hands multiple times);

[0043] The active interaction module triggers interactive actions based on the behavior prediction results, such as initiating inquiries or displaying recommended content.

[0044] A multimodal digital employee reception method based on edge computing includes the following steps:

[0045] S1. Data collection: Acquire the user's voice, image and motion data through the multimodal data collection module;

[0046] S2. Data processing: localize voice signals, image data, and infrared signals through the edge computing module to extract voice features, visual features, and body temperature or motion trajectory features;

[0047] S3. Data fusion: Optimize the fusion model parameters through the Test Time Training (TTT) mechanism, and use the Transformer architecture to dynamically assign weights to multimodal data to generate a token sequence;

[0048] S4. Trigger activation: Combine multimodal data to determine user interaction intention and trigger system response;

[0049] S5. Emotion recognition: Generate emotional state by combining speech features, facial expressions, motion trajectories and thermal imaging data through the emotion recognition module;

[0050] S6. Behavior prediction: predict user potential needs based on the fused token sequence and historical data;

[0051] S7. Interactive response: Generate personalized voice, visual or action interactive feedback based on the comprehensive decision.

[0052] As another improvement of the present application, the specific steps of S3 data fusion include:

[0053] Dynamically optimize Transformer model parameters through Test Time Training (TTT), including: adjusting feature extraction weights in the model based on multimodal data collected in real time;

[0054] Apply specific fusion strategies for abnormal environment data (such as high noise, low light or motion occlusion);

[0055] The multimodal data is divided into multiple time period token sequences and processed as follows through the Transformer architecture:

[0056] Use position encoding to add time dimension information to each Token;

[0057] Apply multi-head attention mechanism to focus on speech semantics, voiceprint features, facial expressions, motion trajectories and thermal imaging data respectively;

[0058] The final decision vector is generated by integrating the token sequences of each time period.

[0059] As another improved supplement of the present application, the specific steps of triggering activation in S4 include:

[0060] Determine whether the user has interaction intention based on multimodal data, including:

[0061] Detect that the user enters the camera's field of view and faces the camera;

[0062] Infrared sensors detect when a user approaches a designated area or performs a trigger action (such as waving a hand);

[0063] The microphone array detects voice commands or voiceprint features that meet preset characteristics;

[0064] Dynamically adjust the trigger judgment threshold to reduce the possibility of false triggering and missed triggering;

[0065] After triggering activation, the system confirms the interaction intention to the user through voice or visual signals.

[0066] As another improved supplement of the present application, the specific steps of S5 emotion recognition include:

[0067] Extract emotional states based on multimodal features, including:

[0068] Capture user facial expressions through high-resolution cameras and analyze them using deep learning models;

[0069] Assess emotional characteristics based on intonation, voiceprint, and volume changes of speech signals;

[0070] Use thermal imaging cameras to extract facial temperature distribution features and combine them with deep learning models to judge emotions;

[0071] Analyze the user's action trajectory (such as gestures and head movements) to help determine the current emotion;

[0072] The Transformer architecture is used to fuse the above multimodal sentiment data, and the weights of different modal data are dynamically adjusted through the multi-head attention mechanism to generate the final sentiment judgment result.

[0073] As another improvement of the present application, the specific steps of S6 behavior prediction include:

[0074] Establish user behavior patterns based on the fused token sequence and historical interaction records;

[0075] Predict potential needs when detecting repetitive user actions (such as standing for a long time or waving multiple times);

[0076] The system proactively triggers personalized interactive behaviors based on the prediction results, for example:

[0077] Provide voice to ask if you need help;

[0078] Display relevant information or recommended content based on user historical preferences;

[0079] Provide visual or motion feedback to enhance the user experience.

[0080] In summary, this application has the following beneficial effects:

[0081] 1. The system integrates high-resolution cameras, microphone arrays, and infrared sensors, which can simultaneously collect information such as the user's voice, image, and body temperature trajectory, realize comprehensive perception of multimodal data, and locally process data through the edge computing module to reduce the latency and resource consumption of cloud computing, ensuring the real-time and reliability of interaction. The Test Time Training (TTT) mechanism can adjust the parameters of the data fusion model according to the real-time environment, improve the system's adaptability in complex scenarios (such as low light and high noise), use the Transformer architecture to integrate multimodal data, and use the multi-head attention mechanism to weigh the importance of voice, vision, and thermal imaging data to ensure efficient and accurate interaction decisions.

[0082] 2. The system determines the interaction intention based on the user's facial expressions, voice commands and motion trajectories, dynamically adjusts the trigger judgment threshold, reduces false triggering or missed triggering, and ensures natural and smooth interaction. After activation, the system confirms the user's intention through voice or visual signals to further improve the accuracy of the interaction. Through the multimodal fusion of vision, voice, thermal imaging and motion trajectories, the system can accurately judge the user's emotional state (such as tension, happiness, confusion). The system predicts the user's potential needs based on historical records and real-time data, and actively triggers personalized services, such as recommended information or voice inquiries, to avoid users from explicitly expressing their needs.

[0083] 3. In complex environments with weak light, background noise or motion occlusion, the system dynamically adjusts the fusion weight of multimodal data to ensure the accuracy of emotion recognition and behavior prediction. The combination of edge computing and multimodal fusion enables the system to operate stably under network fluctuations. By using multimodal perception data (such as thermal imaging to replace visual information in low light conditions and infrared sensing to supplement dynamic motion perception), the system can avoid recognition errors caused by the failure of single-modal data.

[0084] 4. The behavior prediction module combines real-time data and historical behavior patterns to provide users with proactive services, such as providing help suggestions when users are confused. The system supports personalized recommendations based on user historical preferences to improve user satisfaction and service quality. The system supports multimodal interaction methods such as voice, vision, and action, and dynamically adjusts the interaction strategy based on emotional state to provide service feedback that meets the user's current needs.

[0085] 5. The edge computing module completes the processing and fusion of voice, image and infrared data locally, reducing dependence on the cloud and effectively protecting user privacy. The data processing process does not need to transmit user sensitive information to external servers, thereby reducing the risk of data leakage. The system dynamically adjusts the judgment threshold for triggering activation to avoid non-target users triggering the system, further improving user privacy and security.

[0086] 6. After each interaction, the system updates the behavior prediction model by recording user behavior data, so that the system can continuously optimize the interaction strategy. The combination of TTT mechanism and Transformer architecture makes the system scalable and adaptable to different application scenarios. It can integrate other sensing devices (such as heart rate detectors or augmented reality cameras) according to actual needs to further enhance the richness of interaction and application scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is the overall structure diagram of this application;

[0088] Figure 2 System interaction for this application Figure 1 ;

[0089] Figure 3 System interaction for this application Figure 2 ;

[0090] Figure 4 System interaction for this application Figure 3 . DETAILED DESCRIPTION

[0091] Three embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0092] Embodiment 1:

[0093] This embodiment provides a multimodal digital employee reception system based on edge computing, and its overall architecture design is as follows:

[0094] Multimodal data acquisition module:

[0095] High-resolution camera: Captures images of the user and the environment, and supports dynamic adjustment of resolution and frame rate to adapt to strong light, low light and complex background conditions.

[0096] Microphone array: collects user voice signals, supports noise reduction, identifies voice content and voiceprint features, and can locate the direction of the sound source.

[0097] Infrared sensor: senses the user's presence, movement trajectory and body surface temperature, and supports long-distance detection and dynamic change tracking.

[0098] Edge computing module:

[0099] Voice processing unit: uses a localized voice recognition algorithm to extract the semantic features (such as keywords, context) and voiceprint features of the user's voice to ensure high accuracy and privacy protection.

[0100] Image processing unit: Uses the convolutional neural network (CNN) model to recognize the user's facial expressions, movement trajectories, and surrounding environment features, supporting real-time processing.

[0101] Infrared data processing unit: analyzes infrared signals, extracts body temperature data and motion trajectory features to assist in user status judgment.

[0102] Data fusion and decision-making module:

[0103] Test Time Training (TTT) mechanism:

[0104] During operation, the feature extraction weights of the Transformer model are dynamically adjusted based on the real-time collected voice, image and infrared data.

[0105] Optimize the adaptability of data fusion models to environmental changes (e.g., poor lighting, background noise).

[0106] Transformer architecture: divides multimodal data into token sequences of multiple time periods, and applies a multi-head attention mechanism to process data of different modalities:

[0107] The first attention head: analyzes the semantics and voiceprint of speech; the second attention head: analyzes facial expressions and movement trajectories; the third attention head: analyzes the temperature distribution of thermal imaging data.

[0108] Position encoding is used to add time dimension information to Token, and multimodal data is integrated through the self-attention network to generate a comprehensive decision vector.

[0109] Trigger activation module:

[0110] Based on the multimodal trigger mechanism, it determines whether the user has the intention to interact, dynamically adjusts the trigger judgment threshold (such as the facial direction detected by the camera, gesture movements within the infrared sensing range, etc.) to reduce false triggers. After activation, it confirms whether the user needs to start the service through voice or visual signals.

[0111] Emotion recognition module: Comprehensively analyzes the user's facial expression features, voice features, movement trajectory and thermal imaging data to determine the emotional state (such as tension, relaxation), and uses the analysis results to assist trigger judgment and personalized interaction.

[0112] Interactive response module: Generates speech synthesis, visual display or mechanical action response based on the comprehensive decision vector.

[0113] Behavior prediction module: Utilizes historical interaction data and real-time input to analyze user behavior patterns, predict potential needs, and proactively trigger personalized interaction actions (such as recommending information or initiating inquiries).

[0114] Working process

[0115] Data collection stage: The high-resolution camera captures the user's facial image and environmental information in real time; the microphone array collects voice signals and separates voice content from background noise; the infrared sensor monitors the user's distance, gestures and body temperature changes, and generates corresponding signals.

[0116] Data processing stage: The edge computing module performs local processing on the collected data and extracts key features, including: semantic features and voiceprint features of speech signals; facial expressions, movement trajectories and environmental background; body temperature and gesture trajectories of infrared signals. After local processing, the data is passed to the data fusion and decision-making module in the form of a Token sequence.

[0117] Data fusion and dynamic adjustment stage: The TTT mechanism adjusts the feature extraction model parameters according to the current environment, so that the data fusion process adapts to real-time changes. The Transformer architecture inputs the Token sequence of multimodal data into the multi-head attention mechanism: the attention mechanism dynamically weighs the weights of speech, image and infrared data; the self-attention network integrates the Token sequences of multiple time periods to generate a comprehensive decision vector.

[0118] Trigger activation and emotion recognition stage: The system determines whether the user needs service through the trigger activation module: when the camera detects that the user is facing the device and making gestures, the trigger conditions are confirmed in combination with the microphone array voice commands; the trigger rules are dynamically adjusted to reduce misjudgments caused by environmental interference. The emotion recognition module analyzes the user's facial expressions, voice intonation and thermal imaging data to determine the user's current emotional state (such as whether he is nervous or confused).

[0119] Interactive response and behavior prediction stage: The comprehensive decision vector guides the interactive response module to generate voice feedback, visual display or trigger mechanical actions. The system analyzes user behavior patterns (such as standing for a long time) through the behavior prediction module and actively provides recommended content or initiates inquiries.

[0120] By utilizing the TTT mechanism and Transformer architecture, the feature extraction and fusion strategies of multimodal data can be dynamically adjusted to adapt to complex and changing environments.

[0121] Local data processing is achieved through edge computing modules, reducing dependence on cloud computing and improving real-time performance and privacy security.

[0122] Comprehensive use of voice, image and infrared sensing technologies supports natural interaction and personalized services to enhance user experience.

[0123] Through multimodal sentiment analysis and behavioral pattern learning, the system can proactively perceive user needs and provide appropriate services.

[0124] The multi-modal trigger activation mechanism can dynamically adjust the response rules and significantly reduce the incidence of false triggering and missed triggering.

[0125] From system architecture design to specific module implementation and the complete workflow of multimodal data processing, this embodiment demonstrates the technical advantages of the system in the field of intelligent interaction and provides comprehensive support for achieving efficient, stable and natural digital employee reception.

[0126] Embodiment 2:

[0127] This embodiment describes a multimodal digital employee reception method based on edge computing, combining the complete process of multimodal data collection, processing, fusion and decision-making, and achieving real-time adaptability and efficient interaction through Test Time Training (TTT) and Transformer architecture.

[0128] This method consists of the following steps, covering the entire process from multimodal data collection to interactive response:

[0129] S1 Data Collection:

[0130] The multimodal data collection module is used to obtain user information in real time, including:

[0131] High-resolution camera: captures the user's facial expressions, gestures and surrounding environment information, supports low-light enhancement and noise suppression, microphone array: collects user voice signals, supports multi-channel processing to locate the direction of the sound source, and reduces background noise interference, infrared sensor: senses the user's distance, dynamic gestures and body surface temperature, and is used to supplement environmental perception in cases of insufficient light or motion obstruction.

[0132] S2 Data Processing:

[0133] The edge computing module is used to perform local processing on the collected data, including:

[0134] Speech feature extraction: analyze the semantic features (such as keywords, context) and voiceprint features (such as timbre and intonation) of the user's speech; visual feature extraction: analyze the user's facial expressions and movement trajectories through a deep learning model; infrared signal analysis: extract user body temperature data and dynamic gesture trajectories to assist in judging the user's status and intentions.

[0135] S3 Data Fusion:

[0136] Test Time Training (TTT) dynamic optimization model:

[0137] The feature extraction weights in the Transformer model are adjusted according to the real-time collected data. For abnormal environments (such as high noise, low light or motion occlusion), the weight distribution of each modal data is dynamically adjusted to ensure fusion stability.

[0138] Transformer architecture multimodal fusion:

[0139] Divide multimodal data into token sequences of multiple time periods.

[0140] The specific implementation of the multi-head attention mechanism is as follows: the first attention head: analyzes the semantics and voiceprint features of speech data, the second attention head: analyzes facial expressions and movement trajectories, and the third attention head: analyzes the temperature distribution of thermal imaging data.

[0141] Position encoding: Add time dimension information to each token to capture the correlation of data in multiple time periods.

[0142] The tokens of each time period are integrated to generate a comprehensive vector for decision making.

[0143] S4 trigger activation:

[0144] Determine whether the user has interaction intention based on multimodal data, including:

[0145] Vision detection: Confirms whether the user is in the camera's field of view and facing the device.

[0146] Infrared detection: determines whether the user is close to the designated area or makes a trigger gesture (such as waving).

[0147] Voice command detection: The microphone array detects whether the user's voice command or voiceprint signal meets the preset trigger conditions.

[0148] Dynamically adjust the trigger threshold to reduce false triggering or missed triggering caused by environmental interference.

[0149] After the activation is triggered, the user's interaction intention is confirmed through voice or visual feedback.

[0150] S5 Emotion Recognition:

[0151] Use the multimodal sentiment analysis module to comprehensively evaluate the user's emotional state:

[0152] Visual data: Analyze user facial expression characteristics (such as smiling, frowning).

[0153] Voice data: Extract voice intonation, volume, and voiceprint features to determine the user's emotional state (such as happiness, tension).

[0154] Thermal imaging data: Capture facial temperature distribution through infrared sensors to infer emotional fluctuations.

[0155] Action trajectories: Analyze the strength and direction of gestures to supplement emotion assessment.

[0156] S6 behavior prediction:

[0157] Analyze user behavior patterns based on comprehensive decision vectors and historical interaction data.

[0158] Make high-confidence predictions of users’ repeated actions (such as standing for a long time and waving multiple times) to infer their potential needs.

[0159] S7 interactive response:

[0160] Generate personalized responses based on user needs, including:

[0161] Voice interaction: The speech synthesis module generates the reply content.

[0162] Visual interaction: Display recommendation information through screen or projection equipment.

[0163] Action feedback: triggering mechanical actions (such as flashing lights and indicating directions).

[0164] Working process

[0165] Initialization: After the system is started, the multimodal data acquisition module is activated to monitor the user's environment in real time. All data is processed by the edge computing module and input into the data fusion module in the form of a Token sequence.

[0166] Feature extraction and model optimization: Voice, vision and infrared signals are optimized through the TTT mechanism, and the weights of feature extraction are adjusted to ensure the adaptability and robustness of the fusion model.

[0167] Multimodal triggering and emotion analysis: The system confirms the user's intention by triggering the activation module and determines the interaction content based on the analysis results of the emotion recognition module.

[0168] Behavior prediction and personalized interaction: The system uses the behavior prediction module to analyze user needs and generate targeted services, such as recommended content or voice guidance.

[0169] Real-time feedback and updates: After the system completes the interaction, it updates the behavior record to provide data support for subsequent predictions.

[0170] The TTT mechanism dynamically adjusts model parameters, and the Transformer architecture quickly integrates multimodal data, significantly improving decision-making efficiency. Users can start interactions through voice, gestures, or facial expressions. The system provides personalized services based on emotions and behaviors. The localized processing of edge computing reduces dependence on the cloud, ensuring that the system can still run stably when the network fluctuates. The dynamic trigger threshold adjustment rules, combined with multimodal fusion results, effectively reduce the false trigger and missed trigger rates. The integrated emotion recognition and behavior prediction modules enable the system to identify the user's emotional state and actively provide services to enhance the user experience.

[0171] Through the above method description, this embodiment demonstrates the complete implementation process from multimodal data collection to personalized interactive response, providing an intelligent, efficient and humanized solution for the digital employee reception scenario.

[0172] Embodiment 3:

[0173] This embodiment further refines the specific steps of S5 emotion recognition and S6 behavior prediction, aiming to achieve accurate user emotion analysis and behavior prediction, thereby providing personalized services.

[0174] S5 Emotion Recognition:

[0175] Multimodal emotion feature extraction:

[0176] 1) Visual features: The user’s facial image is captured in real time through a high-resolution camera, and based on a deep learning model (such as convolutional neural network, CNN), the user’s facial expression features (such as smiling, frowning) are analyzed to generate a preliminary emotional state (such as happiness, confusion, tension).

[0177] 2) Voice features: Extract intonation, voiceprint and volume changes from the voice signal, analyze the semantics in context, and determine whether the user is in a state of emotional fluctuation.

[0178] 3) Thermal imaging data: The user’s facial temperature distribution characteristics are extracted through a thermal imaging camera, and body temperature changes are analyzed using a deep learning model (such as a recurrent neural network, RNN) to assist in judging emotions.

[0179] 4) Action trajectory: Analyze the user’s gestures and head movements (such as nodding and waving), match the action features with visual and voice data, and further enhance the accuracy of emotion judgment.

[0180] Multimodal emotion fusion:

[0181] The Transformer architecture is used to convert the above multimodal data into a Token sequence.

[0182] The specific implementation of the multi-head attention mechanism is as follows: the first attention head: analyzes the semantics, voiceprint and intonation of the speech signal; the second attention head: processes facial expression features; the third attention head: analyzes the temperature distribution of thermal imaging data; the fourth attention head: analyzes gestures and head movement trajectories.

[0183] Dynamic weight adjustment: Dynamically assign weights of different modalities according to the environment and modal data quality (such as enhancing the weight of thermal imaging data under low-light conditions), integrate the data of each modality, and generate the final emotion judgment result.

[0184] S6 behavior prediction:

[0185] Behavior pattern establishment: Based on the fused token sequence and combined with the user's historical interaction records (such as voice commands and gesture trajectories), a behavior pattern database is constructed using the long-term memory network (LSTM).

[0186] Potential demand prediction: When a user's repetitive actions are detected (such as waving multiple times or standing for a long time), the contextual information of the action is analyzed, combined with historical behavior patterns, and the user's possible needs (such as needing help or paying attention to an object) are predicted with high confidence.

[0187] Personalized interactive behavior triggering: The system actively triggers interactive behaviors based on prediction results, including:

[0188] 1) Voice feedback: Generate a voice to ask if you need help, 2) Information recommendation: Display relevant information or recommended content based on the user's historical preferences, 3) Action feedback: Physically interact with the user through light prompts, direction guidance or screen display.

[0189] Working process

[0190] Emotion recognition stage: After the system is started, it captures the user's facial expressions, voice signals, body temperature data and movement trajectories in real time. The edge computing module localizes the above data and extracts emotional features. The data is processed through the Transformer architecture to generate emotional judgment results, such as whether the user is nervous, happy or confused.

[0191] Behavior prediction stage: The system uses real-time data and historical records to analyze user behavior patterns. When it detects that the user may have needs (such as standing for a long time), the behavior prediction module generates a high-confidence potential demand prediction.

[0192] Personalized interactive feedback: Based on the emotion and behavior prediction results, the system selects the optimal interaction method:

[0193] If the user shows confusion, the system generates a gentle voice inquiry. If it detects that the user is paying attention to an object, the system displays relevant information or recommended content.

[0194] Result feedback and model update: The system records interaction data, updates the behavior pattern database, and provides optimization support for subsequent predictions.

[0195] Vision, voice, thermal imaging and motion data are used to judge emotions, avoiding the shortcomings of single-modal data due to environmental limitations. The dynamic weight adjustment mechanism further improves the robustness and accuracy of emotion recognition.

[0196] The system comprehensively analyzes user behavior through historical records and real-time data, predicts potential needs, and proactively triggers personalized services, thereby improving the intelligence level of user experience.

[0197] The results of emotion recognition are closely integrated with behavior predictions, enabling the system to understand user intentions more accurately and provide interactive feedback that fits the context. Users do not need to explicitly express their needs, and the system can proactively provide services, enhancing the naturalness and fluency of the interaction.

[0198] In low light, noisy or complex motion backgrounds, the system can dynamically adjust weights based on multimodal data to ensure the stability of emotional judgment and behavior prediction.

[0199] After each interaction, the system updates the behavior pattern database, allowing the prediction model to be continuously optimized and gradually improve service quality.

[0200] This embodiment demonstrates the intelligence and efficiency of the digital employee reception system in emotion perception and behavior prediction by refining S5 emotion recognition and S6 behavior prediction and combining them with the collection and analysis of multimodal data, laying a solid foundation for providing personalized services.

[0201] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.

Claims

1. A multimodal digital employee reception system based on edge computing, characterized by: include: The multimodal data collection module is used to collect the user's voice, image and motion information, including: A high-resolution camera to capture image information of the user and the surrounding environment; Microphone array, used to collect user's voice signal; Infrared sensor, used to sense the user's presence, movement trajectory and body temperature; Edge computing module, used for local processing of multimodal data, including: A speech processing unit, used to extract semantic features and voiceprint information from speech signals; An image processing unit, used to recognize the user's facial expressions, movement trajectories, and thermal imaging data; Infrared data processing unit, used to process infrared signals to detect user's body temperature and movement trajectory; The data fusion and decision module dynamically fuses and weights multimodal data through Test Time Training (TTT) and Transformer architecture, including: Test Time Training (TTT) mechanism: Dynamically fine-tune the parameters of the Transformer model based on real-time collected voice, image, and infrared data; dynamically adjust the feature extraction weights of each modal data in response to environmental changes (such as light intensity, background noise, or changes in user voice tone) to improve the real-time adaptability of the model; Transformer Architecture: The multimodal data (speech features, facial expressions, motion trajectories, thermal imaging data) are divided into multiple token sequences according to time periods. The multi-head attention mechanism is used to focus on the features of different modal data. The first attention head: analyzes the semantics and voiceprint of the speech; the second attention head: analyzes facial expressions and motion trajectories; the third attention head: analyzes the temperature distribution of thermal imaging data. Add time dimension information to each token through position encoding, and use a multi-layer self-attention network to integrate token sequences of multiple time periods; Generate the final comprehensive decision vector for subsequent interactive response and behavior prediction; The trigger activation module is used to determine the user's interaction intention based on multimodal data and trigger the system response; Emotion recognition module, which is used to judge the user's emotional state by combining facial expressions, movement trajectories, voice features and thermal imaging data; The interactive response module generates voice, visual or action interactive feedback based on the fused data; The behavior prediction module is used to predict user behavior based on multimodal data and historical interaction records, and actively trigger interactions.

2. According to the multimodal digital employee reception system based on edge computing according to claim 1, it is characterized by: The trigger activation module comprises: Multimodal trigger mechanism: triggers system response based on one or a combination of the following conditions: Detection of the user entering the camera's field of view and facing the camera; The infrared sensor detects when the user approaches the designated interaction area or performs a specific action (such as waving); The microphone array collects voice commands or voiceprint signals that meet preset characteristics; Trigger judgment rules: Dynamically adjust the trigger judgment threshold by combining the matching degree of multi-modal data to reduce the possibility of false triggering; Activation signal confirmation mechanism: After being triggered, the user's intention is confirmed through voice prompts or visual signals to avoid accidental activation of non-target objects.

3. According to the multimodal digital employee reception system based on edge computing according to claim 1, it is characterized by: The emotion recognition module comprises: Expression feature analysis unit: captures the user's facial image through a high-resolution camera, extracts facial expression features using a deep learning model, and generates a preliminary emotional state; Action trajectory analysis unit: monitors the user's action trajectory (such as gestures and head movements) through cameras and infrared sensors, and identifies the user's current emotional tendency based on historical action patterns; Voice feature emotion judgment unit: judges the emotional information contained in the voice based on the tone, volume and voiceprint changes of the user's voice signal; Thermal imaging emotion analysis unit: uses an infrared camera to obtain the user's facial thermal image, extracts temperature distribution features, and combines deep learning models to infer the user's emotional state; Emotion fusion unit: Based on the Transformer architecture, the above multimodal emotion data is converted into a token sequence, and the weights of different modal data are dynamically allocated through the multi-head attention mechanism to generate the final emotion judgment result.

4. According to the multimodal digital employee reception system based on edge computing according to claim 1, it is characterized by: The behavior prediction module includes: Long-term memory network based on multimodal token sequence, used to analyze user behavior patterns; Generate high-confidence predictions of potential demand when users stand for long periods of time or perform repeated actions (such as waving their hands multiple times); The active interaction module triggers interactive actions based on the behavior prediction results, such as initiating inquiries or displaying recommended content.

5. A multimodal digital employee reception method based on edge computing, characterized in that: The following steps are involved: S1. Data collection: Acquire the user's voice, image and motion data through the multimodal data collection module; S2. Data processing: localize voice signals, image data, and infrared signals through the edge computing module to extract voice features, visual features, and body temperature or motion trajectory features; S3. Data fusion: Optimize the fusion model parameters through the Test Time Training (TTT) mechanism, and use the Transformer architecture to dynamically assign weights to multimodal data to generate a token sequence; S4. Trigger activation: Combine multimodal data to determine user interaction intention and trigger system response; S5. Emotion recognition: Generate emotional state by combining speech features, facial expressions, motion trajectories and thermal imaging data through the emotion recognition module; S6. Behavior prediction: predict user potential needs based on the fused token sequence and historical data; S7. Interactive response: Generate personalized voice, visual or action interactive feedback based on the comprehensive decision.

6. According to claim 5, a multimodal digital employee reception method based on edge computing is characterized in that: The specific steps of S3 data fusion include: Dynamically optimize Transformer model parameters through Test Time Training (TTT), including: adjusting feature extraction weights in the model based on multimodal data collected in real time; Apply specific fusion strategies for abnormal environment data (such as high noise, low light or motion occlusion); The multimodal data is divided into multiple time period token sequences and processed as follows through the Transformer architecture: Use position encoding to add time dimension information to each Token; Apply multi-head attention mechanism to focus on speech semantics, voiceprint features, facial expressions, motion trajectories and thermal imaging data respectively; The final decision vector is generated by integrating the token sequences of each time period.

7. The multimodal digital employee reception method based on edge computing according to claim 5 is characterized in that: The specific steps of the S4 trigger activation include: Determine whether the user has interaction intention based on multimodal data, including: Detect that the user enters the camera's field of view and faces the camera; Infrared sensors detect when a user approaches a designated area or performs a trigger action (such as waving a hand); The microphone array detects voice commands or voiceprint features that meet preset characteristics; Dynamically adjust the trigger judgment threshold to reduce the possibility of false triggering and missed triggering; After triggering activation, the system confirms the interaction intention to the user through voice or visual signals.

8. The multimodal digital employee reception method based on edge computing according to claim 5 is characterized in that: The specific steps of the S5 emotion recognition include: Extract emotional states based on multimodal features, including: Capture user facial expressions through high-resolution cameras and analyze them using deep learning models; Assess emotional characteristics based on intonation, voiceprint, and volume changes of speech signals; Use thermal imaging cameras to extract facial temperature distribution features and combine them with deep learning models to judge emotions; Analyze the user's action trajectory (such as gestures and head movements) to help determine the current emotion; The Transformer architecture is used to fuse the above multimodal sentiment data, and the weights of different modal data are dynamically adjusted through the multi-head attention mechanism to generate the final sentiment judgment result.

9. The multimodal digital employee reception method based on edge computing according to claim 5, characterized in that: The specific steps of the S6 behavior prediction include: Establish user behavior patterns based on the fused token sequence and historical interaction records; Predict potential needs when detecting repetitive user actions (such as standing for a long time or waving multiple times); The system proactively triggers personalized interactive behaviors based on the prediction results, for example: Provide voice to ask if you need help; Display relevant information or recommended content based on user historical preferences; Provide visual or motion feedback to enhance the user experience.

Images