An adaptive service interaction method and computer system based on emotional perception
By introducing an environmental signal-to-noise ratio adjustment mechanism and an edge computing module, and combining touch feature recognition to identify user status, the interactive interface of the self-service system is dynamically adjusted. This solves the problem that the self-service system cannot adapt to different user emotions, realizes personalized adaptive service, and improves user experience and task completion rate.
Patent Information
- Application Number
- CN202610501484.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-04-16
Smart Images

Figure CN122195261B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and in particular to an adaptive service interaction method and computer system based on emotion perception. Background Technology
[0002] With the popularization of unmanned economy and service robots, self-service terminals have been widely used in various public places such as hospital registration, shopping mall guidance, and government affairs processing. For example, a self-service robot torso structure and self-service robot disclosed in patent announcement number CN216442566U and a multi-functional self-service robot disclosed in utility model patent announcement number CN208342856U provide users with information inquiry, business processing and other services through screen touch or voice interaction.
[0003] However, during use, it was found that existing self-service interaction systems generally adopt fixed interface layouts and interaction logics, resulting in monotonous interfaces. They cannot adjust service strategies according to the emotional states of different users (such as anxious travelers, curious children, and elderly people with hearing impairments), leading to poor user experience and low service success rates. Therefore, there is an urgent need for an adaptive service interaction method and computer system based on emotion perception to improve the above problems. Summary of the Invention
[0004] To address the aforementioned technical issues, this invention provides a mechanism that overcomes the technical bias of single-voice modality failure caused by strong noise in public places by introducing a dynamic adjustment mechanism for attention weights based on environmental signal-to-noise ratio. This mechanism automatically increases the fusion weights of touch behavior features in noisy environments and utilizes features such as the touch trajectory curvature change rate sequence and click time interval sequence as auxiliary discrimination channels, ensuring the continuity and accuracy of user state recognition. Furthermore, it transforms abstract emotion judgment into quantifiable engineering indicators such as operational proficiency scores, emotional arousal scores, and intent clarity scores, enabling the machine to read the user's mood like a human waiter. It provides a minimalist and efficient interface for low-proficiency, high-arousal states and a patiently guided interface for low-proficiency, low-clarity states, truly achieving personalized and adaptive emotional services. By integrating a spatial awareness module, the system can detect user behavior in front of the terminal in real time and proactively initiate help requests based on facial expression recognition. This transforms the service model from one where users seek services to one where services seek users, significantly improving task completion rates and user satisfaction in unattended scenarios. Touch behavior characteristics rely on signals that can be collected from a general-purpose capacitive touchscreen, eliminating the need for costly pressure-sensitive screen hardware upgrades. This lowers the barrier to upgrading existing self-service terminals and facilitates large-scale deployment. By deploying an edge computing module locally on the terminal to process sensitive facial and voice data, and only uploading anonymized interaction status scores and policy logs to the cloud, this system achieves continuous iterative optimization of the system model while strictly protecting user privacy—a method and computer system based on emotion perception for adaptive service interaction.
[0005] The present invention provides an emotion-based adaptive service interaction method and computer system, comprising: Multimodal acquisition module: used to acquire multimodal user behavior data in interactive scenarios in real time. The multimodal user behavior data includes at least the user's facial video stream, voice stream, and touch behavior sequence data generated when in contact with the interactive interface. Spatial perception module: used to detect the relative positional relationship between the user and the interactive terminal; Edge computing inference module: Deployed locally on the interactive terminal, connected to the multimodal acquisition module and the spatial perception module, used to process the multimodal user behavior data based on a pre-trained lightweight emotion fusion neural network, output the user's current interaction state parameter set, and generate corresponding interaction strategy instructions according to the interaction state parameter set, wherein the interaction state parameter set includes at least operation proficiency score, emotion arousal score and intent clarity score. Interactive execution module: connected to the edge computing inference module, used to respond to the interactive strategy instructions and dynamically adjust the graphical user interface layout parameters and speech synthesis broadcast parameters of the interactive terminal; By introducing a dynamic attention weight adjustment mechanism based on environmental signal-to-noise ratio, the technical bias of single voice modality failure caused by strong noise in public places is overcome. In noisy environments, the fusion weight of touch behavior features is automatically increased. Features such as the touch trajectory curvature change rate sequence and click time interval sequence are used as auxiliary discrimination channels to ensure the continuity and accuracy of user state recognition. Abstract emotion judgment is transformed into quantifiable engineering indicators such as operation proficiency score, emotional arousal score, and intent clarity score. This enables the machine to read between the lines like a human waiter, providing a minimalist and efficient interface for low proficiency and high arousal states, and a patiently guided interface for low proficiency and low intent clarity states, truly achieving personalized and emotionally adaptive responses. The system provides services; by integrating a spatial awareness module, it can detect user behavior in front of the terminal in real time and proactively initiate help requests based on facial expression recognition, realizing a shift from a "people seeking services" model to a "services seeking people" model, significantly improving task completion rate and user satisfaction in unattended scenarios; the touch behavior characteristics rely on signals that can be collected by a general capacitive touchscreen, eliminating the need for costly pressure-sensitive screen hardware upgrades, lowering the barrier to upgrading existing self-service terminal equipment, and facilitating large-scale application; by deploying an edge computing module locally on the terminal to process sensitive facial and voice data, and only uploading anonymized interaction status scores and policy logs to the cloud, the system model achieves continuous iterative optimization while strictly protecting user privacy.
[0006] Preferably, the multimodal acquisition module includes: Visual acquisition subunit: used to acquire the user's facial video stream through a visual sensor; Audio acquisition subunit: used to acquire the user's voice stream through an audio sensor; Touch acquisition subunit: used to acquire the sequence data of touch behavior generated when the user interacts with the interface through touch sensors; Touch behavior sequence data includes at least the curvature change rate sequence of touch swipe trajectory, the time interval sequence of continuous click operation, and the fluctuation variance sequence of touch contact area.
[0007] Preferably, the spatial perception module includes: Distance detection subunit: used to detect the relative distance between the user and the interactive terminal using an ultrasonic sensor or an infrared sensor; Dwelling timer subunit: connected to the distance detection subunit, used to record the continuous dwell time of the user within the preset warning area.
[0008] Preferably, the edge computing inference module includes: Feature extraction subunit: connected to the multimodal acquisition module, used to extract facial motion unit features from facial video stream, extract speech feature parameters from speech stream, and extract touch behavior features from touch behavior sequence data; Lightweight emotion fusion neural network subunit: connected to the feature extraction subunit, used to receive features of each modality and output the set of interaction state parameters; Rule Engine Subunit: Connected to the Lightweight Emotion Fusion Neural Network Subunit, it has a pre-stored mapping table between scoring thresholds and interaction strategies, which is used to generate corresponding interaction strategy instructions based on the interaction state parameter set; Data anonymization subunit: Connected to the rule engine subunit, it receives the interaction status parameter set and interaction strategy instructions, and performs anonymization processing; The de-identification process includes: deleting or hashing direct identifiers associated with user identity, and retaining only the interaction status score, triggered policy label, de-identified device identifier, and timestamp.
[0009] Preferably, the lightweight emotion fusion neural network subunit includes: Single-modal coding layer: corresponding to facial action unit features, speech feature parameters and touch behavior features, used to map each modality feature to the same feature space; Attention weight calculation layer: connected to the single-modality coding layer, used to calculate the multimodal attention allocation weights in the current environment; Environmental noise assessment interface: connected to the audio acquisition subunit, used to obtain the environmental signal-to-noise ratio of the current audio stream and output the environmental signal-to-noise ratio to the attention weight calculation layer; Fusion output layer: used to perform weighted fusion of the encodings of each modality based on the attention weight coefficients, and output the set of interaction state parameters; Specifically, when the environmental signal-to-noise ratio is lower than the preset clarity threshold, the attention weight calculation layer increases the attention weight coefficient corresponding to the touch behavior features and correspondingly decreases the attention weight coefficient corresponding to the speech feature parameters.
[0010] Preferably, the mapping relationship table pre-stored within the rule engine subunit includes: First mapping entry: When the operation proficiency score is lower than the first threshold and the emotional arousal score is higher than the second threshold, output an efficient interaction strategy instruction; Second mapping item: When the operation proficiency score is lower than the first threshold and the intent clarity score is lower than the third threshold, output a guided interaction strategy instruction; The third mapping entry: When the rate of change of curvature of the touch swipe trajectory in the touch behavior sequence data is greater than the fourth threshold and the emotional arousal score is within the preset pleasant range, an immersive interaction strategy instruction is output.
[0011] Preferably, the interactive execution module includes: Interface rendering subunit: Used to respond to interaction strategy commands and adjust the layout parameters of the graphical user interface; Speech synthesis subunit: Used to respond to interactive strategy commands and adjust the speech rate and / or timbre parameters of the speech playback.
[0012] Preferably, cloud data platforms are also included; The cloud data platform includes: De-identified data receiving interface: connected to the edge computing inference module, used to receive the de-identified set of interaction status parameters and the corresponding interaction strategy instruction logs; Offline training unit: Used to incrementally train the network weights of lightweight emotion fusion neural network subunits based on accumulated de-identified data; Model distribution interface: Used to distribute the updated network weight parameters to the lightweight sentiment fusion neural network subunit in the edge computing inference module.
[0013] Preferably, the method for providing emotion-aware adaptive services using an emotion-aware adaptive service interaction computer system includes the following steps: S1. Multimodal behavior data acquisition steps: The user's facial video stream, voice stream, and touch behavior sequence data are acquired through the multimodal acquisition module. The touch behavior sequence data includes at least the curvature change rate sequence of the touch swipe trajectory and the time interval sequence of continuous click operations. S2, Spatial Awareness Step: The spatial awareness module detects the relative positional relationship between the user and the interactive terminal; S3. User interaction state recognition steps: The multimodal data is processed by the lightweight emotion fusion neural network in the edge computing inference module deployed locally on the interactive terminal, and the current user interaction state parameter set is output. The interaction state parameter set includes at least the operation proficiency score, emotion arousal score and intent clarity score. S4. Interaction strategy adaptive matching step: The rule engine in the edge computing inference module matches the corresponding interaction strategy instruction according to the interaction state parameter set. S5. Dynamic feedback execution steps: The interactive execution module adjusts the layout of the interactive interface and voice broadcast according to the interactive strategy instructions.
[0014] Preferably, the spatial perception step is as follows: when the spatial perception module detects that the user stays in the preset warning area for a longer period of time than a set time threshold, and the facial video stream contains facial motion unit features associated with a confused expression, an active inquiry pop-up is triggered, and high-frequency service options are pushed to the front of the screen.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By introducing a dynamic adjustment mechanism for attention weights based on environmental signal-to-noise ratio, the technical bias of single voice modality failure caused by strong noise in public places is overcome. The fusion weight of touch behavior features is automatically increased in noisy environments. Features such as the curvature change rate sequence of touch trajectory and the click time interval sequence are used as auxiliary discrimination channels to ensure the continuity and accuracy of user status recognition. 2. Transform abstract emotion judgments into quantifiable engineering indicators such as operational proficiency scores, emotional arousal scores, and intent clarity scores. This enables machines to read people's expressions like human waiters, providing a simple and efficient interface for low proficiency and high arousal states, and a patient and guiding interface for low proficiency and low intent clarity states, truly achieving personalized and adaptive emotional services for each individual. 3. By integrating a spatial perception module, the system can detect the user's lingering behavior in front of the terminal in real time, and actively initiate help requests in combination with facial expression recognition, realizing the transformation from people looking for services to services looking for people, significantly improving the task completion rate and user satisfaction in unattended scenarios; 4. The touch behavior characteristics rely on signals that can be collected by a general capacitive touch screen, without the need for high-cost pressure-sensitive screen hardware modifications, which lowers the threshold for upgrading existing self-service terminal equipment and facilitates large-scale promotion and application. 5. By deploying an edge computing module locally on the terminal to process sensitive facial and voice data, and only uploading anonymized interaction status scores and policy logs to the cloud, the system model can be continuously iterated and optimized while strictly protecting user privacy. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the structure of the adaptive service interaction computer system based on emotion perception according to the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal acquisition module of the present invention; Figure 3 This is a schematic diagram of the spatial perception module of the present invention; Figure 4 This is a schematic diagram of the edge computing inference module of the present invention; Figure 5 This is a schematic diagram of the structure of the lightweight emotion fusion neural network subunit of the present invention; Figure 6 This is a schematic diagram of the interactive execution module of the present invention; Figure 7 This is a schematic diagram of the structure of the cloud data platform of the present invention. Detailed Implementation
[0017] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. The present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
[0018] like Figures 1 to 7 As shown, an adaptive service interaction method and computer system based on emotion perception includes: Multimodal acquisition module: used to acquire multimodal user behavior data in interactive scenarios in real time. The multimodal user behavior data includes at least the user's facial video stream, voice stream, and touch behavior sequence data generated when in contact with the interactive interface. Spatial perception module: used to detect the relative positional relationship between the user and the interactive terminal; Edge computing inference module: Deployed locally on the interactive terminal, connected to the multimodal acquisition module and the spatial perception module, used to process the multimodal user behavior data based on a pre-trained lightweight emotion fusion neural network, output the user's current interaction state parameter set, and generate corresponding interaction strategy instructions according to the interaction state parameter set, wherein the interaction state parameter set includes at least operation proficiency score, emotion arousal score and intent clarity score. Interactive execution module: connected to the edge computing inference module, used to respond to the interactive strategy instructions and dynamically adjust the graphical user interface layout parameters and speech synthesis broadcast parameters of the interactive terminal; The multimodal acquisition module includes: Visual acquisition subunit: used to acquire the user's facial video stream through a visual sensor; Audio acquisition subunit: used to acquire the user's voice stream through an audio sensor; Touch acquisition subunit: used to acquire the sequence data of touch behavior generated when the user interacts with the interface through touch sensors; The touch behavior sequence data includes at least the curvature change rate sequence of the touch swipe trajectory, the time interval sequence of continuous click operations, and the fluctuation variance sequence of the touch contact area; The spatial perception module includes: Distance detection subunit: used to detect the relative distance between the user and the interactive terminal using an ultrasonic sensor or an infrared sensor; Dwelling timer subunit: connected to the distance detection subunit, used to record the continuous dwell time of the user within the preset warning area; The edge computing inference module includes: Feature extraction subunit: connected to the multimodal acquisition module, used to extract facial motion unit features from facial video stream, extract speech feature parameters from speech stream, and extract touch behavior features from touch behavior sequence data; Lightweight emotion fusion neural network subunit: connected to the feature extraction subunit, used to receive features of each modality and output the set of interaction state parameters; Rule Engine Subunit: Connected to the Lightweight Emotion Fusion Neural Network Subunit, it has a pre-stored mapping table between scoring thresholds and interaction strategies, which is used to generate corresponding interaction strategy instructions based on the interaction state parameter set; Data anonymization subunit: Connected to the rule engine subunit, it receives the interaction status parameter set and interaction strategy instructions, and performs anonymization processing; The de-identification process includes: deleting or hashing direct identifiers associated with user identity (such as raw facial image data and raw voice data), and retaining only the interaction status score, the triggered policy label, the de-identified device identifier, and the timestamp; The lightweight emotion fusion neural network subunit includes: Single-modal coding layer: corresponding to facial action unit features, speech feature parameters and touch behavior features, used to map each modality feature to the same feature space; Attention weight calculation layer: connected to the single-modality coding layer, used to calculate the multimodal attention allocation weights in the current environment; Environmental noise assessment interface: connected to the audio acquisition subunit, used to obtain the environmental signal-to-noise ratio of the current audio stream and output the environmental signal-to-noise ratio to the attention weight calculation layer; Fusion output layer: used to perform weighted fusion of the encodings of each modality based on the attention weight coefficients, and output the set of interaction state parameters; When the environmental signal-to-noise ratio is lower than the preset clarity threshold, the attention weight calculation layer increases the attention weight coefficient corresponding to the touch behavior feature and decreases the attention weight coefficient corresponding to the voice feature parameter accordingly. The pre-stored mapping tables within the rule engine subunit include: First mapping entry: When the operation proficiency score is lower than the first threshold and the emotional arousal score is higher than the second threshold, output an efficient interaction strategy instruction; Second mapping item: When the operation proficiency score is lower than the first threshold and the intent clarity score is lower than the third threshold, output a guided interaction strategy instruction; Third mapping entry: When the rate of change of curvature of touch sliding trajectory in touch behavior sequence data is greater than the fourth threshold and the emotional arousal score is in the preset pleasant range, an immersive interaction strategy instruction is output. The interactive execution module includes: Interface rendering subunit: Used to respond to interaction strategy commands and adjust the layout parameters of the graphical user interface; Speech synthesis subunit: used to respond to interactive strategy commands and adjust the speech rate parameters and / or timbre parameters of the speech playback; In response to efficient interaction strategy commands, the interface rendering subunit enlarges the area of the function button trigger hot zone to 120% to 150% of the base size and blocks unnecessary transition animations; the speech synthesis subunit increases the speech playback rate to 1.2 to 1.5 times the base rate. It also includes cloud data platforms; The cloud data platform includes: De-identified data receiving interface: connected to the edge computing inference module, used to receive the de-identified set of interaction status parameters and the corresponding interaction strategy instruction logs; Offline training unit: Used to incrementally train the network weights of lightweight emotion fusion neural network subunits based on accumulated de-identified data; Model distribution interface: Used to distribute the updated network weight parameters to the lightweight sentiment fusion neural network subunit in the edge computing inference module.
[0019] A method for providing emotion-aware adaptive services using the emotion-aware adaptive service interaction computer system according to any one of claims 1-8 includes the following steps: S1. Multimodal behavior data acquisition steps: The user's facial video stream, voice stream, and touch behavior sequence data are acquired through the multimodal acquisition module. The touch behavior sequence data includes at least the curvature change rate sequence of the touch swipe trajectory and the time interval sequence of continuous click operations. S2, Spatial Awareness Step: The spatial awareness module detects the relative positional relationship between the user and the interactive terminal; S3. User interaction state recognition steps: The multimodal data is processed by the lightweight emotion fusion neural network in the edge computing inference module deployed locally on the interactive terminal, and the current user interaction state parameter set is output. The interaction state parameter set includes at least the operation proficiency score, emotion arousal score and intent clarity score. S4. Interaction strategy adaptive matching step: The rule engine in the edge computing inference module matches the corresponding interaction strategy instruction according to the interaction state parameter set. S5. Dynamic feedback execution steps: The interactive execution module adjusts the layout of the interactive interface and voice broadcast according to the interactive strategy instructions; The spatial perception step is as follows: when the spatial perception module detects that the user stays in the preset warning area for more than the set time threshold, and the facial video stream contains facial motion unit features associated with a confused expression, an active inquiry pop-up is triggered, and high-frequency service options are pushed to the front of the screen. Example
[0020] S1. Multimodal behavior data acquisition steps: After the system detects that the user has entered the interaction area (e.g., less than 1 meter from the screen), it automatically activates the visual acquisition subunit to acquire the facial video stream, the audio acquisition subunit to acquire the voice stream, and the touch acquisition subunit to listen to touch events in real time and generate touch behavior sequence data. If the user only performs a click operation without uttering a voice, the voice stream data is marked as empty, and subsequent recognition steps automatically downgrade to rely on visual and touch modalities. S2, Spatial Awareness Step: Parallel to step S1, the spatial awareness module continuously detects the relative distance between the user and the interactive terminal. When the user enters the preset warning area (e.g., between 0.5 meters and 1.5 meters from the screen), the dwell timer subunit starts timing. S3. User interaction state recognition steps: The feature extraction subunit in the edge computing inference module extracts features from the original data of each modality. The lightweight emotion fusion neural network subunit receives the feature vector and combines the attention weight to adjust the logic output interaction state parameter set. The attention weight adjustment logic is as follows: obtain the current environmental signal-to-noise ratio through the environmental noise assessment interface, determine whether the signal-to-noise ratio is lower than the preset clarity threshold (e.g., 15dB), if it is lower than the threshold, reduce the attention weight of speech features in the attention weight calculation layer and increase the attention weight of touch behavior features proportionally; if it is higher than the threshold, maintain the default weight allocation. S4. Adaptive matching steps for interaction strategies: The rule engine subunit reads the above three scores, traverses the internal mapping relationship table, and determines the interaction strategy instruction corresponding to the current user state. If the user state satisfies multiple strategies at the same time (e.g., both anxious and confused), the system can arbitrate and output according to the preset priority (e.g., guiding type takes precedence over efficient type takes precedence over immersive type). S5. Dynamic feedback execution steps: The interactive execution module receives strategy instructions, the interface rendering subunit adjusts the screen element layout, and the speech synthesis subunit adjusts the speech broadcast parameters. S6. This step can run in parallel with the main process described above. When the dwell time exceeds the set time threshold (e.g., 8 seconds) and the touch acquisition subunit does not detect any valid click operation during this period, and the visual acquisition subunit recognizes that the facial action unit matches the frowning or looking around template (i.e., facial action unit features associated with a confused expression), the system triggers an active service. The interactive execution module forces a semi-transparent pop-up window to display the message "Do you need help?" and directly pushes the three most frequently used service functions (such as registration, report retrieval, and payment) to the front of the screen as quick access buttons for users to access with one click.
[0021] To enable those skilled in the art to better understand and implement the present invention, the following provides an example of a preferred range of values for each threshold and parameter. It should be emphasized that the following values are merely illustrative examples, and those skilled in the art can make adaptive adjustments according to actual application scenarios, which do not constitute a limitation on the scope of protection of the present invention. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An adaptive service interaction computer system based on emotion perception, characterized in that, include: Multimodal acquisition module: used to acquire multimodal user behavior data in interactive scenarios in real time. The multimodal user behavior data includes at least the user's facial video stream, voice stream, and touch behavior sequence data generated when in contact with the interactive interface. Spatial perception module: used to detect the relative positional relationship between the user and the interactive terminal; Edge computing inference module: Deployed locally on the interactive terminal, connected to the multimodal acquisition module and the spatial perception module, used to process the multimodal user behavior data based on a pre-trained lightweight emotion fusion neural network, output the user's current interaction state parameter set, and generate corresponding interaction strategy instructions according to the interaction state parameter set, wherein the interaction state parameter set includes at least operation proficiency score, emotion arousal score and intent clarity score. Interactive execution module: connected to the edge computing inference module, used to respond to the interactive strategy instructions and dynamically adjust the graphical user interface layout parameters and speech synthesis broadcast parameters of the interactive terminal; The edge computing inference module includes: Feature extraction subunit: connected to the multimodal acquisition module, used to extract facial motion unit features from facial video stream, extract speech feature parameters from speech stream, and extract touch behavior features from touch behavior sequence data; Lightweight emotion fusion neural network subunit: connected to the feature extraction subunit, used to receive features of each modality and output the set of interaction state parameters; Rule Engine Subunit: Connected to the Lightweight Emotion Fusion Neural Network Subunit, it has a pre-stored mapping table between scoring thresholds and interaction strategies, which is used to generate corresponding interaction strategy instructions based on the interaction state parameter set; Data anonymization subunit: Connected to the rule engine subunit, it receives the interaction status parameter set and interaction strategy instructions, and performs anonymization processing; The de-identification process includes: deleting or hashing direct identifiers associated with user identity, and retaining only the interaction status score, the triggered policy label, the de-identified device identifier, and the timestamp; The lightweight emotion fusion neural network subunit includes: Single-modal coding layer: corresponding to facial action unit features, speech feature parameters and touch behavior features, used to map each modality feature to the same feature space; Attention weight calculation layer: connected to the single-modality coding layer, used to calculate the multimodal attention allocation weights in the current environment; Environmental noise assessment interface: connected to the audio acquisition subunit, used to obtain the environmental signal-to-noise ratio of the current audio stream and output the environmental signal-to-noise ratio to the attention weight calculation layer; Fusion output layer: used to perform weighted fusion of the encodings of each modality based on the attention weight coefficients, and output the set of interaction state parameters; When the environmental signal-to-noise ratio is lower than the preset clarity threshold, the attention weight calculation layer increases the attention weight coefficient corresponding to the touch behavior feature and decreases the attention weight coefficient corresponding to the voice feature parameter accordingly. The pre-stored mapping tables within the rule engine subunit include: First mapping entry: When the operation proficiency score is lower than the first threshold and the emotional arousal score is higher than the second threshold, output an efficient interaction strategy instruction; Second mapping item: When the operation proficiency score is lower than the first threshold and the intent clarity score is lower than the third threshold, output a guided interaction strategy instruction; The third mapping entry: When the rate of change of curvature of the touch swipe trajectory in the touch behavior sequence data is greater than the fourth threshold and the emotional arousal score is within the preset pleasant range, an immersive interaction strategy instruction is output.
2. The adaptive service interaction computer system based on emotion perception as described in claim 1, characterized in that, The multimodal acquisition module includes: Visual acquisition subunit: used to acquire the user's facial video stream through a visual sensor; Audio acquisition subunit: used to acquire the user's voice stream through an audio sensor; Touch acquisition subunit: used to acquire the sequence data of touch behavior generated when the user interacts with the interface through touch sensors; The touch behavior sequence data includes at least the curvature change rate sequence of the touch swipe trajectory, the time interval sequence of continuous click operations, and the fluctuation variance sequence of the touch contact area.
3. The adaptive service interaction computer system based on emotion perception as described in claim 1, characterized in that, The spatial perception module includes: Distance detection subunit: used to detect the relative distance between the user and the interactive terminal using an ultrasonic sensor or an infrared sensor; Dwelling timer subunit: connected to the distance detection subunit, used to record the continuous dwell time of the user within the preset warning area.
4. The adaptive service interaction computer system based on emotion perception as described in claim 1, characterized in that, The interactive execution module includes: Interface rendering subunit: Used to respond to interaction strategy commands and adjust the layout parameters of the graphical user interface; Speech synthesis subunit: Used to respond to interactive strategy commands and adjust the speech rate and / or timbre parameters of the speech playback.
5. The adaptive service interaction method and computer system based on emotion perception as described in claim 1, characterized in that, It also includes cloud data platforms; The cloud data platform includes: De-identified data receiving interface: connected to the edge computing inference module, used to receive the de-identified set of interaction status parameters and the corresponding interaction strategy instruction logs; Offline training unit: Used to incrementally train the network weights of lightweight emotion fusion neural network subunits based on accumulated de-identified data; Model distribution interface: Used to distribute the updated network weight parameters to the lightweight sentiment fusion neural network subunit in the edge computing inference module.
6. An adaptive service interaction method based on emotion perception, characterized in that, A method for providing emotion-aware adaptive services using the emotion-aware adaptive service interaction computer system according to any one of claims 1-5 includes the following steps: S1. Multimodal behavior data acquisition steps: The user's facial video stream, voice stream, and touch behavior sequence data are acquired through the multimodal acquisition module. The touch behavior sequence data includes at least the curvature change rate sequence of the touch swipe trajectory and the time interval sequence of continuous click operations. S2, Spatial Awareness Step: The spatial awareness module detects the relative positional relationship between the user and the interactive terminal; S3. User interaction state recognition steps: The multimodal data is processed by the lightweight emotion fusion neural network in the edge computing inference module deployed locally on the interactive terminal, and the current user interaction state parameter set is output. The interaction state parameter set includes at least the operation proficiency score, emotion arousal score and intent clarity score. S4. Interaction strategy adaptive matching step: The rule engine in the edge computing inference module matches the corresponding interaction strategy instruction according to the interaction state parameter set. S5. Dynamic feedback execution steps: The interactive execution module adjusts the layout of the interactive interface and voice broadcast according to the interactive strategy instructions.
7. The adaptive service interaction method based on emotion perception as described in claim 6, characterized in that, The spatial perception step is as follows: when the spatial perception module detects that the user stays in the preset warning area for more than the set time threshold, and the facial video stream contains facial motion unit features associated with a confused expression, an active inquiry pop-up is triggered, and high-frequency service options are pushed to the front of the screen.
Citation Information
Patent Citations
Multi -functional self -service service robot
CN208342856U
Self-service robot trunk structure and self-service robot
CN216442566U
Guidance service providing system for self-service business terminal
CN111915820A
Human-computer interaction public device with real-time emotion recognition function
CN113791690A
Method for driving emotion interaction of intelligent device based on multi-modal understanding
CN120994067A
