Multi-user interaction response strategy automatic switching method fusing behaviors and emotions

Abstract

The invention provides a multi-user interaction response strategy automatic switching method fusing behaviors and emotions, and the method comprises the steps: S1, collecting the behavior data of a plurality of users in a current environment in real time through a microphone array, a camera and a depth sensor, S2, detecting the behavior event of each user based on the behavior data, s3, identifying the current emotional state and emotional intensity of each user based on the voice signal, the image sequence and the spatial position information; s4, constructing a four-dimensional conflict evaluation model between any two users, and calculating a comprehensive conflict level between the user pairs; and S5, automatically selecting and switching to a target response strategy from a plurality of preset response strategies so as to respond to interaction of multiple users. The method is high in response real-time performance, interaction efficiency is improved, conflict recognition is accurate, intelligent decision-making ability is improved, multi-mode emotion recognition is fused, the humanization level is enhanced, scene self-adaption is supported, and system generalization ability is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically a method for automatically switching multi-user interaction response strategies that integrates behavior and emotion. Background Technology

[0002] With the widespread adoption of intelligent systems such as service robots, smart voice assistants, and virtual customer service, one of the core challenges they face in scenarios such as exhibition guidance, shopping mall assistance, medical care, and educational companionship is how to efficiently and naturally handle concurrent interaction requests from multiple users. While current mainstream multi-user interaction solutions have made some progress, they still have significant limitations.

[0003] The first typical approach employs a batch processing mechanism with a fixed time window. For example, the patent document "A Method, Device, and Storage Medium for Interactive Communication" (application number CN202080004243.6) proposes switching targets by detecting whether the current interactive object has stopped interacting. This system relies on a preset time window to cache and uniformly process user commands. Its main drawbacks are: first, the fixed waiting window inevitably introduces response latency, resulting in a poor user experience in high-concurrency scenarios; second, this approach completely disregards the user's emotional state, failing to identify user anxiety or dissatisfaction caused by waiting, and unable to adjust the response method according to emotions, leading to a stiff interaction and a lack of humanized care.

[0004] The second approach focuses on matching needs through data analysis. For example, the patent document "Multi-user Communication Method, System, and Readable Storage Medium Based on AI Robot" (application number CN202411788569.9) obtains user data through a pre-defined interface and performs tag-based matching. This approach can handle multi-user needs to some extent, but its conflict assessment mechanism is relatively rigid. Its shortcomings are: first, conflict judgment usually relies only on a single dimension such as semantics or time, lacking comprehensive consideration of multi-dimensional information such as spatial location and user emotions, leading to inaccurate conflict identification; second, it uses fixed weights or rules, making it impossible to dynamically adjust the judgment criteria according to different application scenarios (such as a noisy exhibition versus a quiet medical environment), resulting in poor system scenario adaptability.

[0005] The third approach attempts to introduce multimodal information to enhance the naturalness of interaction. The patent document "Multimodal Intelligent Robot Interaction Method and Intelligent Robot" (application number CN201680001734.9) integrates multimodal data such as voice and images for user recognition and interaction. However, this approach lacks dynamic adjustment capabilities in conflict detection and response strategies when facing complex and dynamic multi-user concurrent scenarios. The system cannot adaptively adjust the importance (weight) of each perception dimension and switch response strategies based on real-time interaction dynamics (such as sudden changes in user emotions or spatial relationships) and specific scenario requirements, thus limiting its flexibility and intelligence in practical applications.

[0006] In summary, existing technologies face core challenges in multi-user concurrent interaction scenarios, including high response latency, inaccurate and limited conflict identification dimensions, a complete lack of emotional perception and adaptation, and weak scenario adaptability. These shortcomings make it difficult for existing systems to meet the growing demands of intelligent interactive applications that require higher real-time performance, accuracy, and user-friendliness. Therefore, there is an urgent need for a new method capable of real-time perception, comprehensive judgment, and intelligent and flexible response to multi-user concurrent requests. Summary of the Invention

[0007] This invention addresses a series of problems in multi-user concurrent interaction scenarios, such as high response latency, weak conflict recognition capability, lack of emotion perception, and rigid strategies.

[0008] To address the aforementioned problems, this invention provides a method, system, electronic device, and storage medium for automatically switching multi-user interaction response strategies that integrates behavior and emotion.

[0009] In a first aspect, the present invention provides a method for automatically switching multi-user interaction response strategies that integrates behavior and emotion, comprising the following steps: S1: Real-time collection of behavioral data of multiple users in the current environment through microphone array, camera and depth sensor, the behavioral data including voice signals, image sequences and spatial location information; S2: Based on the behavioral data, detect the behavioral events of each user, including voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject; S3: In response to the triggering of any of the aforementioned behavioral events, obtain the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on the voice signal, image sequence and spatial location information; S4: Construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pair; the four-dimensional conflict assessment model includes at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension, and emotional conflict dimension; S5: Dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model according to the current application scenario, user identity and / or historical interaction data, and automatically select and switch to the target response strategy from a variety of preset response strategies based on the comprehensive conflict level after weight adjustment, in order to respond to the multi-user interaction.

[0010] Optionally, in step S2, the behavioral event further includes a user priority change event, wherein the priority is determined based on at least one of user identity, historical interaction frequency, and current emotional state.

[0011] Optionally, in step S3, identifying the current emotional state and intensity of each user using multimodal emotion recognition technology includes: Acoustic features are extracted and classified from the speech signal to obtain a first emotion category and a first emotion intensity; Facial feature point detection and expression classification are performed on the image sequence to obtain a second emotion category and a second emotion intensity; The image sequence and / or depth sensor data are subjected to skeletal keypoint detection and posture / motion classification to obtain a third emotion category and a third emotion intensity. Based on the first emotion intensity, the second emotion intensity, and the third emotion intensity, the final emotion intensity is determined by weighted fusion or decision-level fusion, and the final emotion state is determined according to the first emotion category, the second emotion category, and the third emotion category.

[0012] Optionally, in step S4, the overall conflict level is calculated using the following formula: ; in, Indicates user With users The overall level of conflict between them Indicates the time synchronization score. Indicates spatial overlap score, Indicates semantic similarity. Indicates emotional conflict rating. , , and The corresponding weights.

[0013] Optionally, the emotional conflict score is calculated using the following formula: ; In the formula, Indicates user The intensity of emotion, Indicates user The intensity of emotion, This represents the average emotional intensity weight. Indicates the weight of emotional differences. Indicates user With users Emotional conflict rating between them.

[0014] Optionally, in step S5, the preset multiple response strategies include at least: a parallel response strategy, a sequential response strategy, a prompt confirmation strategy, a pause execution strategy, and a background intervention strategy; when the overall conflict level is higher than the first threshold, the prompt confirmation strategy, the pause execution strategy, or the background intervention strategy is selected; when the overall conflict level is between the first threshold and the second threshold, the sequential response strategy is selected; when the overall conflict level is lower than the second threshold, the parallel response strategy is selected.

[0015] Optionally, in step S5, dynamically adjusting the weights according to the current application scenario includes: Identify the current application scenario based on environmental images, number of users, and interactive content characteristics; According to the preset scene weight mapping table, initial weights are assigned to each dimension in the four-dimensional conflict assessment model.

[0016] Optionally, in step S5, when automatically selecting and switching to the target response strategy, the response method is also adjusted in conjunction with the current emotional state; when the user's emotional state is identified as anxious or angry, a soothing tone of voice and / or a priority response is adopted; when the user's emotional state is identified as calm or happy, a normal tone of voice and / or additional service options are adopted.

[0017] Secondly, embodiments of the present invention provide an automatic switching system for multi-user interaction response strategies that integrates behavior and emotion, the system comprising: The data acquisition module is configured to collect behavioral data of multiple users in the current environment in real time through a microphone array, camera and depth sensor. The behavioral data includes voice signals, image sequences and spatial location information. The behavior event detection module is configured to detect each user's behavior events based on the behavior data. The behavior events include voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject. The emotion recognition module is configured to, in response to the triggering of any of the aforementioned behavioral events, acquire the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on the voice signal, image sequence, and spatial location information. The conflict assessment module is configured to construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pair; the four-dimensional conflict assessment model includes at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension, and emotional conflict dimension. The strategy control module is configured to dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model based on the current application scenario, user identity, and / or historical interaction data, and automatically select and switch to the target response strategy from a variety of preset response strategies based on the comprehensive conflict level after weight adjustment, in order to respond to the multi-user interaction.

[0018] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the automatic switching method for multi-user interactive response strategies that integrates behavior and emotion as described in the first aspect.

[0020] The beneficial effects of the automatic switching method for multi-user interaction response strategies that integrates behavior and emotion in this invention are as follows: Abandoning the traditional time window mechanism, this system adopts an event-driven approach, dynamically triggering conflict detection and response strategy switching processes based on user behavioral events such as voice endings, gestures, spatial location changes, and priority shifts. This achieves low-latency, high-concurrency intelligent responses. A four-dimensional conflict assessment model is constructed, incorporating time synchronization scoring, spatial overlap scoring, semantic similarity scoring, and emotional conflict scoring. A dynamic weighting mechanism is used to weight different dimensions, enabling accurate identification and level assessment of multi-user interaction conflicts. A method combining user emotional states to adjust conflict levels improves the accuracy of conflict identification and the adaptability of response strategies in complex multi-user scenarios. Based on the identified current application scenario (such as exhibition guidance, shopping mall guidance, medical companionship, etc.), response strategies are automatically switched (such as parallel response, sequential response, pause intervention, etc.), and the response method is optimized based on user emotional states. This achieves true scenario adaptation and humanized response, improving user experience and system intelligence. Attached Figure Description

[0021] Figure 1 This is a flowchart of the automatic switching method for multi-user interaction response strategies that integrates behavior and emotion in an embodiment of the present invention; Figure 2This is a schematic diagram of the automatic switching method for multi-user interaction response strategies that integrates behavior and emotion in an embodiment of the present invention. Figure 3 This is a structural block diagram of the automatic switching system for multi-user interaction response strategies that integrates behavior and emotion, as described in an embodiment of the present invention. Figure 4 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0022] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0024] First, a brief description of this plan: This invention proposes an automatic switching method for multi-user interaction response strategies that integrates behavior and emotion. This method aims to address the problems of high response latency, high false positive rate, and poor user experience inherent in traditional methods when handling concurrent multi-user commands. By introducing a non-sliding window event mechanism, a four-dimensional conflict assessment model (temporal, spatial, semantic, and emotional), a dynamic weight allocation mechanism, multimodal emotion recognition technology, and a conflict detection weight adjustment method that supports scene adaptation, the system can achieve intelligent perception and response scheduling of multi-user behavior in complex scenarios, significantly improving the system's interaction efficiency and personalized service level.

[0025] The core technology solution consists of the following key components: (1) Event-driven real-time triggering mechanism: Instead of relying on fixed time windows for batch processing, conflict detection is triggered based on the user's actual behavior events (such as voice ending, gesture action, spatial movement, priority change, etc.).

[0026] (2) Multimodal emotion recognition technology: integrates multiple modal information such as voice, facial expression, and posture to accurately identify the user's emotional state and use it as a conflict level correction factor to optimize response strategy.

[0027] (3) Four-dimensional conflict assessment model: Combining time synchronization score, spatial overlap score, semantic similarity and emotional conflict score, a comprehensive and accurate conflict assessment model is constructed to improve the accuracy of the system's conflict judgment.

[0028] (4) Scene Adaptation and Dynamic Weight Allocation Mechanism: The system automatically adjusts the weights of each dimension based on different scenarios, user identities, historical preferences, and emotional states to improve the system's adaptability and intelligence. In addition, the system can dynamically adjust the weight parameters of conflict detection through data-driven methods according to the characteristics of the current scenario (such as exhibition guidance, shopping mall guidance, medical companionship, etc.) to adapt to changes in needs under different scenarios.

[0029] like Figure 1 and Figure 2 As shown in the embodiment of the present invention, an automatic switching method for multi-user interaction response strategies that integrates behavior and emotion includes the following steps: S1: Real-time collection of behavioral data from multiple users in the current environment via microphone array, camera, and depth sensor. The behavioral data includes voice signals, image sequences, and spatial location information. S2: Based on behavioral data, detect behavioral events of each user, including voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject; In step S2, the behavioral event also includes a user priority change event, where the priority is determined based on at least one of the user's identity, historical interaction frequency, and current emotional state.

[0030] It should be noted that the traditional time window mechanism has significant limitations when handling concurrent instructions from multiple users: it relies on pre-set time intervals to cache user input and process it uniformly. However, in real-world applications, user interactions are often irregular and unpredictable. This fixed batch processing method may not only lead to response delays but also cause unnecessary waste of computing resources.

[0031] To address the aforementioned issues, an event-driven real-time triggering mechanism is proposed. Its core idea is to move away from relying on a fixed time window length and instead trigger conflict detection and response decisions based on actual user behavior events (such as the end of a voice message, gestures, spatial movement, etc.). This approach significantly improves the system's response speed and flexibility, ensuring timely responses to every critical event and avoiding delays caused by waiting window periods.

[0032] To achieve comprehensive perception of user behavior, this method requires real-time monitoring of several typical event types, including voice start and end, gestures (waving, pointing, nodding, etc.), and spatial position changes (approaching, moving away from, turning towards the robot). Whenever these events occur, the corresponding conflict detection and response scheduling process will be immediately initiated to ensure that every user's needs are addressed promptly.

[0033] This method monitors various user behavioral events and converts them into identifiable signals. Specifically: Voice event detection: The system captures the user's voice signal using a microphone array and uses a Voice Activity Detection (VAD) algorithm to determine whether the user has started or stopped speaking. Voice start detection: When the VAD algorithm detects that a continuous voice signal exceeds a certain threshold, it is marked as a "voice start" event. Voice end detection: When the voice signal strength remains below a certain threshold for a period of time, it is marked as a "voice end" event.

[0034] Gesture detection: Utilizing cameras and depth sensors to capture user body movements, and combining this with computer vision algorithms to analyze the user's gesture characteristics. Common gesture recognition: Including basic gestures such as waving, pointing, and nodding; Gesture recognition for specific scenarios: For different application scenarios (such as shopping mall guides, medical companions, etc.), customized recognition of specific gestures (such as "confirm," "cancel," etc.) can be achieved.

[0035] Spatial position change detection: The system acquires the user's real-time spatial coordinates via camera or positioning device and calculates their azimuth angle θ relative to the robot. Azimuth angle calculation: Using the robot's front as the reference direction (0°), the system calculates the angle difference between each user and the robot. Position change judgment: When the user's azimuth angle change exceeds a certain threshold, a "position change" event is triggered.

[0036] S3: In response to the triggering of any behavioral event, obtain the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on voice signals, image sequences and spatial location information; In step S3, multimodal emotion recognition technology is used to identify the current emotional state and intensity of each user, including: Acoustic features are extracted and classified from the speech signal to obtain the first emotion category and the first emotion intensity; Facial feature point detection and expression classification are performed on the image sequence to obtain the second emotion category and the second emotion intensity; Skeletal keypoint detection and pose / action classification are performed on image sequences and / or depth sensor data to obtain the third emotion category and the third emotion intensity. Based on the first, second, and third emotional intensities, the final emotional intensity is determined through weighted fusion or decision-level fusion, and the final emotional state is determined based on the first, second, and third emotional categories.

[0037] Specifically, once an event is detected, the corresponding conflict detection and response scheduling process will be executed immediately. The specific steps are as follows: Get the latest command status of all active users: including user ID, command content, semantic features, spatial location, and most recent interaction time; Construct a real-time conflict graph: Treat each user as a node in the graph, establish edges between each pair to represent potential conflict relationships, and the edge weight is the conflict intensity (combining semantic similarity and spatial perspective difference).

[0038] Conflict level assessment: Cluster analysis is performed on the conflict map to identify high-conflict subgraphs, and response strategies (such as pause, prompt, parallel response, etc.) are determined based on the comprehensive conflict level.

[0039] Automatic switching of response strategies: Select the optimal response method (such as parallel response, prompt confirmation, sequential response, or background intervention) based on the conflict assessment results.

[0040] Furthermore, emotion recognition refers to determining a user's current emotional state by analyzing their multimodal information (such as voice, facial expressions, and gestures). In human-computer interaction, a user's emotional state often influences their interaction intentions and the urgency of their needs. Therefore, accurately identifying user emotions is crucial for improving the system's intelligence and user experience.

[0041] To achieve a comprehensive understanding of users' emotional states, a multimodal emotion recognition technology was designed, which mainly includes the following aspects: (1) Voice emotion recognition The user's emotional state is determined by analyzing the characteristics of their voice signal (such as tone, speech rate, and volume changes). The specific implementation steps are as follows: Speech signal preprocessing: Performing preprocessing operations such as noise reduction and frame segmentation on the acquired speech signal; Feature extraction: Extract feature vectors such as spectrogram and Mel frequency cepstral coefficients from each frame of speech signal; Emotion classification: Deep learning models (such as CNN, LSTM, etc.) are used to classify the extracted feature vectors and output the emotion category (such as anger, anxiety, calm, happiness, etc.) and emotion intensity.

[0042] (2) Facial expression recognition The system determines a user's emotional state by analyzing facial expressions (such as smiling, frowning, and widening eyes). The specific implementation steps are as follows: Facial image capture: using a camera to capture the user's facial image; Facial feature point detection: using visual tools to detect key facial features (such as eyes, nose, mouth, etc.). Facial Expression Classification: Based on the positional change patterns of feature points and combined with a pre-trained facial expression recognition model, the system outputs the emotion category and emotion intensity.

[0043] (3) Posture and motion recognition The user's emotional state is determined by analyzing their body language characteristics (such as arm movements, leaning forward, etc.). The specific implementation steps are as follows: Attitude data acquisition: Capture the user's 3D attitude data using a depth sensor; Skeletal key point detection: Use visual tools to detect the main skeletal key points of the user's body (such as head, shoulders, elbows, wrists, etc.). Action classification: Based on the change patterns of skeletal key points and combined with a pre-trained action recognition model, the emotion category and emotion intensity are output.

[0044] S4: Construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pairs; the four-dimensional conflict assessment model shall include at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension, and emotional conflict dimension; To achieve accurate identification and intelligent response to concurrent commands from multiple users, emotion recognition is introduced as an independent and crucial dimension. A pioneering TSLE conflict assessment model, integrating four dimensions—time (T), space (S), semantics (L), and emotion (E)—is designed to achieve more accurate conflict level judgment, more human-centered response strategy selection, stronger user emotional adaptability, and a more natural enhanced interactive experience. The specific definitions are as follows: Time (T): Synchronization of user speaking times, i.e., the time difference between speaking times; Space (S): Difference in user spatial location, i.e., azimuth difference; Semantic (L): Instruction semantic similarity, i.e. semantic embedding vector similarity; Emotion (E): The user's emotional state, i.e., the result of multimodal emotion recognition; Based on the information from the above four dimensions, a comprehensive conflict level assessment formula can be constructed: ; in, Indicates user With users The overall conflict level between them (0~1). This indicates the time synchronization score (the smaller the time difference, the higher the score). The score represents the spatial overlap (the smaller the azimuth difference, the higher the score). Represents semantic similarity (output of models such as BERT). This indicates the emotional conflict score (the more intense the emotion, the higher the score). , , and The corresponding dynamic weights are automatically adjusted based on the scenario, user identity, historical preferences, etc.

[0045] For example, time synchronization score Calculate the time difference between two users' messages. And normalize it into a time synchronization score: ; in, It is a preset time threshold, usually set to about 1 second. The smaller, The closer it is to 1, the higher the time synchronization between the two.

[0046] Spatial overlap score Calculate the azimuth difference between two users. And normalize it into a spatial overlap score: ; in, It is the difference in azimuth angle between the two users and the robot. The larger, The closer it is to 1, the lower the spatial overlap between the two.

[0047] semantic similarity Use a pre-trained language model (such as BERT) to extract the semantic feature vector of each instruction and calculate the pairwise similarity: ; in , users respectively and The semantic embedding vector. The closer the value is to 1, the higher the semantic similarity between the two.

[0048] To incorporate the results of emotion recognition into the conflict assessment model, emotion conflict scoring is also included. The emotional conflict score is calculated using the following formula: ; In the formula, Indicates user The emotional intensity value (0~1). Indicates user The intensity of emotion, This represents the average emotional intensity weight. Indicates the weight of emotional differences. Indicates user With users Emotional conflict rating between them.

[0049] It should be noted that the more intense the emotions and the greater the emotional differences, the higher the score, and such conflicts will be dealt with first.

[0050] The role of emotion recognition in this method is as follows: First, it serves as a conflict level correction factor: users with agitated emotions are more likely to cause system misjudgment when their commands conflict, and should be prioritized. Second, it serves as a basis for adjusting response strategies: when users are anxious or angry, the system should adopt a soothing response. Third, it serves as a basis for prioritizing users: users with abnormal emotions (such as urgency or anger) should receive higher response priority. Finally, it enhances personalized interaction: the robot's tone, speech rate, and feedback method are adjusted according to the user's emotions to improve friendliness.

[0051] S5: Dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model based on the current application scenario, user identity, and / or historical interaction data. Based on the comprehensive conflict level after weight adjustment, automatically select and switch to the target response strategy from a variety of preset response strategies to respond to multi-user interactions.

[0052] In step S5, the preset multiple response strategies include at least: parallel response strategy, sequential response strategy, prompt confirmation strategy, pause execution strategy, and background intervention strategy; when the overall conflict level is higher than the first threshold, the prompt confirmation strategy, pause execution strategy, or background intervention strategy is selected; when the overall conflict level is between the first threshold and the second threshold, the sequential response strategy is selected; when the overall conflict level is lower than the second threshold, the parallel response strategy is selected.

[0053] In step S5, the weights are dynamically adjusted according to the current application scenario, including: Identify the current application scenario based on environmental images, number of users, and interactive content characteristics; Based on the preset scene weight mapping table, initial weights are assigned to each dimension in the four-dimensional conflict assessment model.

[0054] In step S5, when automatically selecting and switching to the target response strategy, the response method is also adjusted based on the current emotional state; when the user's emotional state is identified as anxious or angry, a soothing tone of voice and / or a priority response is used; when the user's emotional state is identified as calm or happy, a regular tone of voice and / or additional service options are provided.

[0055] Specifically, in order to adapt to the interaction needs in different scenarios, this method supports dynamic adjustment of the weights of four-dimensional features and automatically switches the engine to select the optimal response method through response strategy.

[0056] Dynamic weight allocation mechanism: Based on different scenarios, user identity, historical preferences and emotional state, the weight of each dimension is automatically adjusted to improve adaptability and intelligence.

[0057] Scenario-Adaptive Weight Configuration: Operations personnel can set initial weights based on different application scenarios (such as exhibition guidance, shopping mall guidance, medical companionship, educational support, etc.). For example, in the medical companionship scenario, user emotions are more important, so the emotional dimension is given higher weight. The model is trained through user feedback (such as waiting time, number of interruptions, satisfaction ratings), learns online, and dynamically optimizes the weights; weight adjustments are triggered based on indicators such as the current number of users, spatial distribution density, time synchronization, and emotional state.

[0058] Based on the conflict map assessment results, the most suitable response strategy is automatically selected, enabling automatic switching of response strategies: High conflict level (>0.7): prompt for confirmation, suspend execution, or require background intervention.

[0059] Medium conflict level (0.4~0.7): respond sequentially, and guide users to speak in order via voice.

[0060] Low conflict level (<0.4): parallel response, personalized feedback.

[0061] Furthermore, this method will dynamically adjust the response strategy based on the user's emotional state: For example: Calm and friendly: Processes according to standard strategies and supports parallel responses; Anxious and impatient: Prioritize responses, reduce waiting time, and use a gentle tone of voice; Anger, agitation: prompt for confirmation to avoid escalation of conflict; backend intervention may be necessary. If you are happy or satisfied: you can extend the interaction period and provide suggestions for additional services.

[0062] In summary, this invention patent proposal organically combines an event-driven real-time triggering mechanism, multimodal emotion recognition technology, a four-dimensional conflict assessment model, and a dynamic weight allocation mechanism to achieve intelligent perception and response scheduling of multi-user behaviors in complex scenarios. This method improves the interactive intelligence and response efficiency of service robots, and also enhances their adaptability and user satisfaction in complex scenarios.

[0063] The event-driven mechanism replaces the traditional fixed-time-window mechanism, avoiding the response delay caused by batch processing; based on user behavior events to trigger responses, it significantly improves the system's response speed and efficiency to concurrent interactions of multiple users, and is especially suitable for densely populated scenarios such as exhibitions and shopping malls.

[0064] A four-dimensional conflict assessment model based on time, space, semantics, and emotion is constructed to comprehensively evaluate the degree of conflict between users from multiple dimensions. Compared with traditional single-dimensional judgment methods (such as semantics or time only), this method can more accurately identify potential conflicts in multi-user interactions and improve the accuracy and rationality of system decisions.

[0065] Integrating multiple modalities of emotion recognition technology, including voice, facial expressions, and gestures, it accurately perceives users' emotional states. Emotional states serve as conflict level correction factors to optimize response strategies and achieve an emotion-oriented response mechanism. In emotionally sensitive scenarios such as medical care and educational companionship, it can significantly enhance the friendliness of interactions and user satisfaction.

[0066] The system automatically identifies the current scene through scene recognition algorithms; it automatically adjusts the conflict detection weights and response strategies according to different scenes (such as exhibition guidance, shopping mall guidance, medical care, and educational companionship); and achieves a true "scene adaptive" response mechanism, enabling the system to have stronger generalization capabilities and application scalability.

[0067] This invention also provides an automatic switching system for multi-user interactive response strategies that integrates behavior and emotion. This system is used to implement the above-described method embodiments, and details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0068] like Figure 3 As shown, Figure 3 This is a structural block diagram of the multi-user interaction response strategy automatic switching system that integrates behavior and emotion in this invention. The system includes: The data acquisition module 101 is configured to collect behavioral data of multiple users in the current environment in real time through a microphone array, camera and depth sensor. The behavioral data includes voice signals, image sequences and spatial location information. The behavior event detection module 102 is configured to detect the behavior events of each user based on the behavior data. The behavior events include voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject. The emotion recognition module 103 is configured to, in response to the triggering of any of the aforementioned behavioral events, acquire the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on the voice signal, image sequence, and spatial location information. The conflict assessment module 104 is configured to construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pair; the four-dimensional conflict assessment model includes at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension and emotional conflict dimension. The strategy control module 105 is configured to dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model according to the current application scenario, user identity and / or historical interaction data, and automatically select and switch to the target response strategy from a variety of preset response strategies based on the comprehensive conflict level after weight adjustment, in order to respond to the multi-user interaction.

[0069] This system is used to implement the methods described in the above embodiments, so it will not be described again here.

[0070] like Figure 4As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: S1: Real-time collection of behavioral data of multiple users in the current environment through microphone array, camera and depth sensor, the behavioral data including voice signals, image sequences and spatial location information; S2: Based on the behavioral data, detect the behavioral events of each user, including voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject; S3: In response to the triggering of any of the aforementioned behavioral events, obtain the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on the voice signal, image sequence and spatial location information; S4: Construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pair; the four-dimensional conflict assessment model includes at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension, and emotional conflict dimension; S5: Dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model according to the current application scenario, user identity and / or historical interaction data, and automatically select and switch to the target response strategy from a variety of preset response strategies based on the comprehensive conflict level after weight adjustment, in order to respond to the multi-user interaction.

[0071] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically switching multi-user interaction response strategies that integrates behavior and emotion, characterized in that, Includes the following steps: S1: Real-time collection of behavioral data of multiple users in the current environment through microphone array, camera and depth sensor, the behavioral data including voice signals, image sequences and spatial location information; S2: Based on the behavioral data, detect the behavioral events of each user, including voice start / end events, preset gesture events, and spatial position change events relative to the interactive subject; S3: In response to the triggering of any of the aforementioned behavioral events, obtain the current interaction state of all active users, and use multimodal emotion recognition technology to identify the current emotional state and intensity of each user based on the voice signal, image sequence and spatial location information; S4: Construct a four-dimensional conflict assessment model between any two users and calculate the comprehensive conflict level between the user pair; the four-dimensional conflict assessment model includes at least the time synchronization dimension, spatial overlap dimension, semantic similarity dimension, and emotional conflict dimension; S5: Dynamically adjust the weights of each dimension in the four-dimensional conflict assessment model according to the current application scenario, user identity and / or historical interaction data, and automatically select and switch to the target response strategy from a variety of preset response strategies based on the comprehensive conflict level after weight adjustment, in order to respond to the multi-user interaction.

2. The method according to claim 1, characterized in that, In step S2, the behavioral event also includes a user priority change event, wherein the priority is determined based on at least one of user identity, historical interaction frequency and current emotional state.

3. The method according to claim 1, characterized in that, In step S3, the step of identifying the current emotional state and intensity of each user using multimodal emotion recognition technology includes: Acoustic features are extracted and classified from the speech signal to obtain a first emotion category and a first emotion intensity; Facial feature point detection and expression classification are performed on the image sequence to obtain a second emotion category and a second emotion intensity; The image sequence and / or depth sensor data are subjected to skeletal keypoint detection and posture / motion classification to obtain a third emotion category and a third emotion intensity. Based on the first emotion intensity, the second emotion intensity, and the third emotion intensity, the final emotion intensity is determined by weighted fusion or decision-level fusion, and the final emotion state is determined according to the first emotion category, the second emotion category, and the third emotion category.

4. The method according to claim 1, characterized in that, In step S4, the overall conflict level is calculated using the following formula: ； in, Indicates user With users The overall level of conflict between them Indicates the time synchronization score. Indicates spatial overlap score, Indicates semantic similarity. Indicates emotional conflict rating. , , and The corresponding weights.

5. The method according to claim 4, characterized in that, The emotional conflict score is calculated using the following formula: ； In the formula, Indicates user The intensity of emotion, Indicates user The intensity of emotion, This represents the average emotional intensity weight. Indicates the weight of emotional differences. Indicates user With users Emotional conflict rating between them.

6. The method according to claim 1, characterized in that, In step S5, the preset multiple response strategies include at least: parallel response strategy, sequential response strategy, prompt confirmation strategy, pause execution strategy, and background intervention strategy; when the overall conflict level is higher than the first threshold, the prompt confirmation strategy, pause execution strategy, or background intervention strategy is selected; when the overall conflict level is between the first threshold and the second threshold, the sequential response strategy is selected; when the overall conflict level is lower than the second threshold, the parallel response strategy is selected.

7. The method according to claim 1, characterized in that, In step S5, dynamically adjusting the weights according to the current application scenario includes: Identify the current application scenario based on environmental images, number of users, and interactive content characteristics; According to the preset scene weight mapping table, initial weights are assigned to each dimension in the four-dimensional conflict assessment model.

8. The method according to claim 1, characterized in that, In step S5, when automatically selecting and switching to the target response strategy, the response method is also adjusted based on the current emotional state; when the user's emotional state is identified as anxious or angry, a soothing tone of voice and / or a priority response is used; when the user's emotional state is identified as calm or happy, a normal tone of voice and / or additional service options are provided.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the automatic switching method for multi-user interaction response strategies that integrates behavior and emotion as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the automatic switching method for multi-user interactive response strategies that integrates behavior and emotion as described in any one of claims 1 to 8.

Images