A method and system for active sensing based tabletop robot interaction

Abstract

The application discloses a tabletop robot interaction method and system based on active sensing. The method comprises the following steps: collecting user visual and voice information to obtain user state data; based on the data, selecting corresponding motion primitives from a preset motion semantic library and generating motion control instructions containing social distance parameters, the motion semantic library containing multiple motion primitives and being mapped to semantic meanings, applicable scenes and parameter configurations, and when a touch primitive is selected, generating motion parameters based on four-dimensional strategies of distance control, speed adjustment, posture design and timing selection; controlling the robot to execute the motion instructions, synchronously outputting visual and / or audible information matched with interaction intentions, continuously detecting user avoidance behaviors during execution of the touch primitive, and if detection is performed, stopping execution and switching to a withdrawal motion primitive. The application makes physical motion of the robot become an interaction language carrying emotions and intentions, and realizes low-intrusive active contact through four-dimensional strategies and avoidance detection.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to a desktop robot interaction method and system based on active perception. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, desktop robots are gradually moving from industrial settings into homes and offices, taking on functions such as companionship, reminders, education, and entertainment. However, existing desktop robots still have the following shortcomings in terms of interaction methods.

[0003] First, the interaction paradigm is primarily passive. Existing desktop robots typically require users to trigger interactions through specific wake words or touch operations. The robots themselves lack the ability to proactively perceive the user's state and initiate interactions, resulting in a human-machine relationship that remains at the level of "tool-like interaction," failing to establish a genuine sense of companionship. Second, motion control functions are limited. The motion control of existing desktop robots is mostly limited to a few degrees of freedom, such as head rotation and screen angle adjustment. Their physical movements cannot be flexibly and richly expressed according to the interaction scenario and user state, making it difficult to form multi-layered interactive feedback in conjunction with voice and vision. Furthermore, multimodal interaction lacks synergy. Although some existing solutions involve the collection of information from multiple channels such as voice and vision, there is a lack of deep integration and synergistic expression between these channels. Vision, voice, and motion often work independently, making it difficult to form a unified and natural interactive experience. Finally, there is a lack of respect for user boundaries. Existing robots often adopt fixed interaction methods without distinguishing between interaction scenarios and user states, especially lacking fine-grained control strategies in terms of physical contact, which can easily cause users a sense of intrusion or discomfort.

[0004] Therefore, how to transform desktop robots from passive executors into proactive companions with active perception, motor expression, and low-intrusion interaction capabilities is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a desktop robot interaction method and system based on active perception. By deeply integrating a motion semantic library, social distance parameters, and multi-dimensional control strategies for tentative contact with multimodal perception and collaborative output, the physical motion of the desktop robot itself becomes an interactive language that can carry emotions and intentions.

[0006] In a first aspect, the present invention provides a desktop robot interaction method based on active perception, including a multimodal perception step, a motion intention mapping step, and a multimodal collaborative output step; Multimodal perception steps: Collect the user's visual and voice information to obtain user status data; Motion intent mapping steps: Based on the user state data, select the corresponding motion primitives from the preset motion semantic library, and generate motion control instructions containing social distance parameters according to the current interaction scenario; The motion semantic library contains a variety of motion primitives, each of which is mapped to a preset semantic meaning, applicable scenario, and motion parameter configuration. The motion parameter configuration includes at least motion speed and / or distance from the user. When the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design and timing selection; Multimodal collaborative output steps: Control the desktop robot to execute the motion control command, and synchronously output visual and / or auditory information that matches the current interaction intention; During the execution of the motion control command corresponding to the touch primitive in the multimodal collaborative output step, the method further includes: continuously detecting user avoidance behavior; if user avoidance behavior is detected, stopping the execution of the current motion control command and switching to a preset withdrawal motion primitive.

[0007] Secondly, the present invention provides a desktop robot interaction system based on active perception, comprising: The multimodal perception module is used to collect the user's visual and voice information to obtain user status data; The motion intent mapping module is used to select the corresponding motion primitives from the preset motion semantic library based on user state data, and generate motion control instructions containing social distance parameters according to the current interaction scenario. The motion semantic library contains a variety of motion primitives, each of which is mapped to a preset semantic meaning, applicable scenario, and motion parameter configuration. When the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design and timing selection; The tentative contact control module is used to continuously detect user avoidance behavior during the execution of the motion control command corresponding to the touch primitive by the multimodal collaborative output module. If user avoidance behavior is detected, the execution of the current motion control command is stopped and switched to the preset withdrawal motion primitive. A multimodal collaborative output module is used to control the desktop robot to execute the motion control commands and synchronously output visual and / or auditory information that matches the current interaction intent; The decision module is used to receive the user status data, determine the current interaction intent according to the preset intervention strategy rules, and send the interaction intent to the motion intent mapping module and the multimodal collaborative output module.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the first aspects and further embodiments above.

[0009] Compared with existing technologies, this invention has the following advantages: It acquires the user's state through a multimodal perception step, maps the interaction intent into semantically meaningful motion primitives and social distance parameters using a motion semantic library, and controls the robot's physical motion and outputs visual and auditory information synchronously in the multimodal collaborative output step. This transforms the desktop robot from a passive responder to an active perceiver and interactive one, with its physical motion itself becoming an interactive language carrying emotions and intentions, forming a unified multimodal expression together with vision and hearing. Simultaneously, it generates motion parameters for touch primitives through a four-dimensional strategy and continuously detects and withdraws user avoidance behavior during execution, achieving low-intrusive active contact. While expressing care, it fully respects user boundaries, enhancing the safety and naturalness of the interaction. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of the overall architecture of a desktop robot active interaction system according to an embodiment of this application is shown; Figure 2 A flowchart illustrating an example of a desktop robot interaction method based on active perception according to an embodiment of this application is shown; Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0012] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following description is for illustrative purposes only and should not be construed as limiting the scope of the invention. Those skilled in the art can adjust the parameters, thresholds, and specific algorithms in the embodiments according to actual application scenarios without departing from the concept of the present invention, and such adjustments all fall within the protection scope of the present invention.

[0013] It should be noted that in current related technologies, some research attempts to combine multimodal perception with active interaction to improve the interactive intelligence level of desktop robots. For example, some solutions establish a spatial digital twin with the desktop robot's head as the perception center, acquire user event information and determine demand information, initiate corresponding interactive scenarios, and control the head's omnidirectional rotation and projection module for projection interaction. This type of method elevates the desktop robot from passive wake-up to active perception, broadening the interaction triggering methods of desktop robots to some extent; however, its motion control is mostly limited to a few degrees of freedom such as head rotation and screen angle adjustment. Its physical movement itself does not carry interactive semantics and cannot convey emotional information such as attention, respect, and hesitation like human body language. Interactive expression still mainly relies on screen display and voice broadcast.

[0014] With the development of multimodal perception technology, some studies have begun to introduce multi-channel information such as vision, speech, and touch into robot interaction systems. For example, one approach collects multimodal interaction data, including touch, speech, image, and environmental status, through a perception layer. The decision layer performs unified preprocessing of the multimodal data and communicates with the cloud, achieving multimodal collaborative responses such as facial expression display, motion control, and voice playback through a dual-processor architecture. Another approach collects user vital signs through a non-contact vital sign acquisition module and analyzes the user's emotions through a central processing unit. Yet another approach uses a modal input unit to acquire the user's face and posture using a camera and a microphone to acquire speech, while the environmental input unit uses object detection algorithms to determine the scene. These technologies demonstrate that multimodal perception has formed a certain foundation in the field of robot interaction, enabling the acquisition of user state information from multiple channels and the understanding of intent. However, the collaboration between these multiple channels is mainly concentrated at the perception level. At the output expression level, vision, speech, and motion often work independently, failing to form a unified multimodal expression system centered on motion semantics. The naturalness and consistency of the interaction remain limited.

[0015] In the area of ​​robot physical contact, some research has been conducted on the control of human-robot physical contact. Service robots typically employ obstacle avoidance algorithms to prevent accidental collisions with users, and some social robots attempt to perform simple contact behaviors through preset action sequences. However, existing solutions often adopt fixed contact methods without distinguishing between interaction scenarios and user states. In particular, they lack sophisticated strategies for comprehensive control of multiple dimensions such as distance, speed, posture, and timing in physical contact, and they also fail to introduce continuous user avoidance behavior detection and immediate withdrawal mechanisms during contact execution, which can easily cause users a sense of intrusion or discomfort.

[0016] In summary, current technologies have laid a certain foundation in areas such as active interaction of desktop robots, multimodal perception fusion, and human-robot physical contact control. However, these technologies tend to focus more on single-level functional implementation. In active interaction scenarios for desktop robots, low-cost, low-degree-of-freedom motion control struggles to support rich interactive semantics. Multimodal output channels lack motion-centric collaborative expression mechanisms, and physical contact control lacks refined, multi-dimensional safety strategies. Without technical means to integrate motion semantic libraries, adaptive social distance, multimodal collaborative output, and tentative contact safety control into the active interaction process, it is difficult to transform desktop robots from passive executors into proactive companions with semantic motion expression and low-intrusion interaction capabilities. Therefore, how to construct a desktop robot active interaction method centered on semantic motion control, integrating multimodal perception and collaborative output, and possessing low-intrusion tentative contact capabilities remains a technical issue that needs further consideration in the field of active interaction technology for desktop robots.

[0017] The following will combine Figures 1-3 This application provides a detailed description of the desktop robot active interaction method and system.

[0018] in, Figure 1 This is a schematic diagram of the overall architecture of the desktop robot active interaction system of the present invention.

[0019] like Figure 1 As shown, the desktop robot active interaction system 100 in this application adopts a layered architecture design, including three core layers: perception layer 101, decision layer 102, and execution layer 103.

[0020] The perception layer 101 is responsible for multimodal signal acquisition and feature extraction. This layer deploys a multimodal perception module to acquire the user's visual and speech information, obtaining user state data. Specifically, the multimodal perception module includes a visual perception submodule and a speech perception submodule. The visual perception submodule acquires the user's visual information, which may include face images, body images, and scene images, to support subsequent functions such as face recognition, expression recognition, gesture recognition, and object recognition. The speech perception submodule acquires the user's speech information to support subsequent functions such as speech recognition and speech emotion analysis. The perception layer 101 preprocesses and extracts features from the acquired raw signals, outputting user state data to the decision layer 102.

[0021] The decision layer 102 is responsible for intent determination and motion mapping. This layer contains a motion intent mapping module. The motion intent mapping module receives user state data output from the perception layer 101, selects the corresponding motion primitive from a preset motion semantic library, and generates motion control instructions containing social distance parameters based on the current interaction scenario. The motion semantic library contains various motion primitives, each mapped to a preset semantic meaning, applicable scenario, and motion parameter configuration. When the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design, and timing selection. The motion intent mapping module sends the generated motion control instructions to the execution layer 103.

[0022] Execution layer 103 is responsible for motion execution, safety control, and multi-channel output. This layer deploys an exploratory contact control module and a multimodal collaborative output module. The multimodal collaborative output module controls the desktop robot to execute motion control commands generated by the motion intent mapping module and simultaneously outputs visual and / or auditory information matching the current interaction intent. The exploratory contact control module continuously detects user avoidance behavior while the multimodal collaborative output module executes the motion control command corresponding to the touch primitive. If user avoidance behavior is detected, the execution of the current motion control command is stopped and switched to a preset withdrawal motion primitive.

[0023] The three-layer collaborative operation enables the system to acquire the user's visual and vocal state information through the perception layer 101. The motion intention mapping module in the decision layer 102 maps the interaction intention into motion control commands with clear semantic meanings, such as moving closer to express attention, moving back to express respect, and hovering to express waiting. Then, the execution layer 103 controls the robot to execute the motion commands and simultaneously outputs visual and auditory information. At the same time, when performing touch-based interactions, a four-dimensional strategy is used to generate motion parameters and continuously detect avoidance behaviors to ensure interaction safety. As a result, the desktop robot transforms from a passive responder to an active perceiver and active interactor. Its physical movement itself becomes an interactive language that carries emotions and intentions, forming a three-dimensional expression space together with vision and hearing.

[0024] The following will combine Figure 2 The specific implementation methods of each module are described in detail according to the execution steps of the method of the present invention.

[0025] S100: Multimodal sensing steps.

[0026] In this step, visual and voice information of the user is collected to obtain user status data.

[0027] It should be noted that during proactive interaction, desktop robots need to comprehensively perceive the user's identity, behavior, emotions, and intentions to provide a reliable basis for subsequent interaction decisions. Single-modal perception, such as relying solely on vision or solely on voice, is easily interfered with by factors such as lighting and noise in complex home environments and cannot fully reflect the user's state. For example, it is difficult to determine the content and tone of a user's speech based solely on vision, and it is impossible to obtain the user's facial expressions and body movements based solely on voice. Therefore, this step integrates multi-dimensional information from both visual and voice channels to provide a comprehensive foundation for user state perception in subsequent intention mapping.

[0028] In this embodiment, the visual perception submodule acquires the user's visual information, including facial images, human body images, and scene images, through a camera. The voice perception submodule acquires the user's voice information through a microphone array. Preprocessing operations such as time alignment, spatial matching, outlier removal, and unit standardization are performed on the acquired raw signals to form standardized input data.

[0029] In terms of speech perception, the speech perception submodule acquires the user's speech information through a microphone array and performs the following preprocessing operations: It uses microphone array beamforming technology to achieve sound source localization and noise suppression, extracting the speech signal from the target direction from multiple microphone signals and suppressing environmental noise; it uses echo cancellation technology to prevent interference from the robot's own audio playback; and it uses voiceprint recognition technology to verify the user's identity, comparing speech features with the voiceprint models of registered users to assist in user identity confirmation. The preprocessed speech signal can be converted into text through streaming automatic speech recognition, and intent recognition can be performed through natural language understanding to extract user needs. Furthermore, a context management mechanism maintains the coherence of multi-turn dialogues.

[0030] Streaming automatic speech recognition can employ end-to-end model-based Automatic Speech Recognition (ASR) engines, such as speech recognition models based on CTC (Connectionist Temporal Classification) or RNN-T (Recurrent Neural Network Transducer) architectures. These models support real-time output of intermediate recognition results during user speech, reducing interaction latency. Natural language understanding can utilize a joint intent classification and slot filling model based on pre-trained language models (such as BERT or its lightweight variants). This model maps the text output by ASR to predefined intent categories (such as information query, item recognition, rejection, greetings, etc.) while extracting key entity information (such as item name, time, location, etc.). Context management mechanisms can employ dialogue state tracking to maintain the context state of the current dialogue and associate current user input with historical dialogue information in multi-turn dialogues.

[0031] After obtaining the raw signals, feature extraction is performed on the visual information to obtain multiple types of visual perception features, including at least facial expression recognition results and gesture recognition results; feature extraction is performed on the speech information to obtain speech emotion recognition results. Furthermore, the emotion recognition results from the visual channel and the emotion recognition results from the speech channel are fused, and the weights are dynamically adjusted based on the signal quality of each channel to obtain a comprehensive user emotional state.

[0032] Specifically, the emotion recognition results from the visual channel E vis Emotion recognition results from the voice channel speech Perform fusion. Fusion result E fused Calculate using the following formula: E fused = ·E vis + (1- )·E speech .

[0033] The fusion weight α is determined by the following formula: ; Among them, SNR vis and SNR speech Both are signal-to-noise ratios (SNR) on a linear scale (i.e., the ratio of signal power to noise power, dimensionless). SNR of the visual channel. vis This can be determined based on image quality assessment metrics, such as the ratio of the mean to variance of image brightness or a mapping value based on face detection confidence; the signal-to-noise ratio (SNR) of the speech channel. speechThe SNR can be determined based on the ratio of the signal power after microphone array beamforming to the estimated noise power. In a simplified implementation, the SNR... vis It can be directly mapped to the highest confidence value output by the facial expression recognition branch, SNR speech It can be directly mapped to the highest confidence value output by the speech emotion recognition branch, thereby using the existing recognition confidence as a proxy indicator of signal quality and reducing additional computational overhead.

[0034] Regarding speech emotion recognition, analysis can be performed based on the prosodic features of the speech signal, including pitch (fundamental frequency F0), energy (short-time average energy), and duration (speech rate). For example, high pitch, high energy, and rapid speech rate may correspond to excitement or anger, while low pitch, low energy, and slow speech rate may correspond to sadness or fatigue. Furthermore, semantic sentiment analysis can be performed by combining the text content converted from speech recognition, extracting words and sentence structures with emotional tendencies, and combining prosodic and semantic features to obtain the speech emotion recognition results and a quantitative value of emotion intensity.

[0035] After the above processing, the final user status data output in this step may include: identity recognition results, facial expression recognition results and intensity, gesture recognition results, object recognition results, comprehensive emotional state and state change trend.

[0036] This step integrates multi-dimensional information from the visual and voice channels, providing a comprehensive foundation for user state perception in subsequent intent mapping; the dynamic weight fusion strategy based on signal quality can still obtain reliable comprehensive emotion judgment even when the signal quality of multiple channels is inconsistent; and temporal modeling enables the system not only to identify the current state but also to analyze the trend of state changes.

[0037] S200: Motion Intent Mapping Step.

[0038] In this step, based on the user state data obtained by S100, a corresponding motion primitive is selected from a preset motion semantic library, and motion control instructions containing social distance parameters are generated according to the current interaction scenario. The motion semantic library contains various motion primitives, each mapped to a preset semantic meaning, applicable scenario, and motion parameter configuration. The motion parameter configuration includes at least motion speed and / or distance from the user. When the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design, and timing selection.

[0039] It should be noted that the motion control of traditional desktop robots is mostly limited to mechanical movements such as head rotation and screen angle adjustment. Their physical movements themselves do not carry interactive semantics and cannot convey emotional information such as attention, respect, or hesitation like human body language. The core of this step lies in establishing a mapping relationship between interactive intentions and motion parameters through a motion semantics database, enabling the robot's physical movements to become a form of "body language" that can carry emotions and intentions. For example, slowly approaching can express attention and listening, while actively backing away can express respect and giving space, rather than simply performing functional actions of positional movement.

[0040] In this embodiment, the motion intent mapping step can be executed by the motion intent mapping module 220. The motion intent mapping module may internally include a motion planning submodule, a semantic mapping submodule, and a low-level control submodule. The motion planning submodule is responsible for path planning and obstacle avoidance, the semantic mapping submodule is responsible for converting interaction intents into motion control commands, and the low-level control submodule is responsible for executing the motion control commands and achieving precise speed and position control. The specific implementation methods of each submodule are described in detail below.

[0041] The motion semantic library is the core data foundation of the semantic mapping submodule. It stores various motion primitives, each associated with a predefined semantic meaning, applicable scenario, and motion parameter configuration. A motion primitive is the smallest executable unit of motion for a robot, such as approaching, moving backward, touching, swinging, hovering, or circling. Each primitive corresponds to a basic motion pattern. The motion parameter configuration specifies parameters such as the motion speed and / or distance from the user when executing the primitive, making different primitives clearly distinguishable in terms of speed, distance, etc.

[0042] During the mapping process, the current interaction intent is first identified from the user state data output by S100. Interaction intent identification can be based on a preset rule engine: for example, when the probability of "sadness" is highest in the overall emotional state and its intensity exceeds a threshold, and no rejection gesture is detected, it is determined to be a reassurance intent; when a rejection gesture is detected, or a rejection keyword is recognized in speech recognition, it is determined to be a rejection intent. In an optional implementation, interaction intent identification can also employ a deep learning-based intent classification model, using user state data as input features and directly outputting the interaction intent category to improve the accuracy of intent recognition in complex scenarios.

[0043] After identifying the current interaction intent, the system queries the motion semantic library for matching motion primitives and their parameter configuration ranges, and determines the social distance parameters in combination with the current interaction scenario. Finally, it generates motion control instructions that include parameters such as motion primitive type, target speed, target distance, motion amplitude, and duration.

[0044] Regarding the determination of social distance parameters: Social distance parameters are not fixed values, but are dynamically determined based on the type of the current interaction scenario. For example, when the interaction scenario is one of emotional comfort or companionship, the social distance parameter is set to a smaller intimate distance range (0 to 0.5 meters), allowing the robot to accompany the user at close range; when the interaction scenario is one of information retrieval or task assistance, the social distance parameter is set to a moderate personal distance range (0.5 to 1.2 meters); when the interaction scenario is one of receiving unfamiliar visitors or initial interaction, the social distance parameter is set to a larger social distance range (1.2 to 3.0 meters), maintaining a polite and safe distance.

[0045] Special handling for touch primitives. Touch primitives differ fundamentally from other motion primitives: approaching, retreating, swinging, hovering, and circling are all non-contact spatial movements with clear safety boundaries. As long as they don't collide with people or obstacles, they are safe. Touch, however, involves physical contact, crossing the user's physical boundaries. Improper execution can directly lead to user discomfort, aversion, or even injury. Therefore, when the selected motion primitive is a touch primitive, the motion parameter configuration does not use the conventional mapping method but is generated based on a four-dimensional strategy.

[0046] The four-dimensional strategy comprehensively controls the motion parameters of the touch primitives from four dimensions: distance control, speed adjustment, posture design, and timing selection. Distance control maintains the contact distance between the robot and the user at a preset social distance boundary, gradually reducing the distance only when the user does not avoid the contact. Speed ​​adjustment reduces the contact speed to below a preset non-threatening threshold, approaching at an extremely slow speed. Posture design enables the robot to approach in a low posture below the user's line of sight, reducing the sense of threat. Timing selection ensures that contact is only initiated when the user is idle or relaxed. Through comprehensive control of these four dimensions, the intrusiveness of tentative contact is minimized.

[0047] The motion planning submodule adopts a fast path planning algorithm based on RRT, combined with a dynamic obstacle avoidance algorithm to detect obstacles in the environment in real time and adjust the motion path, and generates a continuous and smooth motion trajectory through smooth trajectory optimization technology.

[0048] The underlying control submodule adopts a dual-wheel differential drive control method, which achieves precise speed and position control through PID closed-loop control. It also has position and speed feedback mechanisms to ensure that motion control commands are executed accurately.

[0049] Furthermore, the underlying control submodule can also achieve multi-level smooth control from fine-tuning to large-amplitude movement. Specifically, the system determines the amplitude level of the current movement based on the parameter configuration of the motion primitives. Touch primitives correspond to the fine-tuning level, with a speed less than 0.03 m / s and distance changes at the centimeter level; approach and retreat primitives correspond to the medium level, with a speed of 0.05-0.2 m / s and distance changes at the decimeter to meter level; and circling primitives correspond to the large-amplitude level, with multi-angle movement and a distance maintained at 0.8-1.5 m. The underlying control submodule achieves smooth transitions between different amplitude levels through PID closed-loop control combined with a smooth trajectory optimization algorithm, avoiding sudden speed changes or attitude jumps caused by motion mode switching, ensuring a natural and smooth user experience during interaction.

[0050] Smooth trajectory optimization can be achieved using polynomial curve fitting or Bézier curve interpolation. When the motion primitive changes (e.g., from hovering to approaching), the underlying control submodule generates a smooth velocity transition curve based on the current velocity and the target velocity. The curve's initial slope is the current acceleration, and its final slope is zero. By solving for the polynomial coefficients that satisfy the boundary conditions, continuous velocity changes and no abrupt acceleration are ensured. In one implementation, the velocity transition curve can use a fifth-order polynomial: v(t) = a0 + a1t + a2t² + a3t³ + a4t 4 + a5t 5 By setting boundary conditions such as initial velocity, final velocity, initial acceleration of 0, and final acceleration of 0, the polynomial coefficients are solved to generate a smooth velocity command sequence.

[0051] Through this step, the motion semantic library standardizes and transforms abstract interaction intentions into specific motion control parameters, making the robot's physical motion an interactive language that carries emotions and intentions; the contextual determination of social distance parameters enables the robot to exhibit appropriate spatial behavior in different interaction scenarios; the introduction of the four-dimensional strategy provides a safety mechanism for touch primitives to be comprehensively controlled from four dimensions: distance, speed, posture, and timing, making touch primitives different from other non-contact primitives and achieving low-intrusion active contact.

[0052] S300: Multimodal collaborative output steps.

[0053] In this step, the desktop robot is controlled to execute the motion control commands generated by S200, and visual and / or auditory information that matches the current interaction intent is output synchronously.

[0054] It should be noted that in traditional multimodal interaction, vision, speech, and motion often work independently. For example, when an animation is displayed on the screen, a speech is played through a speaker, and a robot performs a movement, there is a lack of temporal coordination and content association between the three, making it difficult to form a unified and natural interactive experience. This step, by synchronizing the output temporal sequence and associating the content semantically, enables physical motion, visual expression, and auditory expression to convey a consistent interactive intent, forming a unified multimodal expression.

[0055] In this embodiment, visual information includes animations and / or information cards displayed on the screen, as well as lighting effects from ambient lighting. Auditory information includes speech synthesis output and / or background sound effects. The output timing of visual and auditory information is synchronized with the execution timing of motion control commands. Based on the start time of the motion control command, visual and auditory outputs start simultaneously, and when the motion control command is completed, visual and auditory outputs end synchronously.

[0056] The specific content of visual and auditory information is determined based on the semantic meaning of the currently selected motion primitive, ensuring that physical movement, visual expression, and auditory expression convey a consistent interactive intent. For example: when the "approach" primitive is selected, the robot slowly approaches the user, while the display shows a warm-toned greeting animation or information card, TTS speech synthesis delivers caring words, and the ambient light switches to warm lighting effects, with all three channels jointly conveying the intent of "attention and greeting"; when the "retreat" primitive is selected, the robot retreats to a non-attentional area, the display shows a gentle transition animation, TTS delivers farewell words, and the ambient light gradually dims, with all three channels jointly conveying the intent of "respect and goodbye"; when the "touch" primitive is selected, the robot approaches in a very slow, low-profile manner, the screen displays a soothing animation, TTS delivers gentle words, and the ambient light pulses softly, with all three channels jointly conveying the intent of "comfort and companionship".

[0057] This step ensures that the output content and motion semantics are matched to convey consistent interactive intent across the visual, auditory, and motion channels, avoiding the sense of incongruity caused by multi-channel information conflicts; the timing synchronization mechanism coordinates and unifies physical motion with information output.

[0058] S400: Trial contact control step.

[0059] In this step, during the execution of the motion control command corresponding to the touch primitive in the multimodal collaborative output step, user avoidance behavior is continuously detected. If user avoidance behavior is detected, the execution of the current motion control command is stopped and switched to the preset withdrawal motion primitive.

[0060] It should be noted that touch primitives involve physical contact. Even though motion parameters have been carefully generated in the S200 using a four-dimensional strategy, the user's feelings and level of acceptance during the interaction are still dynamic. The user may change their mind at any moment before contact. Therefore, this step continuously monitors user feedback during the execution of touch primitives to ensure that the user always has the "veto power" of interaction. Once any avoidance signal is detected, contact is immediately stopped and the user is withdrawn to a safe distance.

[0061] In this embodiment, the detection of user avoidance behavior is based on a comprehensive judgment of multimodal signals. The coordinates of key points on the user's body, facial expression recognition results, and speech recognition results are acquired in real time. When at least one of the following signals is detected: the user's body recoils, the facial expression turns negative (such as anger, disgust, or fear), or the user issues a rejection voice command (such as "no," "go away," or "stop"), it is determined to be user avoidance behavior.

[0062] Upon detecting user avoidance behavior, immediately cease executing the current motion control command and switch the current motion primitive to either a back or hover primitive. If switched to a back primitive, retreat quickly but smoothly to the social distancing boundary to ensure no secondary startling occurs; if switched to a hover primitive, immediately stop moving and maintain the current fixed position, awaiting further instructions from the user.

[0063] This step, through continuous detection of user avoidance behavior and an immediate withdrawal mechanism, ensures that users always retain the right to veto during the interaction. Once a user exhibits any discomfort or rejection signal, the robot immediately terminates contact and withdraws to a safe distance. Thus, while expressing care, it fully respects user boundaries, achieving low-intrusive proactive contact and enhancing the sense of security and user acceptance in the interaction.

[0064] Regarding the implementation details of establishing a mapping model and generating motion control commands in the motion intent mapping step, in some examples of embodiments of this application, a mapping model from interaction intent to motion parameters is first established. This mapping model defines the motion primitives, semantic meanings, and parameter configuration ranges corresponding to various interaction intents. Based on this mapping model, the current interaction intent identified from the user state data is converted into motion control commands.

[0065] It should be noted that the motion semantic library is the data carrier of the mapping model, while the mapping model is the mathematical abstraction and organizational framework of the motion semantic library. The core function of the mapping model is to establish a standardized transformation relationship from high-level interaction intentions to low-level motion parameters. Interaction intentions are "what the robot should express," such as attention, respect, and comfort; motion parameters are "how the robot should move," such as speed, distance, and amplitude. The mapping model establishes a structured correspondence between the two, making the generation process of motion control commands interpretable and scalable: interpretability is reflected in the clear intentional basis for the selection of each motion primitive, and scalability is reflected in the fact that when adding new interaction intentions, only the corresponding mapping relationship needs to be added to the mapping model.

[0066] In this embodiment, the classification system for interaction intents can be divided according to actual application needs. In one implementation, interaction intents can be divided into four main categories: companionship intents, task intents, comforting intents, and reception intents. Companionship intents correspond to scenarios where users need companionship or attention, such as when users are feeling down or have been still for a long time; task intents correspond to scenarios where users need to query information or identify objects, such as when users display objects or ask for information; comforting intents correspond to scenarios where users need emotional comfort, such as when users exhibit sadness or anxiety; and reception intents correspond to scenarios where visitors arrive or users introduce others. In another implementation, the classification can also be based on the user's emotional dimension. For example, when a user is happy, the interaction intent might be interactive entertainment or sharing joy; when a user is sad, the interaction intent might be emotional comfort or silent companionship; and when a user is angry, the interaction intent might be providing space or calm guidance. Regardless of the classification method used, the mapping model defines the motion primitives, semantic meanings, and parameter configuration ranges corresponding to each interaction intent.

[0067] In the intent recognition stage, the current interaction intent is identified based on the user state data output by S100. In one implementation, a preset rule engine can be used for intent recognition: when the probability of "sadness" is the highest in the overall emotional state and the intensity exceeds a preset threshold (e.g., 0.6), and the gesture recognition result does not detect a rejection gesture (such as waving), it is determined to be a soothing intent; when a rejection gesture is detected by gesture, or when the voice recognition detects rejection keywords such as "no" or "go away," it is determined to be a rejection intent; when it is detected that the user is showing an item to the robot and their gaze is focused on the item, it is determined to be an item recognition intent.

[0068] In another implementation, a pre-trained intent classification model can be used for intent recognition. The input feature vector of this intent classification model includes: a comprehensive emotion state vector (probability distribution and intensity values ​​of each emotion category), gesture recognition results (gesture category encoding), keyword matching tags from the speech recognition results, and interaction context features of the current time window (such as the current distance between the user and the robot, continuous working duration, and recent scene types in historical interaction records). The output of this intent classification model is the probability distribution of each candidate interaction intent, and the interaction intent with the highest probability is taken as the current interaction intent. This intent classification model can employ a multilayer perceptron (MLP) or lightweight convolutional neural network structure, using user state data as input features and labeled interaction intent categories as training labels, and is obtained through supervised training based on historical interaction data or manually labeled data. The training data can be obtained by recording the user state data output by S100 during each interaction in the historical interaction process, and automatically labeling the corresponding interaction intent category based on the user's subsequent feedback behavior (such as accepting the robot's service recommendations, rejecting the robot's proactive interaction, etc.), forming training sample pairs.

[0069] In the motion primitive selection stage, matching motion primitives are queried in the mapping model based on the identified interaction intent. In the mapping model, each interaction intent is associated with one or more candidate motion primitives. When multiple candidate motion primitives exist, further selection can be made based on user historical preferences, current scenario constraints, or interaction strategies. For example, for a soothing intent, the mapping model can associate two candidates: a touch primitive and a proximity primitive. The system determines user preferences based on historical interaction records: if the user has historically shown a high acceptance of the touch primitive (i.e., the proportion of no avoidance behavior detected during touch primitive execution is higher than a threshold), the touch primitive is selected first; if the user has historically shown a low acceptance of the touch primitive, the proximity primitive is selected. After selecting a motion primitive, the parameter configuration range of that motion primitive is obtained from the mapping model, and the specific motion control command is determined in conjunction with the current social distance parameters.

[0070] By employing the above method, a structured mapping model between interaction intents and motion parameters is established. This model standardizes and converts the current interaction intent identified from user state data into motion control commands, enabling the motion decision-making process to move beyond hard-coded rule sets and instead rely on a scalable and maintainable mapping model. Whether using a rule engine or a pre-trained intent classification model, the mapping model provides a structured transformation relationship from intent to motion primitives, allowing the robot to exhibit appropriate motion behavior in different scenarios and providing a flexible extension framework for new interaction scenarios and motion primitives.

[0071] Regarding the specific implementation of converting the current interaction intent identified from user state data into motion control commands based on the mapping model, in some examples of embodiments of this application, after identifying the current interaction intent using the above-described method, the mapping model selects the corresponding motion primitives and their parameter configurations according to the different identified interaction intents.

[0072] When the system detects that the user needs companionship, it selects the "approach" verb, moving at a speed within the first speed range (0.05 to 0.1 m / s) and maintaining a preset social distance from the user. The semantic meaning of the "approach" verb is to express attention and listening, allowing the robot to physically move closer to the user and convey care through voice and visual output. For example, in Example 1 (morning wake-up scenario): the system detects that the user has woken up and is within the 06:00 to 09:00 time window, determining it as a companionship intention, and selects the "approach" verb. The robot moves from its standby position to approximately 0.8 meters from the user at a slow speed of about 0.08 m / s, while simultaneously displaying a morning greeting and schedule information on the screen, and providing voice announcements of the day's weather and schedule reminders.

[0073] When the recognition result indicates that the user refuses or needs to be alone, the robot selects the backtrack primitive, moving at a speed within the second speed range (0.1 to 0.2 m / s) and moving away from the user to a distance beyond the user's attention. The semantic meaning of the backtrack primitive is to provide space and express respect, prompting the robot to proactively create distance and avoid making the user feel pressured.

[0074] When the identification result is tentative reassurance, the robot selects the touch primitive, moving at a speed within the third speed range (0.01 to 0.03 m / s) and maintaining a safe distance from the user. This safe distance is the preset minimum distance between the robot and the user before physical contact, ensuring that the robot will not cross the safe distance without explicit acceptance from the user. Taking Example 4 (emotional reassurance scenario) as an example: the system detects that the user is in a state of sadness through multimodal emotion fusion, determines it as a reassurance intention, and selects the touch primitive. The robot first initiates a four-dimensional strategy at a social distance boundary of approximately 0.5 meters, gradually approaching at an extremely slow speed of less than 0.03 m / s, while continuously detecting whether the user exhibits avoidance behavior.

[0075] It should be noted that the lower limit of the first speed range (0.05 m / s) is greater than the upper limit of the third speed range (0.03 m / s), and the lower limit of the second speed range (0.1 m / s) is greater than the upper limit of the first speed range (0.1 m / s), creating a clear gradient in speed among the three core primitives. This gradient design of speed ranges has clear technical significance: the touch primitive executes at an extremely slow speed, giving the user ample time to react and refuse, reflecting maximum caution and respect; the approach primitive executes at a slow but significantly faster speed than the touch primitive, expressing proactive attention while remaining gentle; and the back primitive executes at a medium speed, ensuring timely exit from the user's space and avoiding hesitation that could inconvenience the user.

[0076] Finally, based on the selected motion primitives and their parameter configurations, motion control instructions are generated. These instructions are structured data packets containing at least the following fields: a motion primitive type field, identifying the type of motion primitive being executed (e.g., approach, retreat, touch, swing, hover, or detour); a target speed field, specifying the motion speed value when executing the primitive, determined by combining the speed range corresponding to the selected primitive with the current scene constraints; a target distance field, specifying the target distance between the robot and the user when executing the primitive; a motion amplitude field, specifying the swing amplitude value for the swing primitive or the angle range value for the detour primitive; a motion frequency field, specifying the swing frequency value for the swing primitive; and a duration field, specifying the maximum waiting time for the hover primitive or the maximum contact time for the touch primitive. The motion intent mapping module sends the generated motion control instructions to the multimodal collaborative output module. The multimodal collaborative output module parses the values ​​of each field in the instruction, controls the desktop robot to execute the corresponding physical motion, and determines the specific content of the synchronously output visual and auditory information based on the motion primitive type field.

[0077] For example, when the recognition result indicates that the user needs companionship and selects the "approach" primitive, the generated motion control instruction includes: the value of the motion primitive type field is "approach", the target speed field value is selected from the first speed range (0.05-0.1m / s) (e.g., 0.08m / s), and the target distance field value is determined according to the current social distance parameter (e.g., 0.8m). After receiving this instruction, the multimodal collaborative output module controls the robot to move at a speed of 0.08m / s to a distance of 0.8m from the user, and at the same time outputs a warm-toned greeting animation and caring words according to the semantic meaning of the "approach" primitive.

[0078] In this way, the three core primitives cover the most common interaction needs of desktop robots: approaching to express attention, backing away to express respect, and touching to provide comfort. Furthermore, the gradient design of the speed range ensures that the touching primitive has the lowest intrusiveness in terms of speed, while the backing away primitive has the fastest responsiveness.

[0079] Regarding the extension of motion primitive types in the mapping model, in some examples of embodiments of this application, in addition to the three core primitives mentioned above, the following three supplementary primitives are also included to form a complete six-primitive motion semantic system.

[0080] When the recognition result is an object recognition or judgment, the oscillation primitive is selected, with the oscillation amplitude within the first amplitude range of 3° to 8° (unilateral oscillation amplitude) and the oscillation frequency within the first frequency range of 0.3 to 0.8Hz. The semantic meaning of the oscillation primitive is to express thinking and observation, simulating the natural behavior of humans slightly swaying their bodies when thinking or observing. Taking Example 3 (learning assistant scenario) as an example: the user shows the robot a new book. After the system detects the item through the object recognition branch, it selects the oscillation primitive. The robot oscillates slightly with an amplitude of approximately ±5° and a frequency of approximately 0.5Hz. Combined with the on-screen display of an animation of the item recognition progress and the prompt "Searching for you," it conveys the signal "observing and thinking" to the user.

[0081] When the recognition result indicates waiting for user decision-making, the hovering primitive is selected. Its parameters are configured as fixed position and duration adaptively determined based on user response time (5 to 30 seconds). The semantic meaning of the hovering primitive is to wait for user decision-making, causing the robot to remain still after issuing a query, giving the user ample time to think. For example, during the observation period of Example 2 (focused monitoring scenario), when the user works continuously for more than 30 minutes but less than 45 minutes, the robot executes the hovering primitive, remaining still near the user, and displaying a faint prompt icon on the screen, both reminding the user and not interrupting their work.

[0082] When the recognition result indicates that the robot is observing the user's state, the detour primitive is selected. Its parameters are configured to maintain a distance from the user (0.8 to 1.5 meters) and the movement trajectory revolves around the user from multiple angles. The semantic meaning of the detour primitive is to observe the user's state, enabling the robot to acquire information about the user's activities from multiple angles without intruding on the user's personal space. For example, in Example 2 (focused monitoring scenario), when the system detects a decrease in the user's sitting stability and a wandering gaze, the robot can execute the detour primitive to observe the user's state from different angles, providing more comprehensive information for subsequent intervention decisions.

[0083] By supplementing the above three primitives, the motion semantic library forms a complete six-primitive system including approach, retreat, touch, swing, hover, and detour, which can cover the motion expression needs of desktop robots in various interactive scenarios such as companionship, tasks, comforting, and observation.

[0084] Regarding the method for determining social distance parameters, in some examples of embodiments of this application, the social distance parameters are determined based on the type of the current interaction scenario. First, the type of the current interaction scenario is identified.

[0085] Scene type identification is based on the interaction intent recognition results. Specifically, when the interaction intent recognition result is a comforting or companionship intent, it is mapped to an emotional comforting or companionship scene; when the interaction intent recognition result is a task-related intent (such as object recognition or information query), it is mapped to an information query or task assistance scene; when the user identity recognition result is an unfamiliar user and the interaction intent is a reception intent, it is mapped to a stranger visitor reception or initial interaction scene. The above mapping relationships can be preset in a mapping rule table, and the system determines the type of the current interaction scene by looking up the table based on the interaction intent recognition result and user identity information.

[0086] In one alternative implementation, scene type recognition can also be performed as a parallel step independent of intent recognition. In this approach, the input feature vector of the scene type recognition model includes: user identity information (registered user or unfamiliar user), interaction context features within the current time window (such as the current distance between the user and the robot, ambient lighting conditions, background noise levels, etc.), and recent scene tags from the user's historical interaction records. The model outputs a probability distribution of each candidate scene type, and the scene type with the highest probability is selected as the type of the current interaction scene.

[0087] When the identification result indicates an emotional comfort or companionship scenario, the social distance parameter is set to a preset intimate distance range (0 to 0.5 meters). This distance range is based on the intimate distance concept in Hall's interpersonal distance theory and is suitable for scenarios where the user is feeling down and needs companionship or comfort. In Example 1 (morning wake-up scenario), when the system detects that the user has just woken up and is in a calm mood, the social distance is set at approximately 0.8 meters within the personal distance range; if the system detects that the user is feeling down, the distance can be further reduced to the intimate distance range to express closer companionship.

[0088] When the recognition result is for information query or task assistance scenarios, the social distance parameter is set to a preset personal distance range (0.5 to 1.2 meters). This distance range is suitable for daily information interaction and task assistance, maintaining a sense of closeness without being overly intrusive. In Example 3 (learning assistant scenario), when a user presents an item for recognition, the robot sets the social distance to approximately 1.0 meter, allowing for clear observation of item details without making the user feel pressured.

[0089] When the identification result indicates a scenario involving welcoming unfamiliar visitors or initial interaction, the social distance parameter is set to a preset social distance range (1.2 to 3.0 meters). This distance range is suitable for initial interactions with unfamiliar users, maintaining a polite and safe distance. In Example 5 (Guest Reception Scenario), when the system identifies an unfamiliar visitor, the robot initially maintains a distance at the far end of the social distance range (approximately 2.5 meters), greeting the visitor via voice and screen. The distance is then gradually adjusted as the host introduces the visitor or the visitor approaches.

[0090] It should be noted that the above social distance intervals are defined based on Hall's interpersonal distance theory. This theory divides physical distance in interpersonal interactions into four zones: intimate distance (0 to 0.45 meters), personal distance (0.45 to 1.2 meters), social distance (1.2 to 3.6 meters), and public distance (above 3.6 meters). This embodiment, considering the actual application scenarios and size characteristics of the desktop robot, makes appropriate adjustments to Hall's theory: the intimate distance interval is set to 0 to 0.5 meters (slightly wider than the theoretical value to include the safety boundary when the robot approaches the user), the personal distance interval is set to 0.5 to 1.2 meters, and the social distance interval is set to 1.2 to 3.0 meters. By introducing a mature interpersonal distance theory as the basis for these intervals, the setting of social distance parameters is supported by psychological and behavioral theories, rather than being an arbitrary empirical setting, thus improving the rationality and interpretability of the technical solution.

[0091] In this way, the social distance parameter is no longer a fixed value, but is dynamically determined according to the interaction scenario, enabling the robot to exhibit appropriate spatial behavior in different scenarios and improving the naturalness and comfort of the interaction.

[0092] Regarding the four-dimensional strategy for touching primitives, in some examples of embodiments of this application, the four-dimensional strategy specifically includes the following steps in the process.

[0093] It should be noted that touch primitives differ fundamentally from other motion primitives: approaching, moving backward, swinging, hovering, and circling are all non-contact spatial movements with clear safety boundaries; as long as they do not collide with people or obstacles, they are safe. Touch, however, involves physical contact, crossing the user's physical boundaries. Improper execution can directly lead to user discomfort, aversion, or even injury. Therefore, it is necessary to comprehensively control the motion parameters and execution conditions of touch primitives from multiple dimensions.

[0094] The first dimension, distance control: Maintain the contact distance between the robot and the user at a preset social distance boundary (e.g., 0.5 meters), and continuously detect whether the user exhibits an avoidance signal. Only when the user does not avoid the signal is the contact distance gradually reduced. Specifically, starting from an initial distance d0 (e.g., 0.5 meters), in the i-th step (i=1,2,3,…), if the previous step detected that the user did not exhibit an avoidance signal, then the current distance is set to di = d. i-1 - △d, where △d is the preset step size (which can be 0.2 meters); if an avoidance signal is detected, the robot stops reducing the distance and performs a withdrawal. When the distance is reduced to the preset minimum contact distance (e.g., 0.05 meters), the reduction stops, and preparations are made to perform a physical contact action. Taking Example 4 (emotional comforting scenario) as an example: when the robot detects that the user is in a sad mood and is suitable for contact comforting, the robot first starts the contact process at the 0.5-meter social distance boundary. If the user does not avoid the contact, the distance is gradually reduced to 0.3 meters, 0.1 meters, until the preset minimum contact distance is reached and then the reduction stops.

[0095] The second dimension, speed adjustment: The contact speed is reduced to below a preset non-threatening threshold (e.g., 0.03 m / s). The speed reduction process can adopt an S-shaped speed curve, smoothly transitioning from the current speed to the target low speed. That is, the speed change is gradual at the beginning and end, and slightly faster in the middle, making the entire deceleration process conform to the start and stop characteristics of natural movement, avoiding startling the user due to sudden speed changes. Because the robot approaches at an extremely slow speed, the user has ample time to observe the robot's intentions and react. If the user does not wish to be contacted, they can express their refusal at any time by pulling back, changing their facial expressions, or uttering a refusal voice. Upon detecting the above avoidance signals, the system stops contact and withdraws, thereby reducing the user's tension and discomfort during the contact process.

[0096] The third dimension is posture design: The top of the desktop robot is positioned below the user's eye level to create a low-profile approach. It's worth noting that this low-profile posture is widely considered a non-threatening posture in animal behavior and human interaction, reducing the other party's defensiveness. In one implementation, the robot can achieve this low-profile posture by adjusting its height or body pitch angle; in another, the robot can enhance the expressive effect of the low-profile posture by displaying a docile facial expression on the screen.

[0097] The fourth dimension, timing selection: When a preset idle state condition is detected, the physical contact action corresponding to the touch primitive is executed. The preset idle state conditions include at least one of the following: the user's posture stability is higher than a preset stability threshold (indicating that the user is relaxed rather than tense), the user's gaze deviates from the screen area by more than a preset angle (indicating that the user is not focused on a task), and the user's hand operation frequency is lower than a preset frequency threshold (indicating that the user is not actively operating). By selecting the timing, the contact action is ensured to occur at the time when the user is most likely to accept it, avoiding contact when the user is focused, busy, or tense.

[0098] By comprehensively controlling four dimensions, the intrusiveness of tentative contact is minimized, allowing the touch primitives to express care while giving users ample opportunity to refuse and ensuring safety.

[0099] Regarding the user avoidance behavior detection and withdrawal mechanism, in some examples of the embodiments of this application, user avoidance behavior detection is continuously performed. If user avoidance behavior is detected, the execution of the current motion control command is stopped and switched to a preset withdrawal motion primitive. Specifically, this includes the following steps.

[0100] First, the coordinates of key points on the user's body, facial expression recognition results, and speech recognition results are obtained in real time.

[0101] The aforementioned data can be obtained from the real-time output of the S100 multimodal perception module. During the execution of the touch primitive, the multimodal perception module continuously outputs visual and speech perception results at a preset frequency (e.g., 10 to 30 frames per second), and the tentative touch control module extracts the required data from this output stream. Specifically: body key point coordinates are extracted from the human key point detection results of the gesture recognition branch, including the three-dimensional spatial coordinates (x, y, z) of core torso key points such as the torso center point and shoulder center point, where the z coordinate represents the depth distance of the key point relative to the camera; facial expression recognition results are extracted from the output of the expression recognition branch, including the expression category labels (happy, sad, surprised, angry, disgusted, fearful, calm) and corresponding confidence values ​​recognized in the current frame; speech recognition results are extracted from the output of the speech perception submodule, including real-time text converted by streaming ASR and matching tags based on a preset set of rejection keywords.

[0102] In one alternative implementation, to improve the real-time performance of detection and reduce computational overhead, the tentative contact control module does not independently run the complete visual and speech perception process. Instead, it subscribes to the output data stream of the multimodal perception module and obtains the latest frame results of the three types of data through callbacks or polling. The acquisition frequency can be dynamically adjusted according to the execution stage of the touch primitive: during the distance control stage (when the robot is far from the user), a lower acquisition frequency (e.g., 10 frames per second) can be used; during the gradual reduction of distance stage or before physical contact is about to be performed, the acquisition frequency is increased to 30 frames per second to ensure timely capture of the user's avoidance signals.

[0103] Then, avoidance behavior is determined based on the acquired data. Avoidance behavior is determined based on the following three conditions; if at least one condition is met, the behavior is considered user avoidance behavior: Condition 1, Body Retreat Detection: When the depth position of a user's torso key points changes more than a preset retreat threshold (e.g., 10 cm) within N consecutive frames (e.g., N=5 frames, approximately 0.2 seconds), it is determined to be user avoidance behavior. Body retreat is one of the most instinctive defensive reactions humans have when faced with unwanted approach, and it has high detection reliability.

[0104] Condition 2, Negative Expression Detection: When at least one of the expressions of anger, disgust, or fear appears in the facial expression recognition results, and the confidence level exceeds a preset expression threshold (e.g., 0.6), it is determined to be user avoidance behavior. These three expressions are all negative valence, high arousal emotions, and are highly correlated with avoidance and rejection behaviors.

[0105] Condition 3, Refusal of Speech Detection: When the speech recognition result matches at least one keyword from the preset set of refusal keywords, and the confidence level exceeds the preset speech threshold (e.g., 0.7), it is determined to be user avoidance behavior. The preset set of refusal keywords may include words that clearly express the intention to refuse, such as "don't," "go away," "stop," and "don't touch me," as well as words that express the intention to postpone, such as "wait a minute" and "don't wait."

[0106] Upon detecting user avoidance behavior, immediately cease executing the current motion control command and switch the current motion primitive to either a back or hover primitive. If switched to a back primitive, retreat quickly but smoothly to the social distancing boundary (approximately 1.2 meters) to ensure no secondary startling occurs; if switched to a hover primitive, immediately stop moving and maintain the current fixed position, awaiting further instructions from the user.

[0107] In one alternative implementation, the withdrawal strategy can employ a tiered response mechanism. When a slight avoidance signal is detected (e.g., only condition two is met, and the negative facial expression intensity is low), a hover primitive is executed to temporarily halt the approach, giving the user time to react and confirm. When a strong avoidance signal is detected (e.g., both conditions one and three are met), a backtrack primitive is executed immediately to quickly withdraw to a safe distance. This tiered response can more precisely match the user's level of acceptance, avoiding interaction interruptions due to overreaction.

[0108] Taking Example 4 (emotional soothing scenario) as an example: While approaching the user using touch primitives, the robot monitors the user's state in real time. When the user displays a slight expression of aversion due to sudden discomfort (confidence level 0.65), the system identifies this as avoidance behavior, immediately hovers, and observes the user's reaction. If the user's expression returns to normal, the robot can continue approaching; if the user further withdraws or emits a voice of rejection, the robot immediately retreats to a safe distance of 1.2 meters. Throughout the entire process, the user maintains control of the interaction, and the robot does not forcibly advance the contact process.

[0109] Through the above methods, the multimodal fusion avoidance behavior detection mechanism can comprehensively judge the user's avoidance intention from three channels: body, facial expression and voice, reducing the misjudgment that may be caused by a single channel; the instant withdrawal mechanism ensures that the user always has the right to veto. Once any avoidance signal is detected, the robot immediately stops contact and withdraws to a safe distance, realizing low-intrusion active contact.

[0110] Regarding the specific implementation architecture of the multimodal perception step, in some examples of the embodiments of this application, a multi-task shared backbone network is used to process visual information in parallel to obtain multiple types of visual perception features, including at least facial expression recognition results and gesture recognition results; the visual perception features of consecutive frames are input into a temporal modeling network to obtain the user state change trend.

[0111] It should be noted that desktop robots typically have limited computing resources. Deploying a separate neural network model for each visual perception task (face recognition, expression recognition, gesture recognition, object recognition) would result in excessive computation and insufficient real-time performance. The design concept of a multi-task shared backbone network is to share the same basic feature extraction network for multiple visual tasks, and then branch out different task branches from this network, thereby significantly reducing the overall computational load and meeting the real-time requirements of desktop robots.

[0112] In this embodiment, the backbone network can be a lightweight convolutional neural network, such as MobileNetV3, whose input is a 224×224 pixel RGB image and whose output is a shared feature vector. The shared feature vector contains common visual features in the image (such as edges, textures, shapes, color distribution, etc.), and each task branch performs task-specific feature extraction and inference based on this.

[0113] In one implementation, each task branch is a lightweight sub-network. The face recognition branch is followed by a fully connected layer after the shared feature vector, outputting the face identity classification result; the expression recognition branch is followed by a fully connected layer based on FACS action units after the shared feature vector, outputting the activation intensity of each action unit, and then converted into classification results and intensity values ​​of five basic expressions (happy, sad, surprised, angry, and calm) through a mapping layer; the gesture recognition branch is followed by a keypoint regression head after the shared feature vector, outputting the two-dimensional coordinates of 21 keypoints of the hand; the object recognition branch is followed by a detection head after the shared feature vector, outputting the target bounding box and class confidence. The input of each branch is the same shared feature vector, and the branches are executed in parallel during forward inference without interfering with each other.

[0114] In one alternative implementation, the face recognition branch can employ an improved ArcFace algorithm for face feature extraction and identity classification. The loss function of the ArcFace algorithm is: ; Where N is the batch size, s is the feature scaling factor (typically 64), m is the angle margin (typically 0.5), and y i Let θ be the true class label of the i-th sample. yi Let ∑ be the angle between the L2-normalized feature vector and the target class weight vector, where j is the class index, and ∑ is the angle between the L2-normalized feature vector and the target class weight vector. j≠yi This indicates that for categories other than the true category y i The summation is applied to all other categories except the target category. ArcFace adds an angular margin penalty m to the cosine of the angle between the target categories, which makes the model compress the feature vectors of the same category more tightly in the angular space and push the feature vectors of different categories further apart during training, thereby enhancing the distinguishability between facial features of different identities.

[0115] To enhance robustness under complex conditions, the face recognition branch can integrate an occlusion-robust recognition mechanism. When a user is wearing glasses, a mask, or their face is partially obscured, local feature matching is performed based on the unobscured facial area to maintain high recognition accuracy. Furthermore, the face recognition branch can embed a liveness detection mechanism. By analyzing facial micro-expression changes, eye movements, or requiring the user to perform random actions (such as blinking or nodding), it determines that the captured face image comes from a real user rather than a photo or video, preventing photo spoofing attacks. The expression recognition branch can use the Facial Action Unit (FACS) analysis method, outputting the activation intensity of facial action units through a fully connected layer, and then mapping it to the classification results and intensity values ​​of five basic expressions (happy, sad, surprised, angry, and calm). The gesture recognition branch can use the MediaPipe hand keypoint detection algorithm, outputting the two-dimensional coordinates of 21 key points on the hand, and classifying gestures (nodding, shaking the head, waving, pointing, rejecting, etc.) based on the keypoint coordinates and motion trajectory. The object recognition branch can be based on the YOLOv8 object detection algorithm to output the bounding box, category label and confidence score of the detected object, and supports the recognition of common desktop items (such as books, water cups, mobile phones, stationery, etc.).

[0116] Temporal modeling networks can employ bidirectional LSTM networks. The input to the bidirectional LSTM is a sequence of visual perceptual features from T consecutive frames, which is jointly trained end-to-end with a multi-task shared backbone network during the training phase. Let the sequence of visual perceptual features from the T consecutive frames be {x1, x2, ..., x...}. T}, where x t Let x be the concatenated vector of multi-class visual perception features in frame t. In one implementation, x t The dimensionality depends on the sum of the feature dimensions output by each task branch, for example, a 128-dimensional vector. The hidden layer dimension of a bidirectional LSTM can be set to 64 dimensions, then the comprehensive temporal feature ht of frame t is the concatenation of the forward and backward hidden states, totaling 128 dimensions. Based on these temporal features, trend analysis of state changes can be performed. For example, trend fitting can be performed on the time series of facial expression recognition results to determine whether the user's mood is improving or worsening; anomaly detection can be performed on the time series of key human body points to determine whether the user exhibits abnormal behavior such as prolonged stillness.

[0117] In one alternative implementation, an attention mechanism can be introduced into the multi-task shared backbone network to focus on key regions and improve recognition accuracy. Specifically, a spatial attention module can be introduced in the shared feature extraction stage, enabling the network to automatically focus on the regions in the input image most important to the current task. For example, in face recognition, it automatically focuses on key feature regions such as the eyes and nose; in expression recognition, it automatically focuses on expression feature regions such as the corners of the mouth and eyebrows; and in object recognition, it automatically focuses on the texture and contour regions of the object. By introducing an attention mechanism, the recognition accuracy of each task branch can be improved without significantly increasing the computational load, which is particularly suitable for desktop robot scenarios where image quality is unstable due to changes in user posture and lighting. In another alternative implementation, the temporal modeling network can use a Transformer encoder instead of a bidirectional LSTM, and perform contextual modeling of the feature sequences of consecutive frames through a self-attention mechanism, resulting in better performance in capturing long sequence dependencies.

[0118] Through the above methods, the multi-task shared backbone network realizes the parallel extraction of multiple types of visual perception features, reducing the overall computational load; the temporal modeling network enables the system not only to identify the current state, but also to analyze the trend of state changes, providing more comprehensive temporal dimension information for subsequent motion intention mapping and intervention decisions.

[0119] Regarding the specific implementation of the multimodal collaborative output step, in some examples of the embodiments of this application, visual output and auditory output are simultaneously initiated based on the start time of the motion control command, so that the physical motion and information output of the desktop robot constitute a unified multimodal expression. The specific content of the visual and auditory information is determined based on the semantic meaning of the currently selected motion primitive.

[0120] It should be noted that in traditional multimodal interaction, vision, voice, and motion often work independently. Displaying an animation on the screen, playing a voice message through a speaker, and a robot performing a movement—these three lack temporal coordination and content association, easily creating a sense of disjointedness and unnaturalness for the user. This step, by associating and matching the output content with the semantic meaning of the motion primitives, transforms the multi-channel output from independent information streams into a collaborative expression revolving around the same interactive intent.

[0121] In this embodiment, visual information includes animations and / or information cards displayed on the screen, as well as lighting effects from ambient lighting. Auditory information includes speech synthesis output and / or background sound effects. The output synchronization control method is as follows: based on the start time of the motion control command, visual output and auditory output start at the same time; when the motion control command is completed, visual output and auditory output end synchronously. In an optional implementation, visual output can start 50 to 100 milliseconds before motion execution, utilizing the "preview" effect of vision to give users psychological anticipation of the robot's upcoming physical movements, further enhancing the naturalness of the interaction.

[0122] The correlation and matching relationship between the specific content of visual and auditory information and the semantic meaning of the motion primitives is as follows: When the primitive is selected to be near, the visual channel outputs a greeting animation or information card in a warm color tone (such as warm orange), the ambient light switches to a warm low-frequency pulsating light effect, and the auditory channel broadcasts caring words in a gentle tone through TTS speech synthesis. The three channels work together to convey the intention of "attention and greeting"; when the primitive is selected to be back, the visual channel outputs a soft transition animation, the ambient light gradually dims, and the auditory channel broadcasts farewell words. The three channels work together to convey the intention of "respect and goodbye"; when the primitive is selected to be touched, the visual channel outputs a soft transition animation, the ambient light gradually dims, and the auditory channel broadcasts farewell words. The three channels work together to convey the intention of "respect and goodbye"; when the primitive is selected to be touched, the visual channel outputs a greeting animation in a warm color tone (such as warm orange), the ambient light switches to a warm low-frequency pulsating light effect, and the auditory channel broadcasts farewell words. The three channels work together to convey the intention of "respect and farewell ... The visual channel outputs a soothing animation, and the ambient light pulsates with a soft glow. The auditory channel broadcasts gentle, very slow speech, with all three channels working together to convey the intention of "comfort and companionship." When the swaying primitive is selected, the visual channel outputs a thinking prompt animation, the ambient light maintains a neutral, static lighting effect, and the auditory channel can broadcast inquiries or thoughtful remarks. When the hovering primitive is selected, the visual channel displays a faint prompt icon, the ambient light maintains a static, soft glow, and the auditory channel outputs nothing or only a brief prompt sound. When the detour primitive is selected, the visual channel displays an environmental observation prompt, the ambient light switches to a dynamic color-changing effect, and the auditory channel outputs nothing.

[0123] Furthermore, in one alternative implementation, emotion-adaptive output can be performed based on the user's emotional state. Based on the user's emotional state output by S100, the pitch, speech rate, and volume parameters of the speech synthesis output are adjusted, as are the color, brightness, and frequency of change of the ambient light. Specifically: When sadness is detected in a user, the synthesized voice tone is lowered (fundamental frequency reduced by 10% to 20%), the speech rate is slowed down (reduced by 20% to 30%), and the volume is appropriately reduced (reduced by 10% to 15%). The ambient lighting is adjusted to a warm tone (color temperature 2700 to 3000K) with low-frequency pulsations (frequency 0.2 to 0.5Hz) to create a warm and soothing atmosphere. When happiness is detected in a user, the synthesized voice tone is raised (fundamental frequency increased by 5% to 10%), the speech rate is normal, and the volume is moderate. The ambient lighting is adjusted to a bright color (color temperature 5000 to 6500K) to match the user's pleasant emotional state. When anger is detected in a user, the synthesized voice tone is stable, the speech rate is moderate, and the ambient lighting is adjusted to a cool tone (color temperature 6500 to 8000K) with static lighting effects to soothe and cool down the user.

[0124] In terms of personalized speech synthesis, in addition to adaptive adjustment based on the user's emotional state, personalized synthesis parameters can also be configured based on user preferences recorded in the user's knowledge graph. For example, users can select their preferred voice timbre (such as a gentle female voice, a steady male voice, etc.) and speech rate style (such as standard speech rate, slow and detailed speech rate, etc.) in the initial settings. The system stores these preferences in the user's knowledge graph and automatically applies them in subsequent speech synthesis outputs. In addition, the system can automatically switch voice styles according to different interaction scenarios: a warm and cheerful tone is used in Example 1 (morning wake-up scenario), a gentle and slow tone is used in Example 4 (emotional soothing scenario), and a clear and steady tone is used in the health reminder in Example 2 (focused protection scenario).

[0125] Taking Example 1 (Morning Wake-up Scenario) as an example: After S200 generates a motion control command that approaches the primitive, S300 controls the robot to move from the standby position to a distance of about 0.8 meters from the user at a speed of about 0.08 meters per second. At the same time, the display screen shows a good morning greeting card containing the day's weather and schedule information, the ambient light switches to a warm orange gradient light effect simulating sunrise, and the TTS voice synthesis announces in a warm and cheerful tone, "Good morning, it's a sunny day today, the temperature is 22 degrees Celsius, and you have a department meeting at 9:00 AM." The three channels convey a consistent intention of "attention and greeting," forming a complete and natural morning wake-up experience.

[0126] Furthermore, in one optional implementation, dialogue can be proactively initiated at appropriate times based on a scene knowledge graph. Specifically, the system maintains a scene knowledge graph that stores typical dialogue topics, user interests, and historical dialogue records for different interaction scenarios. When the system determines that the current scene meets preset proactive dialogue triggering conditions—for example, detecting that the user is idle and the last interaction has exceeded a preset time, or the current time matches an interest event recorded in the user's knowledge graph—the system can proactively initiate a dialogue related to the current scene. The dialogue content can be generated based on topics associated with the scene in the scene knowledge graph and personalized by combining the user's historical preferences. For example, in Embodiment 1 (morning wake-up scenario), after completing the morning greeting and schedule reminder, the system can proactively initiate extended dialogues related to the day's weather or schedule; in Embodiment 3 (learning assistant scenario), after completing item recognition, the system can proactively ask the user if they need more information related to the item. Through this proactive dialogue strategy, the robot can proactively initiate relevant dialogues based on scene understanding without the user explicitly issuing instructions, further transforming from passive response to proactive companionship.

[0127] The triggering of proactive dialogue is based on a combination of the following rules: First, the idle state detection rule: when the user state data output by S100 indicates that the user's posture is stable, their gaze is not focused on the screen, their hand movement frequency is below a threshold, and the duration exceeds a preset idle time (e.g., 2 minutes), the user is determined to be in an idle state. Second, the interaction interval rule: when the time since the end of the last interaction exceeds a preset interval threshold (e.g., 30 minutes), the time condition is determined to be met. Third, the interest event matching rule: when the current time window matches an interest event recorded in the user's knowledge graph (such as important schedules mentioned in the user's historical interactions, the user's preferred activity time, etc.), the event condition is determined to be met. When both the idle state detection rule and the interaction interval rule are met, or the interest event matching rule is met, the system triggers proactive dialogue.

[0128] Through the above methods, the association and matching of output content with motion semantics ensures that the visual, auditory and motion channels convey consistent interactive intentions, avoiding the sense of incongruity caused by multi-channel information conflicts; the time synchronization mechanism coordinates and unifies physical motion and information output; and the emotion-adaptive output dynamically adjusts the output parameters according to the user's emotions, improving the personalization and emotional level of the interaction.

[0129] In a preferred embodiment of the present invention, the method further includes a focus detection and intervention step. This step obtains a comprehensive focus score based on the user's posture stability, eye concentration, and movement frequency, and uses a tiered intervention strategy to control the timing and intensity of reminder output based on the comprehensive focus score and continuous working time. The tiered intervention strategy includes: no intervention when the continuous working time is between 0 and a first time threshold; mild prompting preparation when the continuous working time exceeds the first time threshold but not the second time threshold; and proactively outputting an interactive reminder when the continuous working time exceeds the second time threshold. This interactive reminder includes motion control instructions generated by the motion intention mapping step.

[0130] It should be noted that users' posture and attention span gradually change during extended work periods. Their sitting posture may become slack, their gaze may wander, and they may fidget more. These changes are objective indicators of user fatigue or decreased attention. However, abruptly interrupting a user when they are highly focused may cause resentment; failing to remind them when they are clearly fatigued defeats the purpose of health protection. Therefore, this step employs a tiered intervention strategy, gradually increasing the intensity of reminders based on attention scores and continuous work duration, achieving progressive and rhythmic health protection.

[0131] In this embodiment, attention detection is based on three dimensions of evaluation metrics.

[0132] The first dimension is the assessment of sitting posture stability: Based on the coordinates of key points of the upper body (shoulders, neck, head) output by the S100, the positional variance of each key point within consecutive frames is calculated. The number of frames in the evaluation window can be set according to the system sampling rate (e.g., 30 frames, corresponding to approximately 1 second of observation time). The maximum value of the variance of all key points is taken as the sitting posture stability index. The larger the variance, the more unstable the sitting posture and the lower the concentration.

[0133] The second dimension is gaze concentration assessment: gaze concentration is assessed based on head posture angle and eye features. When the gaze direction deviates from the screen area by more than a preset angle (e.g., 30°) and persists for more than a preset duration (e.g., 5 seconds), it is judged as inattention. The gaze concentration score is the ratio of the cumulative time the gaze is focused on the screen area to the total duration of the assessment window.

[0134] The third dimension is motion frequency assessment: motion frequency is assessed based on the range of motion of key points on the hands and torso. The greater the cumulative distance of movement of key points, the more small movements the user makes and the lower their focus.

[0135] The scores from the three dimensions mentioned above are weighted and combined to obtain a comprehensive focus score. The weights can be set to 0.4 (postural stability), 0.3 (visual concentration), and 0.3 (movement frequency), respectively. The comprehensive score ranges from 0 to 1, with a value closer to 1 indicating higher focus.

[0136] The three stages of the tiered intervention strategy are as follows: Silent period (0 to the first duration threshold, e.g., 30 minutes): The system remains silent and does not intervene in any way. During this time, the user is in a normal work rhythm and is usually highly focused.

[0137] Observation period (exceeding the first duration threshold but not the second duration threshold, e.g., 30 to 45 minutes): The system enters a mildly cued preparation state. The motion channel controls the robot to execute the hovering primitive, remaining stationary in its current position; a faint cue icon is displayed on the screen, such as a pale green dot appearing at the edge of the screen; the ambient light remains statically dim. During this stage, only a very low-intensity presence cue is given to the user, without interrupting their work.

[0138] Intervention period (exceeding the second duration threshold, e.g., more than 45 minutes): The system proactively outputs interactive reminders. The motion channel controls the robot to execute the proximity primitive, moving slowly from the standby position to within the user's personal distance (approximately 1 meter); the auditory channel broadcasts health reminders via TTS speech synthesis, such as "You have been working continuously for 45 minutes, it is recommended to get up and move around"; the visual channel displays health tip cards and simple stretching exercise demonstration animations. This interactive reminder includes motion control commands generated by the S200.

[0139] Taking Example 2 (Focus Monitoring Scenario) as an example: A user works continuously at their desk, and the system continuously monitors their focus. The first 30 minutes are a silent period, with the robot remaining on standby. At the 35-minute mark, an observation period begins, during which the robot executes a hovering command and displays a subtle prompt. At the 50-minute mark, an intervention period begins, with the robot executing a proximity command to move near the user, providing a voice prompt to get up and move around, and displaying a stretching demonstration on the screen. If the user stretches according to the demonstration, the system uses the MediaPipe Pose algorithm to detect the user's skeletal points and compares them in real-time with a preset correct movement template, calculating the similarity. When the similarity is higher than a threshold (e.g., 80%), encouraging feedback is generated (e.g., "Well done, keep it up"); when the similarity is lower, corrective feedback is generated (e.g., "You can raise your arm a little higher"), forming a closed-loop health management experience.

[0140] Through the above methods, the multi-dimensional integrated focus detection method can accurately assess the user's work focus status; the tiered intervention strategy gradually upgrades from complete silence to proactive reminders, avoiding the intrusive feeling of suddenly interrupting the user's work, and achieving gradual and rhythmic health protection.

[0141] In a preferred embodiment of the present invention, the method may further include an emotion soothing step. The user's negative emotional state is obtained through multimodal emotion fusion. Based on the intensity and duration of the negative emotional state, a corresponding soothing strategy is selected from a preset set of soothing strategies. The set of soothing strategies includes at least an empathic dialogue mode, a personalized playback mode, and a silent guardian mode. User feedback is continuously monitored during the soothing process. When the user's emotion does not improve or a rejection signal is detected, switching between different modes within the set of soothing strategies is performed.

[0142] It should be noted that different users have different needs and levels of acceptance for comforting methods when they are feeling down. Some users prefer to talk and confide in others, some prefer to relax by listening to music, and some prefer to be alone in peace and quiet. Fixed comforting methods cannot meet the needs of all users. Therefore, this step pre-sets multiple comforting strategies and dynamically switches them based on real-time user feedback.

[0143] In this embodiment, the set of appeasement strategies includes the following three modes: Empathic Dialogue Mode: The robot uses NLU to understand the user's current emotional expression, generates empathic response remarks, and broadcasts them in a gentle, slow tone through emotion-adaptive speech synthesis. Simultaneously, it controls the display screen to show a warm, comforting animation, and adjusts the ambient lighting to a warm, soothing effect. This mode is suitable for situations where the user is willing to express emotions verbally.

[0144] Personalized Playback Mode: Based on user preferences recorded in the user's knowledge graph, this mode plays the user's favorite music, white noise, or natural sound effects (such as rain or ocean waves), while simultaneously adjusting the ambient lighting to synchronize with the music rhythm to create a relaxing atmosphere. This mode is suitable for situations where users wish to alleviate their emotions through sensory experiences.

[0145] Silent Guardian Mode: When the system detects that a user is in a low mood but does not wish to be disturbed (e.g., the user is waving their hand or giving a command like "Let me be alone"), the robot switches to Silent Guardian Mode. It executes the back primitive to retreat to a non-intentional area and maintains the silent guardian state with the hover primitive, keeping only a minimal ambient light to provide the user with ample space. This mode is suitable for situations where the user needs to be alone.

[0146] Regarding the selection of soothing strategies, the initial mode is selected based on the intensity and duration of the negative emotional state: when the intensity of the negative emotion is high (e.g., above 0.7) and the duration is short (e.g., less than 5 minutes), the empathetic dialogue mode is selected first, proactively providing a channel for venting; when the intensity of the negative emotion is moderate (e.g., 0.6 to 0.7) and the duration is long (e.g., more than 5 minutes), the personalized playback mode is selected first, using music and atmosphere to help the user relax; when a clear rejection signal is detected from the user (e.g., a hand gesture or a voice command such as "let me be quiet"), the silent guardian mode is immediately switched.

[0147] Regarding strategy switching, user feedback is continuously monitored during the soothing process. If the system detects that the intensity of the user's negative emotions has not decreased within a preset time under the current soothing strategy, it determines that the current strategy is ineffective and automatically switches to another strategy. For example, switching from an empathetic dialogue mode to a personalized playback mode, or from a personalized playback mode to a silent guardian mode. When the system detects that the user's emotions have significantly improved (e.g., the intensity of negative emotions has dropped below a threshold), it can gradually exit the soothing state and restore the normal interaction mode.

[0148] Taking Example 4 (Emotional Soothing Scenario) as an example: The system detects that the user is in a sad mood (intensity 0.75) and first activates the empathetic dialogue mode, broadcasting empathetic words in a gentle tone. After 5 minutes, if the system detects that the user's mood has not improved and the user is waving their hand, the system determines that the current strategy is ineffective and automatically switches to silent guardian mode. The robot retreats to a non-attention area and remains hovering, with only a faint warm ambient light remaining. After 10 minutes, if the system detects that the user actively calls out to the robot, the system exits the soothing mode and resumes normal interactive services.

[0149] Through the above methods, the preset multiple soothing strategies and the dynamic switching mechanism based on user feedback can provide appropriate soothing responses in different emotional scenarios, while fully respecting the user's wishes and avoiding continued proactive intervention when the user refuses.

[0150] In a preferred embodiment of the present invention, the method may further include a user knowledge graph update step. A user knowledge graph is established and maintained, comprising user preference data, historical interaction records, and social relationship networks. Based on the user knowledge graph, information associations are established between the current interaction scenario and historical interaction scenarios to generate personalized service recommendations.

[0151] It should be noted that desktop robots accompany users for extended periods, and their ability to remember information across conversations and scenarios is one of their core values ​​that distinguishes them from ordinary smart devices. By building a user knowledge graph, the robot can remember the user's preferences, habits, and social relationships, establishing connections across different scenarios and providing consistent, personalized services.

[0152] In this embodiment, the user knowledge graph comprises three main parts: User preference data includes music preferences, content preferences, interaction style preferences (e.g., whether they prefer direct reminders or subtle prompts), and interaction time preferences (e.g., habitual sleep schedules). This preference data is gradually accumulated and updated through user choices and feedback over long-term interactions.

[0153] Historical interaction records include dialogue history, interaction scenario types, service call records, and user feedback (positive / negative). After each interaction, key information from that interaction (interaction scenario type, user emotional changes, user feedback, newly identified preferences, etc.) is extracted and stored.

[0154] Social relationship network: including family member relationships, visitor relationships, and interaction frequency. In Example 5 (guest reception scenario), when a visitor visits multiple times, the system can automatically identify the visitor and associate it with historical conversation memories to provide a consistent reception experience.

[0155] The knowledge graph is updated using an incremental update strategy. After each interaction, key information from that interaction is extracted and matched with existing nodes in the knowledge graph. The update of relationship closeness takes into account both historical accumulation and the weight of the latest interaction.

[0156] Furthermore, the knowledge graph also maintains long-term dialogue memories to enable cross-conversation information association. Specifically, after each dialogue, the system extracts key entities (such as people, items, events, and times), core topics, and user-expressed opinions or preferences, storing them as structured memory units in the knowledge graph. At the start of the next dialogue, the system queries the knowledge graph for historical memory units related to the current scenario to determine if there is any relevant historical dialogue content. If a relevant memory exists, it is referenced when generating a dialogue response, ensuring cross-conversation coherence. For example, in Example 5 (guest reception scenario), when a visitor returns, the system can connect to the dialogue memory from their previous visit, naturally mentioning, "Mr. Wang, last time you mentioned…", making the interactive experience more coherent and user-friendly.

[0157] Based on cross-scenario information association using user knowledge graphs, taking Example 3 (Learning Assistant Scenario) as an example: A user previously queried information about a professional book through the robot (historical record). When the user places the book on the table again, the robot recognizes the item, queries the knowledge graph to connect to the historical query record, and generates recommended content: "This is 'Machine Learning,' which you queried last week. You were reading up to Chapter 3 last time. Would you like to continue?" Taking Example 1 (Morning Wake-up Scenario) as an example: The system queries the user's historical wake-up time, sleep duration, and schedule preferences in the knowledge graph, and combines this with the currently detected user status to generate personalized morning greetings and schedule reminders. Taking Example 5 (Guest Reception Scenario) as an example: A visitor visits for the second time. The system confirms the visitor's identity through facial recognition, queries the knowledge graph to obtain the topic records and shared interests established during the previous visit, and proactively recommends topics: "Mr. Wang, you talked to the host about the recent technology exhibition last time. There are new exhibitions opening recently."

[0158] Through the above methods, the user knowledge graph enables cross-scenario information association and historical memory, allowing the robot to provide a coherent personalized service experience and gradually transform from a general-purpose robot into a user's exclusive companion.

[0159] Compared with existing technologies, the embodiments of this application acquire user status through a multimodal perception step, map interactive intentions into semantically meaningful motion primitives and social distance parameters by combining a motion semantic library, and control the robot's physical motion and output visual and auditory information synchronously in the multimodal collaborative output step. This enables the desktop robot to transform from a passive response to an active perception and interaction, with its physical motion itself becoming an interactive language that carries emotions and intentions, forming a unified multimodal expression together with vision and hearing. At the same time, motion parameters of touch primitives are generated through a four-dimensional strategy, and user avoidance behavior is continuously detected and withdrawn during execution, achieving low-intrusive active contact. While expressing care, it fully respects user boundaries and enhances the safety and naturalness of the interaction.

[0160] To further enhance the interactive experience, the present invention also provides improved implementation methods for haptic interaction.

[0161] Deploy a high-density array of tactile sensors in key contact areas of the robot body (such as the top and handheld parts). The sensor types can be piezoelectric or capacitive tactile sensors, used to sense the user's touch position, force, and swipe trajectory, supporting advanced tactile interaction methods such as gesture recognition and swipe trajectory tracking.

[0162] In terms of haptic feedback, the system is equipped with multi-level haptic feedback components. The vibration feedback unit uses a multi-band vibration motor, supporting an adjustable frequency range of 20Hz to 500Hz, capable of simulating vibration feedback of different textures, such as patting, stroking, and tapping. The temperature feedback unit uses a semiconductor heating / cooling module, supporting precise temperature control within the range of 20℃ to 42℃, capable of simulating warm or cool tactile sensations. Furthermore, in an optional implementation, the haptic feedback components may also include a material feedback unit, simulating the surface feel of different materials through a surface deformation mechanism. For example, by controlling a tiny array of surface protrusions, different material feeles such as rough, smooth, and soft can be simulated, providing users with a richer haptic experience based on the interaction scenario when the robot executes touch primitives. For example, a soft, warm touch can be simulated in a soothing scenario, while a smooth, comfortable touch can be simulated in everyday interactions.

[0163] In terms of haptic-visual-voice fusion, the haptic interaction module works in conjunction with the multimodal perception module and the multimodal collaborative output module. When the haptic sensor array detects a user's touch event, the haptic interaction module encapsulates the touch location coordinates, force value, and touch type (such as tap, long press, swipe) into a haptic event message, which is then sent to the decision layer via the system's internal data bus or message queue. Upon receiving the haptic event message, the decision layer determines the user's intent based on the touch type (for example, a tap on the top usually indicates waking up or greeting, while a long press may indicate a need for comfort), and triggers the visual perception submodule of the multimodal perception module to collect the current user's facial image for identity verification, and / or triggers the voice perception submodule to collect the user's voice for interaction confirmation. Visual identity verification extracts and compares features from the collected facial image through the facial recognition branch, confirms the user's identity, and returns the identity information to the decision layer; voice interaction confirmation recognizes the collected user's voice through the voice recognition branch to extract the user's interaction needs. The decision-making layer integrates the identity verification results and / or voice verification results, and sends the confirmed user intent to the multimodal collaborative output module. The multimodal collaborative output module synchronously outputs corresponding visual and / or auditory information based on the verification results (e.g., a personalized welcome screen is displayed after identity verification, and the user's name and greeting are announced via TTS voice; or corresponding service operations are executed based on voice commands). Thus, the tactile interaction module, multimodal perception module, and multimodal collaborative output module form a complete collaborative working loop, ensuring timely triggering of identity verification and interaction confirmation upon detecting a touch event, improving the reliability and security of the interaction. Tactile and voice collaboration: Users can wake the robot by touching specific parts of it (e.g., a light tap on the top), and further clarify their interaction needs through voice commands, improving the reliability and convenience of the interaction. Multimodal emotional expression: When the robot detects a user's low mood, it can provide comprehensive multimodal emotional support through tactile feedback (e.g., warm, gentle vibrations), combined with comforting voice (e.g., empathetic words in a gentle tone) and visual care (e.g., warm ambient lighting and soothing animations).

[0164] Regarding the collaborative working mechanism, when the haptic sensor array detects a user's touch event, the haptic interaction module encapsulates the touch location coordinates, force value, and touch type (such as tap, long press, swipe) into a haptic event message, which is then sent to the decision layer via the system's internal data bus or message queue. Based on the haptic event message, the decision layer triggers the visual perception submodule of the multimodal perception module to acquire the current user's facial image for identity verification. Simultaneously, it can determine the user's intent based on the touch type (e.g., a tap on the top usually indicates waking up or greeting, while a long press may indicate a need for comfort). After identity verification and intent determination, the decision layer sends the results to the multimodal collaborative output module. Based on the identity verification and intent determination results, the multimodal collaborative output module synchronously outputs corresponding visual and / or auditory information (e.g., a personalized welcome screen is displayed after identity verification, and TTS broadcasts the user's name and greeting).

[0165] The above describes the improved implementation methods for haptic interaction. The overall application effect of the technical solution of this application is summarized below with reference to five embodiments.

[0166] The above five embodiments illustrate the specific implementation of the technical solution of this application from different application scenarios. In Embodiment 1 (Morning Wake-up Scenario), the multimodal perception module detects the user's waking behavior, the motion intention mapping module selects the proximity primitive, and the multimodal collaborative output module simultaneously outputs a greeting animation, voice broadcast, and warm ambient light effects, reflecting the paradigm shift from passive wake-up to active perception. In Embodiment 2 (Focus Protection Scenario), the focus detection and intervention steps assess the user's focus state through multi-dimensional fusion of sitting stability, eye concentration, and movement frequency, and gradually upgrade the reminder intensity using a graded intervention strategy, reflecting a gradual and rhythmic health protection. In Embodiment 3 (Learning Assistant Scenario), the object recognition branch detects desktop items and combines them with the user's knowledge graph to perform cross-scenario information association, reflecting personalized service capabilities. In Embodiment 4 (Emotional Soothing Scenario), the motion intention mapping module uses a four-dimensional strategy to generate motion parameters for the touch primitive, and the tentative contact control module continuously detects and immediately withdraws user avoidance behavior during execution, reflecting a low-intrusive active contact security mechanism. In Example 5 (Guest Reception Scenario), the social distance parameter is dynamically adjusted according to the scenario of unfamiliar visitors, and social lubrication and topic recommendation are combined with the user knowledge graph, which reflects the synergistic effect of social distance adaptation and cross-scenario memory.

[0167] As can be seen from the above embodiments, the technical solution of this application can cover the application needs of desktop robots in various interactive scenarios such as daily companionship, health protection, learning assistance, emotional care, and social reception. Various technical features, including multimodal perception, motion semantic library mapping, social distance adaptation, four-dimensional strategy and avoidance detection, multimodal collaborative output, attention detection and graded intervention, emotional soothing strategy switching, and user knowledge graph association, work synergistically in different application scenarios, jointly realizing the paradigm shift of desktop robots from passive response to active perception and active interaction. Physical motion itself becomes an interactive language carrying emotions and intentions, forming a three-dimensional expression space together with vision and hearing; tactile primitives, protected by four-dimensional strategies and avoidance detection mechanisms, achieve low-intrusive active contact, fully respecting user boundaries while expressing care.

[0168] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, server, or network device) to perform the steps of any of the above-described active perception-based desktop robot interaction methods of this application.

[0169] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the above-described active perception-based desktop robot interaction methods.

[0170] In some embodiments, such as Figure 3 As shown, combined with Figure 3 As shown in the illustration, this disclosure also provides an electronic device for active interaction of a desktop robot based on semantic motion control, including a processor 300 and a memory 301. Optionally, the device may further include a communication interface 302 and a bus 303. The processor 300, communication interface 302, and memory 301 can communicate with each other via the bus 303. The communication interface 302 can be used for information transmission. The processor 300 can call logical instructions in the memory 301 to execute the content generation method of the above embodiments.

[0171] Furthermore, the logic instructions in the aforementioned memory 301 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0172] The memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 300 executes functional applications and data processing by running the program instructions / modules stored in the memory 301, that is, it implements the content generation method in the above embodiments.

[0173] The memory 301 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 301 may include high-speed random access memory and may also include non-volatile memory.

[0174] Specifically, the aforementioned electronic devices may include desktop robots.

[0175] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A desktop robot interaction method based on active perception, characterized in that, include: Multimodal perception steps: Collect the user's visual and voice information to obtain user status data; Motion intent mapping steps: Based on the user state data, select the corresponding motion primitive from the preset motion semantic library, and generate motion control instructions containing social distance parameters according to the current interaction scenario; wherein, the motion semantic library contains a variety of motion primitives, each motion primitive is mapped to a preset semantic meaning, applicable scenario and motion parameter configuration, the motion parameter configuration includes at least motion speed and / or distance from the user; when the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design and timing selection; Multimodal collaborative output steps: Control the desktop robot to execute the motion control command, and synchronously output visual and / or auditory information that matches the current interaction intention; During the execution of the motion control command corresponding to the touch primitive in the multimodal collaborative output step, the method further includes: continuously detecting user avoidance behavior; if user avoidance behavior is detected, stopping the execution of the current motion control command and switching to a preset withdrawal motion primitive.

2. The method according to claim 1, characterized in that, The motion intent mapping step further includes: Establish a mapping model from interaction intent to motion parameters. The mapping model defines the motion primitives, semantic meanings, and parameter configuration ranges corresponding to various interaction intents. Based on the mapping model, the current interaction intent identified from the user state data is converted into the motion control command.

3. The method according to claim 2, characterized in that, Based on the mapping model, the current interaction intent identified from the user state data is converted into the motion control command, specifically including: Identify the current interaction intent in the user state data; When the current interaction intent is that the user needs companionship, select the "approach primitive" option. The parameters of the "approach primitive" option are configured such that the movement speed is within a first speed range and the distance from the user is maintained within a preset social distance. When the current interaction intent is that the user refuses or needs to be alone, the back primitive is selected. The parameters of the back primitive are configured such that the movement speed is within the second speed range and the distance from the user is reduced to the user's non-attention area. When the current interaction intent is tentative reassurance, a touch primitive is selected. The parameters of the touch primitive are configured such that the movement speed is within the third speed range and the distance from the user is maintained at a safe boundary. The safe boundary is the preset minimum distance between the robot and the user before physical contact is performed. Wherein, the lower limit of the first speed range is greater than the upper limit of the third speed range, and the lower limit of the second speed range is greater than the upper limit of the first speed range; The motion control command is generated based on the selected motion primitive and its parameter configuration.

4. The method according to claim 3, characterized in that, Based on the mapping model, converting the current interaction intent identified from the user state data into the motion control command further includes: When the current interaction intent is to identify or judge an object, the swing primitive is selected, and the parameters of the swing primitive are configured such that the swing amplitude is within a first amplitude range and the swing frequency is within a first frequency range. When the current interaction intent is to wait for the user to make a decision, a hover primitive is selected. The parameters of the hover primitive are configured to be fixed in position and the duration is adaptively determined according to the user response time. When the current interaction intent is to follow and observe the user's state, a detour primitive is selected. The parameters of the detour primitive are configured to maintain the distance from the user and the movement trajectory revolves around the user from multiple angles.

5. The method according to claim 1, characterized in that, In the motion intent mapping step, the social distance parameter is determined based on the type of the current interaction scenario, specifically including: Identify the type of the current interaction scenario; When the identification result is an emotional comforting or companionship scenario, the social distance parameter is set to a preset intimate distance range; When the identification result is an information query or task assistance scenario, the social distance parameter is set to a preset personal distance range; When the identification result is a scenario of receiving a stranger or initial interaction, the social distance parameter is set to a preset social distance range.

6. The method according to claim 1, characterized in that, The four-dimensional strategy specifically includes the following steps: The robot maintains the contact distance between itself and the user at a preset social distance boundary and continuously detects whether the user shows any avoidance signals. The contact distance is gradually reduced only when the user does not avoid the robot. Reduce the contact speed to below a preset non-threat threshold; The top of the desktop robot is controlled to be lower than the user's eye level, so as to form an approach posture that is lower than the user's eye level; When a preset idle state condition is detected, the physical contact action corresponding to the touch primitive is executed; wherein the preset idle state condition includes at least one of the following: the user's sitting posture stability is higher than a preset stability threshold, the user's line of sight deviates from the screen area by more than a preset angle, and the user's hand operation frequency is lower than a preset frequency threshold.

7. The method according to claim 1, characterized in that, The continuous detection of user avoidance behavior, if detected, stops the execution of the current motion control command and switches to a preset withdrawal motion primitive, specifically including: Real-time acquisition of user body key point coordinates, facial expression recognition results, and speech recognition results; When the detected user's body retraction exceeds a preset retraction threshold, it is determined to be the user's avoidance behavior; or When the facial expression recognition results show at least one of the expressions of anger, disgust, or fear, and the confidence level exceeds a preset expression threshold, it is determined to be the user's avoidance behavior; or When the speech recognition result matches at least one keyword from the preset set of rejection keywords, and the confidence level exceeds the preset speech threshold, it is determined to be the user's avoidance behavior. Upon determining that the user is engaging in evasive behavior, the execution of the current motion control command is immediately stopped, and the current motion primitive is switched to a back primitive or a hover primitive. Preferably, the multimodal sensing step further includes: The visual information is processed in parallel using a multi-task shared backbone network to obtain multiple types of visual perception features, including at least facial expression recognition results and gesture recognition results. The visual perception features of consecutive frames are input into a temporal modeling network to obtain the trend of user state changes; More preferably, the multimodal collaborative output step further includes: Based on the start time of the motion control command, visual output and auditory output are started simultaneously, so that the physical motion and information output of the desktop robot constitute a unified multimodal expression. The specific content of the visual information and the auditory information is determined based on the semantic meaning of the currently selected motion primitive; Most preferably, the method further includes a focus detection and intervention step: A comprehensive focus score is obtained based on the user's posture stability, visual concentration, and movement frequency. Based on the comprehensive focus score and continuous working time, a tiered intervention strategy is adopted to control the timing and intensity of reminder output; The tiered intervention strategy includes: when the continuous working time is between 0 and a first time threshold, no intervention is performed; when the continuous working time exceeds the first time threshold but does not exceed a second time threshold, a mild prompt is prepared; when the continuous working time exceeds the second time threshold, an interactive reminder is actively output, the interactive reminder containing the motion control command generated by the motion intention mapping step.

8. A desktop robot interaction system based on active perception, characterized in that, include: The multimodal perception module is used to collect the user's visual and voice information to obtain user status data; The motion intent mapping module is used to select the corresponding motion primitives from the preset motion semantic library based on user state data, and generate motion control instructions containing social distance parameters according to the current interaction scenario. The motion semantic library contains a variety of motion primitives, each of which is mapped to a preset semantic meaning, applicable scenario, and motion parameter configuration. When the selected motion primitive is a touch primitive, the motion parameter configuration is generated based on a four-dimensional strategy of distance control, speed adjustment, posture design and timing selection; A multimodal collaborative output module is used to control the desktop robot to execute the motion control commands and synchronously output visual and / or auditory information that matches the current interaction intent; The tentative contact control module is used to continuously detect user avoidance behavior during the execution of the motion control command corresponding to the touch primitive by the multimodal collaborative output module. If user avoidance behavior is detected, the execution of the current motion control command is stopped and switched to the preset withdrawal motion primitive.

9. The system according to claim 8, characterized in that, The system also includes a tactile interaction module, which comprises: An array of tactile sensors deployed in the contact area of ​​the robot's body is used to sense the user's touch position and force; The haptic feedback component includes at least a vibration feedback unit and a temperature feedback unit; The tactile interaction module works in conjunction with the multimodal perception module and the multimodal collaborative output module to trigger visual identity verification and / or voice interaction verification when a touch event is detected.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.

Images