A visual-language interaction method for social robots based on gaze-triggered joint attention.
By acquiring image and depth data in the robot to obtain the user's head pose and gaze direction, and combining this with candidate object detection, the robot is controlled to turn towards the user's focus and provide feedback. This solves the problem of existing robots relying on explicit commands, realizes proactive response and embodied feedback, and improves interaction efficiency and naturalness.
Patent Information
- Application Number
- CN202611263973.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing robots rely on explicit voice commands from users in real-world human-computer interaction, lacking proactive responses to users' natural gaze behavior and changes in head orientation. Furthermore, desktop-level gaze target anchoring is susceptible to depth errors and geometric noise, and lacks embodied feedback loops.
By collecting image and depth data of the interactive scene, the robot obtains the user's three-dimensional head pose and gaze direction. Combined with candidate object detection, a comprehensive anchoring score is performed, and the robot is controlled to turn towards the user's attention target. A joint attention loop is established through eye and head feedback.
It enables proactive interaction based on implicit intent triggers, reducing the user's expressive burden, improving interaction efficiency and social intelligence, and enhancing naturalness and intuitive feedback.
Smart Images

Figure CN123114255A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a visual language interaction method for social robots based on gaze-triggered joint attention. Background Technology
[0002] As social robots evolve from tools for performing single tasks to interactive agents capable of continuous communication in real-world environments, they need not only to possess speech understanding and language generation capabilities, but also to understand the nonverbal social cues that humans extensively use in natural communication, such as eye contact, head orientation, and attention shifts. For social robots, joint attention is one of the core capabilities for establishing natural interaction. Essentially, it lies in the robot's ability to identify the target object currently being focused on by the user and establish a shared focus of attention with the user through observable gaze behavior, thereby enabling both parties to form a common context around the same environmental object.
[0003] In recent years, the development of large language models and visual language models has improved robots' ability to understand natural language and visual scenes, enabling them to generate more natural dialogue responses based on multimodal information such as images, speech, and text. However, in real-world human-computer interaction, existing multimodal dialogue systems for robots still primarily rely on explicit voice commands or clear task requests from users, lacking full utilization of implicit social cues such as natural gaze behavior, head orientation changes, and attention shifts. Therefore, while robots can generate language responses, they still fall short in proactively understanding the user's focus, establishing shared attention, and engaging in natural dialogue around environmental objects. Existing technologies suffer from at least the following problems and shortcomings:
[0004] First, the passive response paradigm leads to a lack of naturalness in interaction. Existing robot dialogue systems mostly rely on explicit verbal commands from users as triggers. The system typically only responds after receiving explicit statements such as "Look at this cup" or "What do you think of this book?" This approach heavily depends on users first identifying the object verbally, resulting in a passive interaction mode. It increases the user's expressive and cognitive burden and fails to fully utilize implicit nonverbal cues naturally present in real face-to-face communication, such as eye contact, gaze shifts, and head orientation changes.
[0005] Second, desktop-level gaze target anchoring is susceptible to depth errors and geometric noise. In 3D space, mapping the user's gaze ray to environmental objects using only simple geometric methods is easily affected by head pose estimation errors, depth measurement noise, camera viewpoint bias, and the spatial proximity of target objects. On the one hand, the instability of depth data can lead to significant uncertainty in the spatial relationship between the gaze ray and candidate objects. On the other hand, in scenarios where desktop objects are densely packed or placed close together, even slight deviations in the user's head or gaze direction can cause frequent switching of target objects, leading to problems such as object referencing ambiguity, misanchoring, and unstable visual dialogue.
[0006] Third, there is a lack of a natural and embodied physical feedback loop for joint attention. Joint attention is not only the result of inference about the user's focus within the system, but should also be expressed to the user as an observable social signal. Some existing systems, even if they can infer the target object being focused on in the background, often only output verbal responses, lacking an embodied feedback mechanism that remaps the semantic object to changes in the robot's eye, head, or base orientation. This makes it difficult for users to intuitively perceive whether the robot is focusing on the same object as them, thus reducing the sense of social intelligence, responsiveness, and naturalness in the interaction process.
[0007] Therefore, how to construct a technical solution that can stably identify user attention targets in real desktop social scenarios, proactively initiate target-oriented visual language dialogues based on continuous gaze, and establish a joint attention loop through embodied gaze feedback from the robot's head and eyes has become an urgent technical problem to be solved in this field. Summary of the Invention
[0008] To address the problem that existing robots, in real-world human-computer interaction processes, still primarily rely on users' explicit voice commands or clear task requests for passive responses, lacking the ability to actively respond to and utilize implicit social cues such as users' natural gaze behavior, changes in head orientation, and attention shifts, this invention proposes a gaze-triggered joint attention-based visual-language interaction method for social robots, which can solve the aforementioned problems.
[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A visual-language interaction method for social robots based on gaze-triggered joint attention includes: Step 1: The robot collects scene images and depth data aligned with the scene images in the interactive scene. Step 2: Obtain the user's facial region from the scene image, perform head pose estimation based on the user's facial region, obtain the user's three-dimensional head pose angle, and generate the user's gaze direction vector g and two-dimensional attention heatmap in the camera coordinate system based on the three-dimensional head rotation angle. Step 3: Perform candidate object detection on the scene image to obtain one or more candidate objects, and output the two-dimensional detection box of the candidate object. Combine the depth data and camera intrinsic parameters to calculate the three-dimensional spatial coordinates of the candidate object in the camera coordinate system. Step 4: Perform a comprehensive anchoring score on each candidate object and the user's gaze direction based on angular consistency, spatial distance, and two-dimensional attention heatmap, and select the candidate object that meets the preset conditions as the target object that the user is currently focusing on. Step 5: Control the robot to turn and gaze at the target object based on the three-dimensional spatial coordinates of the target object in the camera coordinate system.
[0010] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The visual language interaction method of the social robot based on gaze-triggered joint attention of the present invention, on the one hand, collects scene images and depth data in the interaction scene, and obtains the three-dimensional pose angle of the user's head from them, and generates the gaze direction vector g and two-dimensional attention heatmap of the user in the camera coordinate system according to the three-dimensional rotation angle of the head. On the other hand, it performs candidate object detection on the scene image to obtain one or more candidate objects, and performs weighted target anchoring on the multiple candidate objects in combination with the head pose to obtain the target object that the user is currently paying attention to, thereby realizing the establishment of joint attention between the user and the robot based on gaze trigger.
[0011] Breaking away from passive, reactive dialogue, this system enables proactive interaction triggered by implicit intent. Instead of relying on users explicitly naming objects, it utilizes the user's sustained gaze at the target object as an implicit interaction trigger signal. This allows the robot to proactively perceive the object the user is paying attention to and initiate contextual dialogue related to that object, thereby reducing the user's verbal burden and cognitive load during the interaction process.
[0012] By controlling the robot to turn and gaze at the target object, the robot's feedback is no longer limited to semantic output, but is mapped to an embodied feedback mechanism that reflects changes in the robot's eye, head, or base orientation. This allows users to intuitively perceive whether the robot is paying attention to the same object as them, thereby improving interaction efficiency and enhancing the sense of social intelligence, responsiveness, and naturalness during the interaction process.
[0013] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description
[0014] Figure 1 This is a flowchart of an embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention; Figure 2This is a block diagram of the robot visual language interaction system structure in one embodiment of the social robot visual language interaction method based on gaze-triggered joint attention proposed in this invention; Figure 3 This is a schematic diagram of the gaze perception and target anchoring results in one embodiment of the gaze-triggered joint attention method for visual language interaction of social robots proposed in this invention; Figure 4 This is a schematic diagram of a multimodal weighted fusion target anchoring mechanism in one embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention; Figure 5 This is a schematic diagram of a human-computer interaction scenario under non-alignment and gaze alignment conditions in one embodiment of the social robot visual language interaction method based on gaze-triggered joint attention proposed in this invention. Figure 6 This is a schematic diagram of the timing of the user's and robot's gaze objects under gaze alignment conditions in one embodiment of the social robot visual language interaction method based on gaze-triggered joint attention proposed in this invention; Figure 7 This is a schematic diagram illustrating the training performance of a head pose estimation model in one embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention. Figure 8 This is a schematic diagram comparing the proportion of target-related topics and the average number of rounds under different interaction conditions in one embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention. Figure 9 This is a schematic diagram comparing user engagement under different interaction conditions in one embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention. Figure 10 This is a schematic diagram of the robot structure in one embodiment of the visual-language interaction method for social robots based on gaze-triggered joint attention proposed in this invention; Figure 11 yes Figure 10 A schematic diagram of the back structure. Detailed Implementation
[0015] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] Example 1, see Figure 1 , Figure 2 As shown, this embodiment proposes a visual-language interaction method for social robots based on gaze-triggered joint attention, including:
[0018] Step 1: The robot collects scene images and depth data aligned with the scene images in the interactive scene.
[0019] Step 2: Obtain the user's facial region from the scene image, estimate the head pose based on the user's facial region, obtain the user's three-dimensional head pose angle, and generate the user's gaze direction vector g and two-dimensional attention heatmap in the camera coordinate system based on the three-dimensional head rotation angle.
[0020] Step 3: Perform candidate object detection on the scene image to obtain one or more candidate objects, and output the two-dimensional detection box of the candidate object. Combine the depth data and camera intrinsic parameters to calculate the three-dimensional spatial coordinates of the candidate object in the camera coordinate system.
[0021] First, the system receives a sequence of scene images, detects the user's facial region, and estimates the user's 3D head pose to obtain the user's head orientation and approximate gaze direction. Then, it performs object detection on the scene images at the same time, extracts multiple candidate objects, and reconstructs the 3D spatial coordinates of each candidate object based on the depth map and camera intrinsic parameters.
[0022] Step four: Perform a comprehensive anchoring score on each candidate object and the user's gaze direction based on angular consistency, spatial distance, and two-dimensional attention heatmap, and select the candidate object that meets the preset conditions as the target object that the user is currently focusing on.
[0023] Based on the relationship between each candidate object and the user's gaze direction, and combined with the prior of the two-dimensional attention heatmap, a comprehensive anchoring score is calculated for each candidate object, and the candidate object whose score meets the preset conditions is selected as the target object currently being focused on by the user.
[0024] Step 5: Control the robot to turn and gaze at the target object based on the three-dimensional spatial coordinates of the target object in the camera coordinate system.
[0025] The system sends the three-dimensional spatial coordinates of the target object to the robot control module, aligning the robot's head, eyes, and base towards the target object of interest to the user. This action provides the user with observable spatial consistency feedback, helping to establish joint attention between the user and the robot.
[0026] In some embodiments, after each round of goal-oriented interaction, the robot control module controls the robot's gaze to naturally return to the user's face, maintaining the face-to-face characteristic during the communication process. Thus, this embodiment forms a complete closed loop of "gaze perception - target anchoring - robot gaze alignment - visual language dialogue - returning to the user's face," enabling the robot to transform from a passive voice response subject into an active visual language interaction subject based on the user's natural gaze behavior.
[0027] This embodiment of the social robot visual language interaction method based on gaze-triggered joint attention, on the one hand, collects scene images and depth data in the interaction scene, and obtains the user's three-dimensional head pose angle from them, and generates the user's gaze direction vector g and two-dimensional attention heatmap in the camera coordinate system based on the three-dimensional head rotation angle. On the other hand, it performs candidate object detection on the scene image to obtain one or more candidate objects, and performs weighted target anchoring on multiple candidate objects in combination with the user's head pose to obtain the target object that the user is currently paying attention to, thus realizing the establishment of joint attention between the user and the robot based on gaze trigger.
[0028] Breaking away from passive, reactive dialogue, this system enables proactive interaction triggered by implicit intent. Instead of relying on users explicitly naming objects, it utilizes the user's sustained gaze at the target object as an implicit interaction trigger signal. This allows the robot to proactively perceive the object the user is paying attention to and initiate contextual dialogue related to that object, thereby reducing the user's verbal burden and cognitive load during the interaction process.
[0029] By controlling the robot to turn and gaze at the target object, the robot's feedback is no longer limited to semantic output, but is mapped to an embodied feedback mechanism that reflects changes in the robot's eye, head, or base orientation. This allows users to intuitively perceive whether the robot is paying attention to the same object as them, thereby improving interaction efficiency and enhancing the sense of social intelligence, responsiveness, and naturalness during the interaction process.
[0030] The robot is equipped with cameras to capture scene images in the interactive environment, and depth sensors to capture color images and depth information of the user's interaction with the desktop. The depth sensor can be an RGB-D camera or an Azure Kinect camera. The microphone can be a single-channel microphone, a microphone array, or a microphone built into the robot.
[0031] In some embodiments, the robot is also equipped with a microphone for collecting user voice information.
[0032] like Figure 3 The diagram illustrates the process by which the system determines the user's attention target based on head pose, gaze direction vector, two-dimensional attention heatmap, and candidate target detection results. To achieve stable target anchoring in desktop social scenarios, this embodiment employs a coarse-to-fine gaze perception and target anchoring strategy. First, the system performs three-dimensional head pose estimation on the user's facial region, outputting the user's head pitch angle, yaw angle, and roll angle, and further constructs the user's gaze direction vector.
[0033] Head pose estimation can be achieved through a 3D head pose regression model. This model takes a user's facial image as input and outputs the 3D rotational pose parameters of the user's head. To improve the estimation accuracy of subtle head deflections in close-range desktop interaction scenarios, the model can employ convolutional neural networks, residual networks, attention enhancement networks, or other neural network structures capable of regressing 3D head pose parameters.
[0034] In some embodiments, in step two, the user's facial region is input into the ResNet34 depth regression model, which outputs the user's three-dimensional head pose angle.
[0035] In some embodiments, the head pose estimation network employs a deep regression model with ResNet34 as the backbone and embedded CBAM attention modules, thereby enhancing the ability to extract features from key areas of the user's face and suppressing background interference.
[0036] In some embodiments, the system uses an object detection model to detect the interactive scene image and obtain at least one two-dimensional bounding box of a candidate object. Step three uses any one of the YOLO model, Faster R-CNN model, DETR model, or Mask R-CNN model for candidate object detection, or other visual detection models capable of outputting object categories and two-dimensional bounding boxes.
[0037] The three-dimensional attitude angles output by the model can be expressed as: ,in, Indicates pitch angle, Indicates the yaw angle. Indicates the roll angle.
[0038] In some embodiments, the head pose estimation network can be trained on the 300W-LP dataset, with an input image size of 224×224, using the Adam optimizer with a learning rate of 0.0001, a batch size of 64, and 150 training epochs. To improve the sensitivity to subtle head movements in close-up desktop scenarios, Wing Loss can be used as the regression loss function, with parameters set to w=10.0 and ϵ=2.0. The above model structure, dataset, and training parameters are merely one preferred implementation and do not constitute a limitation on the scope of protection of this invention.
[0039] The user's gaze direction vector g in the camera coordinate system is calculated based on the three-dimensional attitude angle. This gaze direction vector represents the user's current head orientation and is used together with the three-dimensional spatial coordinates of the candidate object for subsequent target anchoring.
[0040] In one specific implementation, a head rotation matrix is constructed based on the pitch angle, yaw angle, and roll angle: .
[0041] in, , and Let represent the rotation matrices about the corresponding coordinate axes of the camera coordinate system. Let the forward unit vector of the user's head in the initial pose be . , This indicates that the forward unit vector under the initial pose is represented by the head rotation matrix R. The current head orientation vector obtained after rotation is then represented by the user's gaze direction vector as follows: .
[0042] The initial forward unit vector and rotation order can be set according to the definition of the camera coordinate system.
[0043] This gaze direction vector is used to represent the direction of visual attention currently being directed by the user's head, and is used together with the three-dimensional spatial coordinates of the candidate object for subsequent target anchoring.
[0044] Simultaneously, a two-dimensional attention heatmap is generated based on the user's facial position, head pose parameters, and image spatial location. The two-dimensional attention heatmap describes the probability or intensity of user attention to different regions in the image, with higher heatmap values indicating a greater probability of user attention. The two-dimensional attention heatmap can be generated through gaze direction projection, a deep learning attention estimation model, or a combination of both.
[0045] In some embodiments, such as Figure 3As shown, the two-dimensional attention heatmap is generated using a two-dimensional gaze target estimation model based on GazeFollow. The input to the two-dimensional gaze target estimation model includes a color image of the current interaction scene and a two-dimensional bounding box of the user's face in the color image. The model predicts the distribution of the user's two-dimensional gaze position in the scene image based on the visual features of the user's head region, the spatial position of the user's head in the scene image, and scene context features, and outputs a two-dimensional attention heatmap. .in, Indicates the position of image pixels The relative attention intensity at a location is calculated; the higher the value, the more likely that location is to attract user attention. The 2D attention heatmap is resized and spatially aligned with the original scene image.
[0046] In some embodiments, the method for calculating the three-dimensional spatial coordinates of the candidate object in the camera coordinate system is as follows: Get the The pixel coordinates of the center point of the two-dimensional detection bounding box for each candidate object and the depth of the two-dimensional detection box. .
[0047] The three-dimensional spatial coordinates of the candidate object in the camera coordinate system are: .
[0048] .
[0049] .
[0050] in, These are the camera's internal parameters. , , These represent the three-dimensional spatial coordinate components of the i-th candidate object in the camera coordinate system.
[0051] The three-dimensional spatial coordinates of each candidate object can be obtained using the above method: .
[0052] When the robot needs to perform physical gaze alignment, the system can also transform the 3D coordinates from the camera coordinate system to the robot coordinate system based on the camera extrinsic parameters.
[0053] In some embodiments, the depth of the two-dimensional detection box The method for obtaining it is as follows: Read the valid depth value of the depth data within the 2D detection box. The valid depth value is a value greater than zero. Calculate the arithmetic mean or robust mean of the effective depth values and use it as the depth. .
[0054] For each candidate object, depth information is read within its 2D bounding box. To reduce the impact of missing depth values, occlusion, and local depth outliers on 3D localization, effective depth values within the 2D bounding box are sampled, and the median depth or robust mean depth is used as the representative depth of the candidate object. Using the median depth instead of single-point depth helps suppress 3D localization errors caused by local depth outliers.
[0055] Since candidate objects are typically close together in desktop interaction scenarios, relying solely on the geometric distance between the user's gaze ray and the candidate object can easily lead to misanchoring or target jumps. Therefore, this embodiment employs a multimodal weighted fusion scoring mechanism to comprehensively anchor candidate objects.
[0056] like Figure 4 As shown, this illustrates the process of obtaining a comprehensive anchoring score by weighted fusion of angular consistency score, spatial distance normalized score, and two-dimensional attention heatmap score. For the first... One candidate The system calculates the angle consistency score respectively. Spatial distance normalized score And two-dimensional attention heatmap scoring .
[0057] Angle consistency scoring describes the degree of consistency between the user's gaze direction and the candidate object's direction. Let the starting point of the user's gaze ray, i.e., the user's head reference point, have the following three-dimensional coordinates in the camera coordinate system: , No. The three-dimensional coordinates of the center of each candidate object in the camera coordinate system are: Then, from the user's head reference point, point to the first... The candidate object direction vectors at the centers of each candidate object are represented as follows:
[0058] .
[0059] in, This can be calculated based on the 2D position of the user's facial key points, their corresponding depth values, and camera intrinsic parameters. In one specific implementation, the user's nose tip key point or facial center key point is selected as the head reference pixel. Let this reference pixel be... The corresponding depth is Then the user's head reference point can be represented as:
[0060] .
[0061] The representative depth can be obtained by medianing or other robust statistical calculations of the effective depth values in the neighborhood of the reference pixel.
[0062] In some embodiments, the angle consistency score in step four is calculated as follows: .
[0063] in, Score the angular consistency of the i-th candidate object. Let the gaze direction vector g and the direction vector of the i-th candidate object be... The angle between them This is the angle attenuation parameter.
[0064] The user gaze direction vector is With the direction vector of the candidate object The included angle between them is The calculation method is as follows: .
[0065] Spatial distance normalization score is used to describe the three-dimensional spatial proximity between the user's gaze direction and the candidate object. For candidate objects whose angle with the user's gaze direction is no greater than a preset maximum viewing angle threshold, the user's head reference point is set as follows: The unit vector of the user's gaze direction is , No. The center of each candidate object is Then the first The vertical distance from the center of each candidate object to the axis of the user's gaze direction is expressed as: .
[0066] The method for calculating the spatial distance normalized score is as follows: .
[0067] in, The spatial distance normalization score for the i-th candidate object is calculated. This represents a smoothing constant, used to prevent computational instability caused by excessively small depth values. This represents the normalized distance threshold, and k represents the attenuation slope parameter of the Sigmoid function. By dividing the vertical distance by the depth value of the candidate object, the influence of the depth sensor's ranging noise varying with distance can be compensated to some extent.
[0068] The calculation method for the two-dimensional attention heatmap score is as follows: .
[0069] in, Score the two-dimensional attention heatmap of the i-th candidate object. Let be the number of pixels within the 2D detection bounding box of the i-th candidate object. This is a two-dimensional attention heatmap.
[0070] A comprehensive anchoring score is performed between the i-th candidate and the user's gaze direction. for: .
[0071] , , These are the weighting coefficients.
[0072] In one specific implementation, the weighting coefficient can be set as follows: , , ;at the same time, , and They can be set separately , , The above values are only one possible implementation and do not constitute a limitation on the scope of protection of this invention. In practical applications, adjustments can be made according to different sensor accuracies, robot forms, and interaction scenarios.
[0073] The candidate object with the highest overall anchor score that is greater than the preset anchor threshold is selected as the target object currently of interest to the user. To avoid frequent jumps between adjacent candidate objects, a time stability constraint can be introduced, that is, only when the same candidate object maintains a high overall anchor score for several consecutive frames, or is determined to be the most likely target of interest for several consecutive frames, is it identified as the target object.
[0074] In some embodiments, the candidate object with the highest score is selected as the target object that the user is currently focusing on.
[0075] The robot control module calculates the motion targets of the robot's head, eyes, and base based on the three-dimensional spatial coordinates of the target object.
[0076] In some embodiments, the control method for controlling the robot to turn and gaze at the target object based on the three-dimensional spatial coordinates of the target object in the camera coordinate system in step five includes:
[0077] Calculate the yaw angle of the robot as it turns to look at the target object. and pitch angle : .
[0078] .
[0079] Where X, Y, and Z are the three-dimensional spatial coordinate components of the target object in the camera coordinate system.
[0080] When the absolute value of the target yaw angle is less than or equal to a preset yaw angle threshold, the robot can perform only minor squinting movements with its eyes, while its head and base remain relatively stable. This method is suitable when the target object is located near the front of the robot.
[0081] In some embodiments, step five involves controlling any combination of movements of the robot's eye mechanism, neck mechanism, and chassis based on the yaw angle and pitch angle, so that the robot rotates to face and gaze at the target object.
[0082] In some embodiments, in step five when At that time, control the eye mechanism to rotate. Otherwise, control the chassis movement And control the rotation of the neck mechanism. .
[0083] When the absolute value of the target yaw angle is greater than a preset yaw angle threshold, the robot can perform a large-angle rotation through its head and base, while simultaneously adjusting its eye orientation to avoid the eyes being in a prolonged extreme yaw position. In one specific embodiment, the preset yaw angle threshold can be 0.5 radians, but this value does not constitute a limitation on the scope of protection of this invention.
[0084] To prevent frequent small movements of the robot caused by target detection box jitter, depth noise, or gaze estimation fluctuations, the robot control module can also use trajectory smoothing interpolation and motion dead zone filtering.
[0085] When the difference between the current target and the target at the previous moment is less than the preset dead zone threshold, the robot maintains the previous control command unchanged.
[0086] After the goal-oriented dialogue ends, the robot control module drives the robot to naturally return to the user's facial orientation to resume face-to-face communication. Through this control mechanism, the robot can dynamically allocate attention among the user, the target object, and the dialogue content, and express joint attention through observable physical actions.
[0087] In some embodiments, step five is preceded by timing the duration of the user's gaze at the target object, and step five is executed when the first gaze time threshold is reached.
[0088] When the sustained gaze duration on the target object reaches the first gaze duration threshold, the system determines that the user may be interested in the target object. At this point, the robot control module receives the three-dimensional coordinates of the target object and controls the robot's eyes, head, and base to face the target object. This stage is mainly used to establish spatial alignment and joint attention physical feedback, and does not necessarily initiate verbal dialogue immediately.
[0089] In some embodiments, when the user's continuous gaze time on the target object reaches a second gaze time threshold or when the robot turns to gaze at the target object, an active visual dialogue step is initiated. The scene image containing the two-dimensional detection box and the active dialogue prompt information are input into the visual language model to generate a target-oriented dialogue response around the target object.
[0090] When the sustained gaze duration on the target object reaches the second gaze duration threshold, the system considers the user to have a relatively clear interaction intention. At this point, the dialogue assistance module initiates an active visual dialogue process, inputting a scene image containing the target object marker, user speech recognition text, and active dialogue prompts into the visual language model to generate a goal-oriented dialogue response centered around the target object.
[0091] In one specific implementation, the first gaze duration threshold can be set to 0.5 seconds, and the second gaze duration threshold can be set to 1.5 seconds. When a user briefly glances at an object, the robot can perform only slight gaze alignment; when the user continuously gazes at an object, the robot actively initiates a goal-oriented visual language dialogue. For example, when a user continuously looks at a yellow cup on a table, the robot can generate a response related to the target object, such as "Are you looking at this yellow cup?"
[0092] By using the above-mentioned dual-threshold hierarchical triggering method, the present invention can distinguish between the user's short-term attention shift and stable interaction intention, and reduce the false triggering of active dialogue caused by accidental scanning, target detection jitter or short-term attention changes.
[0093] The dialogue assistance module in this embodiment includes an automatic speech recognition unit (ASR) and a visual language model unit (VLM), and the speech output module includes a speech synthesis unit.
[0094] The automatic speech recognition unit is used to convert user speech information into speech-recognized text. If the user does not speak but continues to look at a target object, visual dialogue prompts can still be actively constructed based on the gaze triggering mechanism; if the user speaks at the same time, the user's speech-recognized text is input into the visual language model as additional semantic information.
[0095] The visual language model unit is used to receive multimodal input and generate goal-oriented dialogue responses. Multimodal input may include: current scene image, 2D bounding box marker of the target object, user speech recognition text, target object category, 3D position of the target object, and proactive dialogue prompts.
[0096] In one specific implementation, a two-dimensional bounding box of the target object is annotated in the original scene image to obtain a scene image containing the target object marker. The target object marker can be a red bounding box, a highlighted box, an arrow, or other image annotation methods, used to guide the visual language model to focus on the target object.
[0097] In one specific implementation, the proactive dialogue prompt information may include the following:
[0098] "You are a social bot. The user is currently staring at a tagged object in an image. Please generate a natural, concise, and friendly Chinese response based on the object's appearance, category, location, and the user's possible intent. The response should focus on the target object and avoid fabricating details that do not exist in the image."
[0099] In passive chat mode, the visual language model can generate responses solely based on user speech recognition text without receiving target object labeled images, thus reducing the risk of visual false triggers and visual hallucinations. In active visual mode, the visual language model receives scene images containing target object labels, user speech recognition text, and active dialogue prompts to generate target-oriented dialogue responses relevant to the target object.
[0100] In a specific example, the visual language model can be Doubao-1.5-Vision-Pro; however, this invention is not limited to a specific commercial model, and any visual language model that can jointly understand images and text and output natural language can be applied.
[0101] The text-to-speech (TTS) unit receives the target-oriented dialogue response text output by the visual language model, converts the text into speech, and outputs it through the robot's speaker, thereby completing the interactive process from the generation of target-oriented dialogue text to the robot's voice feedback.
[0102] This embodiment uses a finite state machine to manage the switching of the social robot between different interaction modes. The finite state machine includes an idle state, a user-facing voice dialogue state, a candidate target gaze state, a joint attention establishment state, an active visual dialogue state, and a user face return state.
[0103] When idle, the robot waits for the user to enter the interaction area, or for voice and visual information input.
[0104] When engaging in voice conversation with a user, the robot primarily focuses on the user's face and interacts with them using standard language based on the user's speech recognition text. In this mode, the robot can avoid accessing target object marker images to reduce the risk of accidental visual triggering.
[0105] In the candidate target gaze state, the system detects that the user may be gazing at a candidate object, but the continuous gaze time has not yet reached the first gaze time threshold. At this time, the system continues to perform target anchoring scoring and gaze time accumulation.
[0106] When joint attention is established, if the user's continuous gaze time on the target object reaches the first gaze time threshold, the robot will perform a physical gaze alignment action toward the target object, so that the user can observe that the robot is paying attention to the same target object.
[0107] In active visual dialogue mode, when the user's continuous gaze time on the target object reaches the second gaze time threshold, the system calls the visual language model to generate a target-oriented dialogue response around the target object and plays it through the voice output module.
[0108] Once the user's face is back in focus, after the goal-oriented dialogue ends, the robot will look at the user's face again and resume the face-to-face communication posture.
[0109] Through the aforementioned finite state machine, the present invention can dynamically switch the interaction mode according to the user's gaze behavior, voice input, and dialogue process, thereby avoiding the robot from frequently triggering proactive dialogue or being away from the user's face for extended periods.
[0110] like Figure 5 The diagram illustrates human-computer interaction scenarios under non-aligned and gaze-aligned conditions in an embodiment of the present invention, used to compare the robot's passive voice response and gaze-triggered active visual language dialogue. As shown in the diagram, under non-aligned conditions, the robot primarily responds passively based on the user's voice, and object-related topics need to be initiated by the user. Under gaze-aligned conditions, the robot can identify the target object based on the user's continuous gaze, shift its attention to that object to establish joint attention, and actively generate dialogue related to the target object. This demonstrates that the present invention can utilize user gaze behavior to trigger goal-oriented interaction, reducing the need for the user to explicitly point out the target object.
[0111] like Figure 6 The figure shows a time-series diagram of the user's and robot's gaze objects under gaze alignment conditions in an embodiment of the present invention. As can be seen from the figure, under gaze alignment conditions, the robot can turn to the corresponding scene object according to changes in the user's gaze target and gaze at the same object with the user at multiple time periods, thereby establishing joint attention oriented towards a specific target object. This result demonstrates that the present invention can identify the user's attention target and provide feedback on the target anchoring result through observable gaze shifts by the robot.
[0112] like Figure 7 The figure shows a schematic diagram illustrating the training performance of the head pose estimation model in this embodiment of the invention, showing the changes in training accuracy, validation accuracy, and mean absolute error. As can be seen from the figure, with the increase in training rounds, the training accuracy and validation accuracy of the head pose estimation model gradually increase and tend to stabilize, while the mean absolute error continues to decrease. The validation accuracy approaches 90% with a 5-degree tolerance, and the mean absolute error converges to below 3 degrees, indicating that the model can provide relatively stable head pose information for gaze direction estimation and target anchoring.
[0113] like Figure 8 The figure shows a comparison of the proportion and average number of rounds of target-related topics under different interaction conditions in an embodiment of the present invention. As can be seen from the figure, the difference in the number of rounds of general topics between the two interaction conditions is small, while the average number of rounds of specific object-related topics under the gaze alignment condition increases from 6.31 rounds to 10.90 rounds, and the proportion of general topics decreases. This result indicates that the present invention enables human-computer dialogue to focus more on the scene objects that the user is actually paying attention to.
[0114] like Figure 9 The diagram shown is a comparison of user engagement under different interaction conditions in an embodiment of the present invention. Figure 9 The comparison results of dialogue turns, total user speaking time, and average interaction time per turn are shown. As can be seen from the figure, compared to the non-aligned condition, the number of dialogue turns under the gaze-aligned condition increased from 8.69 turns to 13.54 turns, the total user speaking time increased from 41.26 seconds to 68.92 seconds, and the average speaking time per turn increased from 4.75 seconds to 6.14 seconds. These results indicate that the present invention can improve the persistence of target-related interactions and user engagement.
[0115] Example 2: This example proposes a robot, such as... Figure 10 , Figure 11 As shown, the robot includes a base 11, a chassis 12 supported on the base 11, a neck mechanism 13 connected to the chassis 12, and an eye mechanism 14 supported on the neck mechanism 13. The base 11 is provided with a chassis drive mechanism (not shown in the figure) for driving the chassis 12 to rotate around a vertical axis. The neck mechanism 13 includes a neck shaft 131 and a neck drive mechanism (not shown in the figure) for driving the neck shaft to perform pitch motion around a horizontal axis. A neck support 132 is fixed to the top of the neck shaft 131. The eye mechanism 14 includes two displays 141 and 142, which are the robot's two eyes. A display drive mechanism (not shown in the figure) is fixed on the neck support 132 and is connected to the display display one by one. The display drive mechanism is used to drive the display display connected to it to rotate around a vertical axis.
[0116] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.
Claims
1. A visual-language interaction method for social robots based on gaze-triggered joint attention, characterized in that, include: Step 1: The robot collects scene images and depth data aligned with the scene images in the interactive scene. Step 2: Obtain the user's facial region from the scene image, perform head pose estimation based on the user's facial region, obtain the user's three-dimensional head pose angle, and generate the user's gaze direction vector g and two-dimensional attention heatmap in the camera coordinate system based on the three-dimensional head rotation angle. Step 3: Perform candidate object detection on the scene image to obtain one or more candidate objects, and output the two-dimensional detection box of the candidate object. Combine the depth data and camera intrinsic parameters to calculate the three-dimensional spatial coordinates of the candidate object in the camera coordinate system. Step 4: Perform a comprehensive anchoring score on each candidate object and the user's gaze direction based on angular consistency, spatial distance, and two-dimensional attention heatmap, and select the candidate object that meets the preset conditions as the target object that the user is currently focusing on. Step 5: Control the robot to turn and gaze at the target object based on the three-dimensional spatial coordinates of the target object in the camera coordinate system.
2. The visual language interaction method for social robots according to claim 1, characterized in that, In step two, the user's facial region is input into the ResNet34 deep regression model, which outputs the user's three-dimensional head pose angle.
3. The visual language interaction method for social robots according to claim 1, characterized in that, In step three, any one of the YOLO model, Faster R-CNN model, DETR model, or Mask R-CNN model can be used for candidate object detection. The method for calculating the 3D spatial coordinates of the candidate object in the camera coordinate system is as follows: Get the The pixel coordinates of the center point of the two-dimensional detection bounding box for each candidate object and the depth of the two-dimensional detection box. ; The three-dimensional spatial coordinates of the candidate object in the camera coordinate system are: ; ; ; in, These are the camera's internal parameters. , , These represent the three-dimensional spatial coordinate components of the i-th candidate object in the camera coordinate system.
4. The visual language interaction method for social robots according to claim 3, characterized in that, Depth of 2D detection box The method for obtaining it is as follows: Read the effective depth value of the depth data within the two-dimensional detection frame, where the effective depth value is a value greater than zero; Calculate the arithmetic mean or robust mean of the effective depth values and use it as the depth. .
5. The visual language interaction method for social robots according to claim 3, characterized in that, The calculation method for the angle consistency score in step four is as follows: ; in, Score the angular consistency of the i-th candidate object. Let the gaze direction vector g and the direction vector of the i-th candidate object be... The angle between them This refers to the angle attenuation parameter; The method for calculating the spatial distance normalized score is as follows: ; in, The spatial distance normalization score for the i-th candidate object is calculated. This represents the vertical distance from the user's gaze ray to the center of the candidate object or the 3D region of the candidate object. Represents the smoothing constant. This represents the normalized distance threshold, and k represents the decay slope parameter of the Sigmoid function; The calculation method for the two-dimensional attention heatmap score is as follows: ; in, Score the two-dimensional attention heatmap of the i-th candidate object. Let be the number of pixels within the 2D detection bounding box of the i-th candidate object. A two-dimensional attention heatmap; A comprehensive anchoring score is performed between the i-th candidate and the user's gaze direction. for: ; , , These are the weighting coefficients; The candidate with the highest score is selected as the target object that the user is currently focusing on.
6. The visual language interaction method for social robots according to claim 3, characterized in that, Step five involves a control method that uses the target object's three-dimensional spatial coordinates in the camera coordinate system to control the robot's turning and gaze at the target object. Calculate the yaw angle of the robot as it turns to look at the target object. and pitch angle : ; ; Where X, Y, and Z are the three-dimensional spatial coordinate components of the target object in the camera coordinate system.
7. The visual language interaction method for social robots according to claim 6, characterized in that, In step five, the robot's eye mechanism, neck mechanism, and chassis are controlled in any combination of movements based on the yaw angle and pitch angle, so that the robot rotates to face and gaze at the target object.
8. The visual language interaction method for social robots according to claim 7, characterized in that, In step five, when At that time, control the eye mechanism to rotate. Otherwise, control the chassis movement And control the rotation of the neck mechanism. .
9. The visual language interaction method for social robots according to any one of claims 1-8, characterized in that, Step five also includes timing the duration of the user's gaze at the target object, and step five is executed when the first gaze time threshold is reached.
10. The visual language interaction method for social robots according to any one of claims 1-8, characterized in that, When the user's continuous gaze time on the target object reaches the second gaze time threshold, or when the robot turns to gaze at the target object and is in position, the active visual dialogue step is initiated. The scene image containing the two-dimensional detection box and the active dialogue prompt information are input into the visual language model to generate a target-oriented dialogue response around the target object.
