Companion robot and method of controlling its eyeball

Abstract

The application relates to the technical field of companion robots, and provides a companion robot and an eyeball control method thereof. The companion robot comprises an eye screen, the eye screen is used for displaying an eye image, the eye image comprises eyeballs, and the method comprises the following steps: if a specified gesture is identified, the eyeballs are controlled to move from a current display position to a target display position in the eye image; wherein, when the eyeballs are located at the target display position, the display direction of the eyeballs relative to a reference display position in the eye image is consistent with the direction of a target hand relative to the companion robot, and the target hand is a hand that makes the specified gesture. Through the application, the interaction process of the companion robot can be more vivid.

Description

[0001] This application claims priority to Chinese patent application filed on November 19, 2025, with application number 202511704085.6, entitled "Companion Robot and Control Method Thereof", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of companion robot technology, and in particular relates to a companion robot and its eye control method. Background Technology

[0003] Companion robots are intelligent devices with emotional interaction capabilities that can provide users with services such as emotional companionship, psychological comfort, and daily life assistance. However, the interaction between existing companion robots and users is not lively enough. Summary of the Invention

[0004] This application provides a companion robot and its eye-control method, which can make the interaction process of the companion robot more vivid.

[0005] In a first aspect, embodiments of this application provide an eye control method for a companion robot, the companion robot including an eye screen for displaying an eye image, the eye image including an eyeball, the method comprising: If a specified gesture is detected, the eyeball is controlled to move from the current display position to the target display position in the eye image; Wherein, when the eyeball is located at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot, and the target hand is the hand that makes the specified gesture.

[0006] In this embodiment, the companion robot includes an eye screen for displaying an eye image, which includes an eyeball. If a specified gesture is detected, the eyeball is controlled to move from its current display position to a target display position within the eye image. When the eyeball is at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot. This enables flexible control over the eyeball's orientation, allowing the companion robot to track the target object making the specified gesture by following the target hand with its eyeball. This enhances the anthropomorphic performance of the companion robot during interaction, making the interaction process more vivid.

[0007] In some embodiments of the first aspect, after the eyeball moves from the current display position to the target display position in the eye image, the method further includes: When the target hand moves and the orientation of the target hand relative to the companion robot changes, the eyeball is controlled to move accordingly in the eye image so that the display orientation of the eyeball relative to the reference display position changes accordingly with the orientation of the target hand relative to the companion robot.

[0008] In some embodiments of the first aspect, the process of controlling the eyeball to move accordingly in the eye image further includes: If the eyeball moves to the boundary position of the movable range in the eye image, and the target hand continues to move to the side corresponding to the boundary position, then the eyeball is controlled to remain displayed at the boundary position.

[0009] In some embodiments of the first aspect, after the eyeball remains displayed at the boundary position, the method further includes: If the eyeball remains displayed at the boundary position for a duration exceeding a preset duration, the rotating part of the companion robot is controlled to rotate so that the rotating part faces the tracking target, the tracking target being the target hand or the target object, the target object being the object to which the target hand belongs; Wherein, the rotating part is at least one of the head and the torso, the head facing the tracking target means that the facial area of ​​the head faces the tracking target, and the torso facing the tracking target means that the front of the torso faces the tracking target; During the rotation of the rotating part or after the rotating part has completed its rotation, the eyeball is controlled to move from the boundary position to the reference display position.

[0010] In some embodiments of the first aspect, the companion robot further includes a camera for detecting the tracking target, and before controlling the rotation of the rotating part of the companion robot, it further includes: If the deflection angle between the direction from which the camera points to the tracked target and the shooting direction of the camera is within a first angle range, then the head is identified as the rotating part; If the deflection angle is within the second angle range, then the body is identified as the rotating part; Wherein, the deflection angles within the first angle range are all smaller than the deflection angles within the second angle range.

[0011] In some embodiments of the first aspect, the process of controlling the eyeball to move accordingly in the eye image further includes: Determine the movement speed of the target hand; If the movement speed exceeds a preset speed threshold, the eyeball will be controlled in any of the following ways: The first predicted hand position of the target hand is determined based on the movement trajectory of the target hand, the first predicted display position of the eyeball in the eye image is determined based on the first predicted hand position, and the eyeball is controlled to move towards the first predicted display position; Alternatively, the movement speed of the eyeball in the eye image can be reduced, and the eyeball can be controlled to move at the reduced movement speed.

[0012] In some embodiments of the first aspect, the process of controlling the eyeball to move accordingly in the eye image further includes: If it is detected that the target hand is at least partially obscured, then a second predicted hand position of the target hand is determined based on the movement trajectory of the target hand before it is obscured; The second predicted display position of the eyeball in the eye image is determined based on the second predicted hand position, and the eyeball is controlled to move toward the second predicted display position.

[0013] In some embodiments of the first aspect, the designated gesture is any preset gesture from a preset gesture set, and after the eyeball moves from the current display position to the target display position in the eye image, the method further includes: If the duration for which no hand making any preset gesture in the preset gesture set is continuously not recognized reaches a first preset duration, then the eyeball is controlled to remain at the first display position where the eyeball is when the duration reaches the first preset duration; If, after a second preset time has elapsed since the eyeball has been held at the first display position, no hand making any of the preset gestures in the preset gesture set has been identified, then the eyeball is controlled to move to the reference display position, or after the target face is determined, the eyeball is controlled to move to the second display position corresponding to the location of the target face.

[0014] In some embodiments of the first aspect, the companion robot further includes a camera, and before controlling the eyeball to move from the current display position to the target display position in the eye image, it further includes: The hand position information of the target hand is obtained, including the two-dimensional coordinates of the target hand in the target image or the three-dimensional coordinates of the target hand in the camera coordinate system, and the target image is an image captured by the camera that recognizes the specified gesture; Based on the camera calibration parameters and the hand position information, the orientation of the target hand in the robot coordinate system is determined; The target display position is determined based on the direction of the target's hand.

[0015] In some embodiments of the first aspect, determining the target display position based on the target hand orientation includes: Determine the current gaze direction corresponding to the current display position in the robot coordinate system; Calculate the horizontal and / or vertical angle difference between the current gaze direction and the direction of the target hand; The horizontal angle difference is converted into the horizontal displacement of the eyeball in the eye image, and / or the vertical angle difference is converted into the vertical displacement of the eyeball in the eye image; The target display position is determined based on the current display position and the horizontal displacement and / or the vertical displacement.

[0016] In some embodiments of the first aspect, controlling the eyeball to move from a current display position to a target display position in the eye image includes: Interpolate the current display position and the target display position to determine at least one intermediate display position between the current display position and the target display position; Control the eyeball to move from the current display position to the target display position via at least one intermediate display position.

[0017] In some embodiments of the first aspect, before controlling the eyeball to move from the current display position to the target display position in the eye image, the method further includes: If multiple hands making the specified gesture are identified, each of the multiple hands is identified as a candidate hand; Obtain target selection information for each of the candidate hands, and select one from the multiple candidate hands as the target hand based on the target selection information; The target selection information includes at least one of the following: the position information of the candidate hand, the gesture recognition confidence of the candidate hand, the duration of the candidate hand making the specified gesture, and the judgment result of whether the candidate hand belongs to the current interactive object.

[0018] In some embodiments of the first aspect, the companion robot further includes a camera, and the step of selecting one of the candidate hands as the target hand based on the target selection information includes: Acquire target images including multiple candidate hands captured by the camera; Based on the position information of each candidate hand, a first distance between the image reference point of each candidate hand in the target image and the image center of the target image, and a second distance between each candidate hand and the companion robot are determined. A first selection score is determined based on the first distance, and a second selection score is determined based on the second distance, wherein the first selection score increases as the first distance decreases, and the second selection score increases as the second distance decreases; The first selection score and the second selection score of the same candidate hand are weighted and summed to obtain the target selection score of each candidate hand; The candidate hand with the highest target selection score is determined as the target hand.

[0019] Secondly, embodiments of this application provide an eye control device for a companion robot, the companion robot including an eye screen for displaying an eye image, the eye image including an eyeball, and the device comprising: An eye control module is used to control the eyeball to move from the current display position to the target display position in the eye image if a specified gesture is detected. Wherein, when the eyeball is located at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot, and the target hand is the hand that makes the specified gesture.

[0020] Thirdly, embodiments of this application provide a companion robot, including an eye screen, a memory, a processor, and a computer program stored in the memory and executable on the processor. The eye screen is used to display an eye image including the eyeball. When the processor executes the computer program, the companion robot performs the method as described in any one of the first aspects above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computer, implements the method as described in any one of the first aspects above.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the method as described in any one of the first aspects above to be performed.

[0023] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application scenario of the companion robot system provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the eye control method for a companion robot provided in an embodiment of this application; Figure 3 This is an example image of an eyeball in a forward-facing state provided in the embodiments of this application; Figure 4 This is an example diagram showing that the target hand is positioned above the companion robot and the eyeball is positioned above the reference display position, as provided in the embodiments of this application. Figure 5 This is another schematic flowchart of the eye control method for the companion robot provided in the embodiments of this application; Figure 6 This is another schematic flowchart of the eye control method for the companion robot provided in the embodiments of this application; Figure 7 This is an example diagram provided in this application embodiment where the target face is located on the left side of the companion robot and the eyeballs are located to the left of the reference display position; Figure 8 This is a schematic diagram of the eye control device for the companion robot provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the companion robot provided in the embodiments of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] It should be noted that the information collection process (such as the facial information collection process) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0031] A companion robot is a robot with emotional interaction and intelligent companionship functions. It aims to interact with a target object through sensing technology, display, voice, and / or movement, providing services such as emotional companionship, psychological comfort, and daily living assistance. It should be understood that companion robots can be categorized into various types based on the service recipient and application scenario, including but not limited to companion robots for the elderly, children, special needs groups, and general-purpose family companion robots. Based on structural form and mode of movement, companion robots can refer to wheeled companion robots, legged companion robots, desktop companion robots, stationary companion robots, or pet-shaped robots, etc.

[0032] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0033] Please see Figure 1 , Figure 1 This illustration shows an application scenario diagram of the companion robot system provided in an embodiment of this application. The companion robot system may include a companion robot 101, a target object 102, and a control device 103, etc. The control device 103 and the companion robot 101 can be connected and communicated via wired or wireless means.

[0034] The number of companion robots 101 can be one or more (i.e., at least two). The companion robots 101 can be used to interact with the target object 102, thereby providing the target object 102 with services such as emotional companionship, psychological comfort, and life assistance.

[0035] The companion robot may include an eye screen for displaying an eye image, the eye image including the eyeball.

[0036] An eye image can refer to an image displayed on an eye screen that includes the eyeball. The eyeball can refer to the display content in the eye image used to represent the pupil and iris, and can include pupil display elements and iris display elements. The eye image may also include sclera display elements and / or eyelid display elements. By changing the display position of the eyeball in the eye image, changes in the gaze direction of the companion robot can be simulated.

[0037] In some embodiments, the eye image can be a complete image, which includes image regions respectively presenting the eyelids, sclera, pupil, and iris. Multiple frames of complete images with the eyeball at different display positions can be generated or selected, and the multiple frames can be displayed sequentially, causing the eyeball to move from its current display position to a target display position within the eye image.

[0038] In other embodiments, the eye image can be generated by superimposing multiple image elements or image layers. For example, the eye image can be generated by superimposing an eyelid image layer, a sclera image layer, a pupil image layer, and an iris image layer; or, it can be generated by superimposing an eyelid image layer, a sclera image layer, and an eyeball image layer including the pupil and iris. By changing the display positions of the pupil image layer and the iris image layer, or by changing the display position of the eyeball image layer, the eyeball can be moved within the eye image.

[0039] The display position of the eyeball can be characterized by the display coordinates of a reference point in the eye image. This reference point can be the center of the pupil, the center of the iris, or the center of the eyeball display area including both the pupil and iris. The display coordinates can be pixel coordinates or normalized image coordinates in the eye image.

[0040] The eye screen can be a liquid crystal display, a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or any other screen capable of displaying images of the eye.

[0041] The target object 102 can be a user of the companion robot 101, such as a specified person or animal, or any person or animal. These people or animals can be living beings (family members, children, the elderly) or virtual beings (such as virtual people or animals on television). This application does not limit the specific type of the target object 102.

[0042] As an example and not a limitation, Companion Robot 101 is a pet robot shaped like a pet (such as a cat or dog). The pet robot can be located in the home environment and interact with family members in the home environment, thereby providing family members with services such as emotional companionship, psychological comfort, and life assistance, and solving problems such as loneliness, health monitoring and emotional deprivation of family members.

[0043] In some embodiments, the companion robot 101 is equipped with at least one camera. This application does not limit the specific type of the camera. For example, the camera may be a red-green-blue (RGB) camera, a depth camera, a stereo camera (e.g., a binocular camera), a fisheye camera, etc.

[0044] It should be understood that this application does not limit the location of the camera on the companion robot 101. For example, the camera can be placed on any location such as the nose, head, torso, or arm of the companion robot 101.

[0045] The control device 103 can be used to perform at least some of the steps in the eye control method provided in the embodiments of this application. The control device 103 can be disposed within the companion robot 101 or disposed independently of the companion robot 101.

[0046] The control device 103 may include at least one of terminal devices, servers, etc. The control device 103 may also be a processor or control chip installed in the companion robot 101. This application does not limit the specific type of the control device 103, which can be selected according to actual needs.

[0047] Terminal devices can be mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other devices.

[0048] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or any of the following: cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), big data or artificial intelligence platforms, etc. There are no restrictions here.

[0049] Please see Figure 2 , Figure 2A flowchart illustrating an eye-control method for a companion robot according to an embodiment of this application is shown. The companion robot includes an eye screen for displaying an eye image, the eye image including the eyeball. As an example and not a limitation, this eye-control method is applied to a control device 103 as an example. Figure 2 The process shown is explained in detail. This eye control method includes the following steps: Step 201: If the specified gesture is recognized, control the eyeball to move from the current display position to the target display position in the eye image.

[0050] Specifically, when the eyeball is located at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot, and the target hand is the hand that makes the specified gesture.

[0051] The specified gesture can be pre-set and used to trigger the companion robot to adjust the display position of its eyes based on the hand making the specified gesture. Optionally, the specified gesture can be set according to actual needs, and this application does not limit the specific gesture type of the specified gesture.

[0052] In some embodiments, the specified gesture may include gestures formed when the target hand is not holding an object, such as waving, pointing to a location, making a shushing gesture, clapping, giving a thumbs-up, making a peace sign, or making an OK gesture; the specified gesture may also include object-holding gestures when the target hand is holding an object, such as the posture of fingers encircling the cup wall when holding a water cup, the posture of palm supporting the phone and fingers wrapping around the edge of the phone when holding a mobile phone, or the posture of full palm gripping a remote control when holding it. Object-holding gestures can be identified based on hand posture, the positional relationship and / or contact relationship between the target hand and the held object. The target display position can be determined based on the position of the target hand, the position of the object held by the target hand, or the position of the overall area formed by the target hand and the held object. When the target display position is determined based on the position of the held object or the position of the overall area, the determined target display position satisfies the following: the display orientation of the eyeball relative to the reference display position is consistent with the orientation of the target hand relative to the companion robot.

[0053] In some embodiments, during the process of the eyeball moving from the current display position to the target display position, the movement speed and / or movement amplitude of the eyeball can be limited, so that the eyeball moves towards the target display position continuously and smoothly, thereby reducing the risk of eyeball tremors or rapid jumps.

[0054] In some embodiments, the shape and size of the eyeball may remain unchanged during eye movement; in other embodiments, the shape and / or size of the eyeball may also be adjusted according to the interaction state.

[0055] In some embodiments, the center of the pupil, the center of the iris, or the center of the eyeball display area including the pupil and iris can be determined as the eyeball reference point. During the same eyeball control process, the same eyeball reference point can be used to characterize the display position of the eyeball.

[0056] The display position of the eyeball can be represented by the display coordinates of the eyeball reference point in the eye image. The current display position is the display coordinate of the eyeball reference point at the current moment, the target display position is the display coordinate that the eyeball reference point needs to reach to complete this movement, and the reference display position can be the display coordinate of the eyeball reference point when the eyeball is in a forward-looking state. The reference display position can be preset according to the layout of the eye image; for example, the reference display position can be located at the center of the eye region corresponding to the eyeball in the eye image. Figure 3 The image shown is an example of an eye image where the eye is in a forward-facing state. The reference display position is located at... Figure 3 The center position of the eye region corresponding to the eyeball shown.

[0057] The display orientation of the eye relative to the reference display position can refer to the orientation of the eye's current display position relative to the reference display position. When the eye's current display position does not coincide with the reference display position, the display orientation can include the left, right, above, below, or diagonally opposite to the reference display position. An diagonal orientation can include the upper left, upper right, lower left, or lower right. When the eye's current display position coincides with the reference display position, the eye can be in a centered or straight-looking state.

[0058] The orientation of the target hand relative to the companion robot can refer to the spatial orientation of the target hand, determined with the companion robot as a reference. This orientation can include the left, right, top, bottom, diagonal, or directly in front of the companion robot. The orientation of the target hand relative to the companion robot can be determined based on the position of the target hand in the companion robot's coordinate system.

[0059] The orientation of the eyeball relative to the reference display position corresponds to the orientation of the target hand relative to the companion robot. This means that when the target hand is to one side of the companion robot, the eyeball is to the corresponding side of the reference display position. For example, if the target hand is to the left of the companion robot, the eyeball is to the left of the reference display position; if the target hand is to the right of the companion robot, the eyeball is to the right of the reference display position; if the target hand is above the companion robot, the eyeball is above the reference display position; if the target hand is below the companion robot, the eyeball is below the reference display position; and if the target hand is at an angle, the eyeball is at the corresponding angle. When the target hand is directly in front of the companion robot, the display position of the eyeball can coincide with the reference display position to make the eyeball appear centered or looking straight ahead. Figure 4 The image shown is an example of an eyeball positioned above a reference display position when the target's hand is above the companion robot.

[0060] In this application, the orientation of the target hand relative to the companion robot is determined according to the orientation observed from the display side of the eye screen. Orientation consistency means that, when viewed from the display side of the eye screen, the orientation of the eyeball relative to the reference display position is the same as the orientation of the target hand relative to the companion robot. For example, when the target hand is located to the left of the companion robot, the eyeball is located to the left of the reference display position; when the target hand is located to the upper right of the companion robot, the eyeball is located to the upper right of the reference display position.

[0061] In this embodiment, the companion robot includes an eye screen for displaying an eye image, which includes an eyeball. If a specified gesture is detected, the eyeball is controlled to move from its current display position to a target display position within the eye image. When the eyeball is at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot. This allows the display position of the eyeball to intuitively reflect the orientation of the target hand, creating a visual effect of the eyeball gazing at the target hand and improving the directionality and naturalness of the companion robot's gaze feedback.

[0062] In some embodiments of this application, before controlling the eyeball to move from the current display position to the target display position in the eye image, the control device 103 further performs the following steps: When multiple hands making a specified gesture are identified, each hand is identified as a candidate hand. Obtain target selection information for each candidate hand, and select one from multiple candidate hands as the target hand based on the target selection information; The target selection information includes at least one of the following: the position information of the candidate hand, the confidence level of the candidate hand's gesture recognition, the duration of the candidate hand making the specified gesture, and the judgment result of whether the candidate hand belongs to the current interactive object.

[0063] The positional information of the candidate hand can include: two-dimensional positional information of the candidate hand in the image used to recognize the specified gesture, or spatial positional information of the candidate hand relative to the companion robot. The two-dimensional positional information can be represented by the display coordinates of the candidate hand's image reference point or preset hand key points in the image; the spatial positional information can be represented by the three-dimensional coordinates of the candidate hand in the robot coordinate system or camera coordinate system.

[0064] The gesture recognition confidence score is used to characterize the reliability of a candidate hand being identified as making a specified gesture. The higher the gesture recognition confidence score, the higher the reliability of the corresponding recognition result.

[0065] In some embodiments, image information and / or key point information of candidate hands can be input into a gesture recognition model. The gesture recognition model outputs a recognition score for each preset gesture corresponding to the candidate hand, and the recognition score corresponding to the recognized specified gesture is determined as the gesture recognition confidence of the candidate hand. The recognition score can be a probability value, a normalized score, or other numerical values ​​used to characterize the reliability of the gesture recognition result. In other embodiments, the degree of matching between the hand posture or key point distribution of the candidate hand and a preset gesture template can be determined, and the gesture recognition confidence is determined based on the degree of matching. The higher the degree of matching between the candidate hand and the preset gesture template, the higher the gesture recognition confidence. Optionally, the gesture recognition confidence can also be adjusted based on the consistency of the gesture recognition results of the candidate hand in multiple consecutive frames of images. For example, if multiple consecutive frames of images recognize the candidate hand as making the same specified gesture, the gesture recognition confidence of the candidate hand can be increased. Gesture recognition confidence is used to characterize the credibility of a candidate hand being identified as making a specified gesture. It is different from hand detection confidence, which is used to characterize the presence of a hand in an image, and also different from target selection score calculated based on multiple target selection information.

[0066] The duration of a candidate hand making a specified gesture can be determined based on the number of consecutive images of a candidate hand making a specified gesture and the image acquisition frequency, or it can be determined based on the time interval between the moment when a candidate hand first makes a specified gesture and the current moment.

[0067] The current interaction object can refer to the object that the companion robot has already identified and is interacting with before recognizing multiple hands making specified gestures. The interaction can include at least one of voice interaction, touch interaction, gesture interaction, or face recognition-based interaction. For example, the current interaction object can be the object the companion robot is currently responding to its voice, the object currently touching the companion robot, the object that previously made an interaction gesture, or the object the companion robot is currently tracking with its face. In some embodiments, the current interaction object can be determined based on at least one of sound source localization results, touch detection results, gesture recognition results, face detection or recognition results, and historical interaction records. For example, if an object is detected uttering a voice and the companion robot responds to the voice, the object uttering the voice can be identified as the current interaction object.

[0068] In some embodiments, it is determined whether a candidate hand belongs to the current interactive object based on human detection results or human pose recognition results. For example, it is determined whether the image reference point or hand key point of the candidate hand falls within the human detection box of the current interactive object. If so, the candidate hand is determined to belong to the current interactive object. If the hand key point of the candidate hand belongs to the same human pose recognition result as the wrist key point, arm key point, or other body key point of the current interactive object, it can also be determined that the candidate hand belongs to the current interactive object.

[0069] In other embodiments, the distance between the candidate hand and the facial position, body center, or preset body key point of the current interactive object can be determined. If the distance is less than a preset association threshold, the candidate hand is determined to belong to the current interactive object.

[0070] In some embodiments, a target hand can be selected from a plurality of candidate hands based on any one of the target selection information.

[0071] When selecting the target hand based on the position information of the candidate hands, the candidate hand that is closest to the center of the image reference point can be determined as the target hand, or the candidate hand that is closest to the companion robot can be determined as the target hand.

[0072] When selecting the target hand based on the gesture recognition confidence of candidate hands, the candidate hand with the highest gesture recognition confidence can be identified as the target hand.

[0073] When selecting a target hand based on the duration of a candidate hand performing a specified gesture, the candidate hand with the longest duration of performing the specified gesture can be identified as the target hand.

[0074] When selecting a target hand based on whether a candidate hand belongs to the current interactive object, the target hand can be preferentially determined from the candidate hands that belong to the current interactive object. For example, if only one candidate hand belongs to the current interactive object, that candidate hand can be determined as the target hand; if multiple candidate hands belong to the current interactive object, the target hand can be further determined from the multiple candidate hands based on their position information, gesture recognition confidence, or the duration of the specified gesture.

[0075] In some embodiments, at least two of the target selection information can be combined to select a target hand from multiple candidate hands. For example, candidate hands can be filtered sequentially using at least two of the target selection information according to a preset priority; or, corresponding selection scores can be determined based on at least two of the target selection information, and a target selection score can be determined for each candidate hand based on each selection score, with the candidate hand having the highest target selection score being identified as the target hand.

[0076] When filtering according to preset priorities, for example, you can first filter out the candidate hands that belong to the current interactive object from multiple candidate hands, and then select the candidate hand with the highest gesture recognition confidence, the longest duration of making the specified gesture, or the closest to the companion robot as the target hand from the selected candidate hands.

[0077] In this embodiment, the type of designated gesture that can trigger the companion robot to perform eye control can be one or more. In an embodiment where only one type of designated gesture is set, eye control is triggered upon recognizing the designated gesture. In an embodiment where multiple types of designated gestures are set, the multiple types of designated gestures can be set as a preset gesture set, and any preset gesture in the preset gesture set can be used as the designated gesture and trigger eye control. When multiple hands making the designated gesture are recognized, the designated gestures made by the multiple hands can be the same or different. For example, if the first hand makes a peace sign and the second hand makes a heart sign, when both the peace sign and the heart sign belong to the preset gesture set, the first hand and the second hand can be determined as candidate hands, and the target hand can be selected from the multiple candidate hands.

[0078] In this embodiment of the application, when multiple hands making a specified gesture are identified, one hand is selected as the target hand based on at least one of the following: the position information of the candidate hand, the confidence level of the gesture recognition, the duration of the specified gesture, and the judgment result of whether it belongs to the current interactive object. This can reduce the risk of gaze target ambiguity or misselection when multiple hands make the specified gesture at the same time, and improve the accuracy and reliability of target hand selection.

[0079] In some embodiments, when a hand making a specified gesture is identified, that hand can be directly identified as the target hand.

[0080] In some embodiments, after the eye moves to the target display position determined based on the target hand, if multiple hands making any preset gesture from the preset gesture set are identified in subsequently acquired images, the target hand selection process described above can be repeated; if only one hand making any preset gesture from the preset gesture set is identified, that hand can be designated as the new target hand. The new target hand can be the original target hand or another hand, and the preset gesture made by the new target hand can be the same as or different from the originally designated gesture.

[0081] like Figure 5 As shown, in some embodiments of this application, the companion robot also includes a camera, and selects one of a plurality of candidate hands as the target hand based on the target selection information, which may include steps 501 to 505.

[0082] Step 501: Obtain target images including multiple candidate hands captured by the camera.

[0083] In some embodiments, the camera on the companion robot can capture images of the surrounding environment in real time and output them to the control device 103 in real time. When the control device 103 detects multiple hands making specified gestures in the current image (i.e., the currently captured image) during hand detection, it can use the current image as the target image. The specified gestures made by the multiple candidate hands can be the same or different. When the specified gestures made by the multiple candidate hands are different, the gestures made by each candidate hand are all specified gestures that can trigger eye control.

[0084] Step 502: Based on the position information of each candidate hand, determine the first distance between the image reference point of each candidate hand in the target image and the image center of the target image, and the second distance between each candidate hand and the companion robot.

[0085] In some embodiments, the center of the hand detection bounding box, the center of the palm, the wrist keypoint, or a specified finger keypoint can be determined as the image reference point of the candidate hand in the target image. Preferably, the center of the hand detection bounding box or the center of the palm can be used as the image reference point of the candidate hand.

[0086] In one embodiment, the second distance can be determined by comparing the spatial reference points of each candidate hand with preset reference points on the companion robot. The preset reference points on the companion robot can be the origin of the robot's coordinate system, the optical center of the camera, or a preset structural position of the companion robot. The spatial reference points of the candidate hands can be the center of the palm, a wrist key point, or other preset hand key points.

[0087] For any candidate hand among the candidate hands, a first distance between the image reference point of the candidate hand and the image center of the target image can be calculated based on the pixel coordinates of the image reference point of the candidate hand; a second distance between the spatial reference point of the candidate hand and a preset reference point on the companion robot can also be calculated based on the three-dimensional coordinates of the spatial reference point of the candidate hand in the coordinate system of the companion robot.

[0088] In some embodiments, the spatial position of the candidate hand in the robot coordinate system can be determined based on the target image. Specifically, the two-dimensional coordinates of the image reference point of the candidate hand in the target image are determined, and the depth information corresponding to the image reference point is obtained; the three-dimensional coordinates of the spatial reference point of the candidate hand in the camera coordinate system are determined based on the camera's intrinsic parameters, the two-dimensional coordinates, and the depth information; the three-dimensional coordinates are converted into the three-dimensional coordinates of the spatial reference point of the candidate hand in the robot coordinate system according to the calibration relationship between the camera coordinate system and the robot coordinate system. After determining the three-dimensional coordinates of the spatial reference point of the candidate hand in the robot coordinate system, the spatial distance between the spatial reference point and a preset reference point on the companion robot can be calculated, and this spatial distance is determined as the second distance.

[0089] Step 503: Determine the first selection score based on the first distance, and determine the second selection score based on the second distance.

[0090] Among them, the first choice score increases as the first distance decreases, and the second choice score increases as the second distance decreases.

[0091] Since the first distance and the second distance have different dimensions, a first scoring rule can be pre-set for the first distance and a second scoring rule can be pre-set for the second distance. Both the first and second scoring rules are configured such that the smaller the distance, the higher the selection score. Based on this, the first distance can be mapped to a first selection score based on the first scoring rule, and the second distance can be mapped to a second selection score based on the second scoring rule. Alternatively, the first and second distances can be normalized separately, and then the same scoring rule can be used to map the normalized first and second distances to the first and second selection scores, respectively.

[0092] Step 504: The first and second selection scores of the same candidate hand are weighted and summed to obtain the target selection score of each candidate hand.

[0093] In some embodiments, for any candidate hand among the candidate hands, the first selection score and the second selection score of the candidate hand can be weighted and summed according to the weight of the first selection score and the weight of the second selection score to obtain the target selection score of the candidate hand. Optionally, the weight of the first selection score and the weight of the second selection score can be set according to actual needs or empirical values.

[0094] Step 505: Select the candidate hand with the highest target selection score as the target hand.

[0095] In some embodiments, when there are two or more candidate hands with the highest target selection scores, the target hand can be further selected from the tied candidate hands based on other target selection information. For example, the candidate hand with the highest gesture recognition confidence among multiple candidate hands can be determined as the target hand, or the candidate hand with the longest duration of performing the specified gesture can be determined as the target hand, or the candidate hand belonging to the current interaction object can be determined as the target hand; if there are still multiple candidate hands, the selection can also be based on the distance priority principle, selecting the candidate hand that is closest to the companion robot as the target hand, so as to prioritize responding to the interaction object closest to the companion robot.

[0096] In some embodiments, after determining the target hand, if it is necessary to re-determine the target hand based on subsequently acquired images, and a hand making a specified gesture is identified in the subsequently acquired images, then that hand can be re-determined as the target hand. If multiple hands making the specified gesture are identified, then each of the multiple hands can be identified as a candidate hand, and the target hand can be re-selected from the multiple candidate hands according to the target hand selection process described in steps 501 to 505. The re-identified hand can be the previously determined target hand or other hands. Even if the original target hand making the specified gesture is no longer identified, the eye can still be controlled to continue moving to the determined target display position before the eyeball moves to the determined target display position. After the eyeball moves to the target display position, the target hand is re-determined based on subsequently acquired images, and a new target display position is determined based on the re-determined target hand. If a hand making the specified gesture is not identified in the subsequently acquired images, the eyeball control process for when the specified gesture is not identified, as described below, can be executed.

[0097] In this embodiment of the application, by simultaneously considering the centering degree of the candidate hand in the target image and its proximity to the companion robot, and selecting the best candidate hand from multiple candidate hands using a weighted scoring method, the risk of target selection ambiguity when multiple specified gestures coexist can be reduced, and the accuracy of target hand determination can be improved.

[0098] After identifying the target hand, the target display position can be determined based on the target hand. Based on this, the eyeball can be controlled to move from the current display position to the target display position in the eye image.

[0099] like Figure 6 As shown, in some embodiments of this application, the companion robot also includes a camera, and the control device 103 can determine the target display position through steps 601 to 603.

[0100] Step 601: Obtain the hand position information of the target hand.

[0101] The hand position information includes the two-dimensional coordinates of the target hand in the target image or the three-dimensional coordinates of the target hand in the camera coordinate system. The target image is an image captured by the camera that recognizes the specified gesture.

[0102] In one embodiment, the two-dimensional or three-dimensional coordinates of a reference point of the target hand can be determined. The reference point of the target hand can be the center of the hand detection frame, the center of the palm, the wrist key point, or other preset hand key points.

[0103] Step 602: Determine the orientation of the target hand in the robot coordinate system based on the camera calibration parameters and hand position information.

[0104] Camera calibration parameters can include intrinsic camera parameters and extrinsic parameters between the camera coordinate system and the robot coordinate system.

[0105] When the hand position information includes the two-dimensional coordinates of the target hand in the target image, the direction of the ray originating from the optical center of the camera can be determined based on the camera's intrinsic parameters and the two-dimensional coordinates, thus obtaining the direction of the target hand in the camera coordinate system. Then, based on the rotation parameters in the camera's extrinsic parameters, the direction of the target hand in the camera coordinate system is converted to the target hand direction in the robot coordinate system. Since this conversion is performed on the direction, it is unnecessary to determine the depth of the target hand along the ray direction, nor is it necessary to use the translation parameters in the camera's extrinsic parameters.

[0106] In some embodiments, the direction of the target hand in the camera coordinate system can be determined based on the field of view of the camera, the size of the target image, and the two-dimensional coordinates of the target hand in the target image. Then, based on the rotation relationship between the camera coordinate system and the robot coordinate system, this direction can be converted into the target hand direction in the robot coordinate system.

[0107] When the hand position information includes the three-dimensional coordinates of the target hand in the camera coordinate system, the direction from the optical center of the camera to the reference point of the target hand can be determined based on the three-dimensional coordinates, thus obtaining the direction of the target hand in the camera coordinate system. Based on the rotation relationship between the camera coordinate system and the robot coordinate system, this direction can be converted into the target hand direction in the robot coordinate system.

[0108] In other embodiments, the depth value of the target hand can be obtained by combining a depth camera, a binocular camera, or a distance sensor. Based on the two-dimensional coordinates of the target hand in the target image, the depth value, and the camera's intrinsic parameters, the three-dimensional coordinates of the target hand in the camera coordinate system are determined. Then, based on the camera's extrinsic parameters, the three-dimensional coordinates are transformed to the robot coordinate system. Thus, the direction of the target hand pointing from the preset reference point to the target hand reference point can be determined based on a preset reference point in the robot coordinate system and the target hand's three-dimensional coordinates in the robot coordinate system.

[0109] It should be noted that the orientation of the target hand in the robot coordinate system refers to the spatial direction represented with respect to the coordinate axes of the robot coordinate system. When determining the orientation of the target hand based on two-dimensional coordinates, the orientation can be represented in the robot coordinate system as the direction of a ray originating from the optical center of the camera, and is not limited to the direction pointing from the origin of the robot coordinate system to the target hand. The orientation of the target hand can be used to quantitatively characterize the position of the target hand relative to the companion robot.

[0110] Step 603: Determine the target display position based on the target's hand direction.

[0111] In this embodiment, the direction of the target hand is spatial orientation information. By converting the direction of the target hand into a two-dimensional display position in the eye image through a preset mapping relationship, the target display position can be obtained.

[0112] In this embodiment, the direction of the target hand in the robot coordinate system is determined based on the camera calibration parameters and the two-dimensional or three-dimensional coordinates of the target hand. Then, the target display position in the eye image is determined based on the direction of the target hand. This enables the virtual gaze direction of the eyeball to correspond to the spatial orientation of the target hand, thereby improving the accuracy and naturalness of the eyeball's gaze direction.

[0113] In some embodiments of this application, determining the target display position based on the target hand orientation includes: Determine the current gaze direction corresponding to the current display position in the robot coordinate system; Calculate the horizontal and / or vertical angle differences between the current gaze direction and the target hand direction; The horizontal angle difference is converted into the horizontal displacement of the eyeball in the eye image, and / or the vertical angle difference is converted into the vertical displacement of the eyeball in the eye image; Determine the target display position based on the current display position and the horizontal and / or vertical displacement.

[0114] The current gaze direction refers to the virtual gaze direction corresponding to the current display position of the eyeball. This current gaze direction is not the actual optical axis direction of the physical eyeball, but rather a gaze direction simulated based on the eyeball's display position in the eye image. The current gaze direction can be represented using the coordinate axes of the robot coordinate system. Similarly, the target hand direction is represented using the coordinate axes of the robot coordinate system. Thus, the horizontal and / or vertical angle differences between the current gaze direction and the target hand direction can be calculated; these angle differences are then converted into the display displacement of the eye image through a preset mapping relationship. The preset mapping relationship refers to the mapping relationship between the angle differences in the robot coordinate system and the displacement in the eye image.

[0115] In some embodiments, a gaze direction mapping relationship between the display position of the eyeball in the eye image and the virtual gaze direction can be established in advance, and the current gaze direction can be determined according to the current display position and the gaze direction mapping relationship. For example, the reference display position can correspond to a preset reference gaze direction; when the eyeball is located to the left, right, above or below the reference display position, it can correspond to a virtual gaze direction that is deflected to the left, right, upward or downward relative to the preset reference gaze direction, respectively.

[0116] In some embodiments, the current horizontal and vertical angles of the eyeball can be calculated based on the current gaze direction; the target horizontal and vertical angles of the eyeball can be calculated based on the target hand direction; the current horizontal angle and the target horizontal angle can respectively represent the deflection angles of the current gaze direction and the target hand direction relative to a preset horizontal reference direction in the robot coordinate system; the current vertical angle and the target vertical angle can respectively represent the deflection angles of the current gaze direction and the target hand direction relative to a preset vertical reference direction in the robot coordinate system. Subtracting the current horizontal angle from the target horizontal angle of the eyeball yields the horizontal angle difference between the current gaze direction and the target hand direction; subtracting the current vertical angle from the target vertical angle of the eyeball yields the vertical angle difference between the current gaze direction and the target hand direction; the horizontal angle difference and the vertical angle difference can be angle differences with positive and negative signs, where the absolute value of the angle difference represents the degree of deflection in the corresponding direction, and the positive or negative sign of the angle difference represents the direction of deflection in the corresponding direction.

[0117] A preset mapping relationship refers to the mapping relationship between angular differences in the robot coordinate system and displayed displacement in the eye image. Based on the preset mapping relationship, horizontal angular differences can be converted into horizontal displacement of the eyeball in the eye image, and vertical angular differences can be converted into vertical displacement of the eyeball in the eye image. The horizontal and vertical displacements can be displacements with positive and negative signs; their absolute values ​​represent the distance the eyeball moves, and their positive and negative signs represent the direction of eyeball movement.

[0118] The preset mapping relationship can be a linear mapping relationship, a piecewise mapping relationship, a nonlinear mapping relationship, or a preset lookup table. For example, the horizontal displacement can be determined by multiplying the horizontal angle difference by a preset horizontal transformation coefficient, and the vertical displacement can be determined by multiplying the vertical angle difference by a preset vertical transformation coefficient. The signs of the preset horizontal and vertical transformation coefficients can be determined based on the coordinate axes of the robot coordinate system and the eye image.

[0119] After determining the horizontal and / or vertical displacement amounts, these amounts can be superimposed onto the display coordinates of the current display position to obtain the target display position. By determining only the horizontal displacement amount, the vertical display coordinates of the eyeball can be kept constant; conversely, by determining only the vertical displacement amount, the horizontal display coordinates of the eyeball can be kept constant.

[0120] In some embodiments, eye display control instructions can be generated based on the current display position and the target display position. In response to the eye display control instructions, the display area of ​​the eye is controlled to move from the current display position to the target display position in the eye image along the horizontal and / or vertical directions, so that the virtual gaze direction of the eye changes from the current gaze direction to the direction of the target hand.

[0121] In some embodiments, if the absolute value of the calculated horizontal angle difference is less than the horizontal angle update threshold, and the absolute value of the calculated vertical angle difference is less than the vertical angle update threshold, the eye display control command may not be updated, and the eye may be kept in the current display position to reduce the risk of the eye frequently moving back and forth near the display position corresponding to the target hand. When only the horizontal angle difference is calculated, the eye display control command may not be updated when the absolute value of the horizontal angle difference is less than the horizontal angle update threshold; similarly, when only the vertical angle difference is calculated, the eye display control command may not be updated when the absolute value of the vertical angle difference is less than the vertical angle update threshold. Optionally, preset amplitude thresholds, horizontal angle update thresholds, and vertical angle update thresholds can be set according to actual needs or empirical values.

[0122] In some embodiments, the control device 103 may also employ a hysteresis threshold to control the updating of eye-view control commands. For example, a first angle threshold and a second angle threshold less than the first angle threshold can be set; when the absolute value of the angle difference between the current gaze direction and the target hand direction is greater than the first angle threshold, the eye-view control command is updated; after the update begins, when the absolute value of the angle difference decreases to below the second angle threshold, the eye-view control command update is stopped. By setting different start and stop update thresholds, the risk of frequent eye movements caused by fluctuations in the angle difference around a single threshold can be reduced.

[0123] In some embodiments, the control device 103 may also employ Kalman filtering, exponential smoothing filtering, or other time-series filtering methods to filter the continuously determined target hand direction or target display position, and generate eye display control commands based on the filtered target hand direction or target display position. For example, the filtered target display position at the current moment can be obtained by weighting the currently determined target display position and the target display position obtained from the previous filtering. By performing time-series filtering on the target hand direction or target display position, the risk of eye display position jittering when the hand detection result fluctuates slightly can be reduced.

[0124] In some embodiments, the amount of display displacement to be calculated can be determined based on the display movement direction supported by the eyeball in the eye image. When the eyeball only supports horizontal movement, the target horizontal coordinate can be determined by the sum of the current horizontal coordinate of the current display position and the horizontal displacement, while keeping the current vertical coordinate of the current display position unchanged. The target display position is then determined by the target horizontal coordinate and the current vertical coordinate.

[0125] When the eyeball only supports vertical movement, the sum of the current vertical coordinates of the current display position and the vertical displacement can be used to determine the target vertical coordinates, while keeping the current horizontal coordinates of the current display position unchanged. The target display position is then determined by the current horizontal coordinates and the target vertical coordinates.

[0126] With eye support for movement in both the horizontal and vertical directions, the sum of the current horizontal coordinate and the horizontal displacement can be used to determine the target horizontal coordinate, and the sum of the current vertical coordinate and the vertical displacement can be used to determine the target vertical coordinate. The target display position is then determined by the target horizontal and vertical coordinates. When there is no angular deviation between the target hand direction and the current gaze direction in one of the directions, the display displacement in that direction can be set to zero.

[0127] In this embodiment, by converting the horizontal and / or vertical angle difference between the current gaze direction and the target hand direction into horizontal and / or vertical displacement amounts in the eye image, and determining the target display position based on the current display position and the display displacement amounts, the display position of the eyeball can be made to change accordingly based on the target hand direction. Furthermore, by setting an angle update threshold, using a hysteresis threshold, or performing temporal filtering, the risk of frequent eyeball repetitive movements caused by small fluctuations in hand detection results can be reduced, improving the stability and naturalness of eyeball movement.

[0128] In some embodiments of this application, controlling the eyeball to move from the current display position to the target display position in the eye image includes: Interpolate the current display position and the target display position to determine at least one intermediate display position between the current display position and the target display position; Control the eye to move from the current display position to the target display position via at least one intermediate display position.

[0129] In some embodiments, when there are multiple intermediate display positions, the display position of the eyeball is controlled to move from the current display position through each intermediate display position to the target display position according to the interpolation order corresponding to each intermediate display position.

[0130] In some embodiments, interpolation can be performed based on the display coordinates of the current display position and the display coordinates of the target display position. Assume the current display position is... The target display location is It needs to be determined The middle display position, If it is a positive integer, then the first... The middle display position It can be determined according to the following formula:

[0131] in, greater than or equal to 1 and less than or equal to 1 integers, For the first The interpolation coefficients corresponding to the intermediate display positions are:

[0132] Therefore, the at least one intermediate display position can be arranged in a direction pointing from the current display position to the target display position. When only one intermediate display position is determined, the eye can be controlled to move from the current display position through that intermediate display position to the target display position; when multiple intermediate display positions are determined, the eye can be controlled to move sequentially through the intermediate display positions to the target display position according to their arrangement. For example, for ease of explanation, let's take the one-dimensional display coordinates of the eye in the horizontal direction as an example. Assuming the current horizontal display coordinate of the eye is 0, the target horizontal display coordinate is 10, and the vertical display coordinate remains unchanged, then four intermediate display positions with horizontal display coordinates of 2, 4, 6, and 8 can be determined by linear interpolation. By updating the horizontal display coordinates of the eye to 0, 2, 4, 6, 8, and 10 sequentially at multiple consecutive display moments, the eye can gradually move through the four intermediate display positions to the target display position, avoiding the direct update of the eye's display position from horizontal display coordinate 0 to horizontal display coordinate 10, thus reducing the risk of visual jumps caused by eye movement. In this example, the interpolation coefficients corresponding to each intermediate display position are 0.2, 0.4, 0.6 and 0.8, respectively.

[0133] In some embodiments, the display position of the eyeball can be updated sequentially according to a preset display time interval or the image refresh cycle of the eye screen, so that the eyeball is displayed sequentially at each intermediate display position and the target display position. Each intermediate display position may correspond to one or more display frames of the eye screen.

[0134] The number of intermediate display positions, the distance between adjacent display positions, and / or the display time interval between adjacent display positions can be determined based on the distance between the current display position and the target display position, the preset eye movement speed, the preset duration for the eye to complete this movement, or the refresh rate of the eye screen. For example, the greater the distance between the current display position and the target display position, the more intermediate display positions can be set.

[0135] Interpolation methods can include linear interpolation, piecewise linear interpolation, spline interpolation, Bézier curve interpolation, or interpolation using a preset easing function. When using linear interpolation, the distance between adjacent display positions can be the same; when using a preset easing function, the display position of the eyeball can change in a manner that first accelerates and then decelerates, gradually accelerates, or gradually decelerates. All intermediate display positions obtained through interpolation can be within the movable range of the eyeball in the eye image.

[0136] In some embodiments, the display distance between the current display position and the target display position can be calculated. The display distance can be the pixel distance between the current display position and the target display position. If the display distance is greater than a preset distance threshold, the current display position and the target display position can be interpolated, and the eye can be controlled to move through at least one intermediate display position to the target display position, thereby reducing the risk of eye jitter or rapid changes in the eye image. If the display distance is less than or equal to the preset distance threshold, shorter movements usually do not cause a visual jump, so intermediate display positions can be omitted, and the eye's display position can be directly updated from the current display position to the target display position. If the current display position and the target display position coincide, the eye can be controlled to remain in the current display position.

[0137] In this embodiment, by interpolating the current display position and the target display position, an intermediate display position is generated and the eyeball is controlled to move sequentially through each position to the target display position. This enables the eyeball to continuously change to the target display position in the eye image, reducing the risk of rapid eyeball jumps or discontinuous movement, and improving the smoothness and naturalness of eyeball movement.

[0138] In some embodiments of this application, after the eyeball moves from the current display position to the target display position in the eye image, the control device 103 further performs the following steps: When the target hand moves and its orientation changes relative to the companion robot, the eyeballs are controlled to move accordingly in the eye image so that the display orientation of the eyeballs relative to the reference display position changes accordingly with the orientation of the target hand relative to the companion robot.

[0139] Target hand movement can refer to a change in the spatial position of the target hand relative to the companion robot. A change in the target hand's orientation relative to the companion robot can refer to a change in the target hand's orientation within the robot's coordinate system, specifically a change in the horizontal and / or vertical angles of the target hand's orientation.

[0140] For example, when the target hand moves from the left to the right of the companion robot, the horizontal orientation of the target hand relative to the companion robot changes, which can control the eyeballs to move accordingly in the horizontal direction in the eye image; when the target hand moves from below to above the companion robot, the vertical orientation of the target hand relative to the companion robot changes, which can control the eyeballs to move accordingly in the vertical direction in the eye image; when the target hand moves diagonally, the eyeballs can be controlled to move in the corresponding diagonal direction in the eye image.

[0141] When the target hand moves only in the direction of approaching or moving away from the companion robot, and the horizontal and vertical orientation of the target hand relative to the companion robot remains unchanged, the eyeball can be controlled to remain in the current display position. Therefore, movement of the target hand does not necessarily cause eyeball movement; when the movement of the target hand changes its orientation relative to the companion robot, the display position of the eyeball can change accordingly.

[0142] In some embodiments, after the eyeball moves to the target display position, the hand position information of the target hand in the subsequent images captured by the camera can be acquired. The target hand orientation in the robot coordinate system is recalculated based on the subsequently acquired hand position information in accordance with the manner described in steps 601 to 603 above. The updated target display position is then determined based on the updated target hand orientation. The eyeball is then controlled to move from the current display position to the updated target display position in the eye image, so that the display orientation of the eyeball relative to the reference display position is adjusted accordingly as the orientation of the target hand changes.

[0143] In some embodiments, if the target hand moves or its orientation relative to the companion robot changes before the eyeball has moved to the determined target display position, the eyeball can continue to be controlled to move to the determined target display position. After the eyeball moves to the determined target display position, the orientation of the target hand and the updated target display position are re-determined based on subsequently acquired images, and the eyeball is controlled to move to the updated target display position.

[0144] In the embodiments of this application, by adjusting the display orientation of the eyeball in the eye image according to the change in the orientation of the target hand relative to the companion robot, the correspondence between the display orientation of the eyeball and the orientation of the target hand can be maintained, enabling the eyeball to continuously follow the orientation change of the target hand and improving the continuity and naturalness of eye tracking.

[0145] In some embodiments of this application, during the process of controlling the eyeball to move accordingly in the eye image, the control device 103 also performs the following steps: If the eyeball moves to the boundary of the movable range in the eye image, and the target hand continues to move to the side corresponding to the boundary position, then control the eyeball to remain displayed at the boundary position.

[0146] The movable range of the eyeball in an eye image can refer to the set of display positions that the eyeball's reference point is allowed to reach within the eye image. The eyeball's reference point can be the center of the pupil, the center of the iris, or the center of the eyeball display area including both the pupil and iris. The movable range can be a rectangular area, a circular area, an elliptical area, or an area of ​​other preset shapes.

[0147] The boundary position of the movable range can refer to the maximum offset position that the reference point of the eyeball can reach along the corresponding movement direction. In other embodiments, the movable range can also be determined based on the overall contour of the eyeball display area. In this case, the boundary position can be the maximum offset position that the eyeball display area can reach without exceeding the preset display area.

[0148] In some embodiments, the eye image can fill the effective display area of ​​the eye screen, where the preset display area is the effective display area of ​​the eye screen. The effective display area refers to the area on the eye screen capable of displaying image content, excluding the screen's borders or other non-display structures. The eye display area can move within the preset display area. To ensure the eye display area does not exceed the effective display area of ​​the eye screen during movement, the movable range of the eye can be determined based on the size of the eye display area and the size of the preset display area. The movable range of the eye can be the set of display positions that the eye's reference point is allowed to reach while ensuring the entire eye display area remains within the preset display area. The boundary position of the movable range can be the display position of the eye's reference point when the outer contour of the eye display area contacts the edge of the preset display area. Thus, even when the eye's reference point is located at the boundary position of the movable range, the eye display area will not exceed the effective display area of ​​the eye screen. For example, if the effective display area of ​​the eye-tracking screen has a horizontal coordinate range of 0 to W, and the width of the eye-tracking display area is w, with the center of the eye-tracking display area as the reference point for the eye, then when the eye-tracking display area needs to be fully displayed, the horizontal range of motion of the reference point for the eye can be from w / 2 to W-w / 2. Correspondingly, when the reference point for the eye is located at w / 2 or W-w / 2, the eye is positioned at the left or right horizontal boundary, respectively.

[0149] The side corresponding to the boundary position can be determined based on a preset orientation mapping relationship between the orientation of the target hand relative to the companion robot and the display orientation of the eyeball relative to the reference display position. For example, when the target hand is located on the left side of the companion robot and the eyeball moves to the left of the movable range accordingly, the left boundary position of the movable range corresponds to the left side of the companion robot; the right boundary position of the movable range corresponds to the right side of the companion robot; and the upper and lower boundary positions of the movable range correspond to the top and bottom of the companion robot, respectively. If the movable range has a corner boundary position, the corner boundary position can correspond to the oblique orientation of the target hand relative to the companion robot.

[0150] When the eyeball moves to the boundary position and the target hand continues to move to the side corresponding to the boundary position, according to the preset orientation mapping relationship, the eyeball movement direction corresponding to the orientation change of the target hand continues to face outwards from the movable range. Since the eyeball's display position is limited by the movable range, it is not necessary to move the eyeball outwards based on the target hand's orientation change; instead, the eyeball can be controlled to remain displayed at the boundary position. Therefore, merely passing the boundary position does not necessarily trigger the holding function; only when the target hand continues to move to the corresponding side after the eyeball reaches the boundary position will the eyeball be controlled to remain displayed at that boundary position. If the target hand moves in the opposite direction and the mapped display position corresponding to the target hand's direction re-enters the movable range, the eyeball can be controlled to leave the boundary position and move inwards from the movable range. When the companion robot's head and / or torso rotate, causing a change in the target hand's orientation relative to the companion robot, the eyeball can also be controlled to move from the boundary position inwards from the movable range based on the changed orientation.

[0151] In some embodiments, when the target display position is initially determined based on the direction of the target hand, the mapped display position obtained according to a preset orientation mapping relationship may be outside the movable range of the eyeball. In this case, the boundary position within the movable range corresponding to the direction of the target hand can be determined as the target display position, and the eyeball can be controlled to move from the current display position to that boundary position. While the mapped display position corresponding to the target hand remains outside the movable range, the eyeball can be controlled to remain displayed at that boundary position. The boundary position corresponding to the direction of the target hand can be the position where the direction from the reference display position to the mapped display position intersects with the boundary of the movable range, or it can be the position on the boundary of the movable range closest to the mapped display position.

[0152] In this embodiment, when the eyeball moves to the boundary of the movable range and the target hand continues to move to the corresponding side, controlling the eyeball to remain displayed at the boundary position can prevent the displayed position of the eyeball from exceeding the preset movable range, reducing the risk of the eyeball display area going out of bounds, being cropped, or changing position. Simultaneously, keeping the eyeball at the boundary position corresponding to the location of the target hand can continue to represent that the target hand is located on the corresponding side of the companion robot, allowing the eyeball to maintain a continuous and natural gaze even under boundary conditions.

[0153] In some embodiments of this application, after the eyeball remains displayed at the boundary position, the control device 103 further performs the following steps: If the eyeball remains displayed at the boundary position for a duration exceeding a preset duration, the rotating part of the companion robot is controlled to rotate so that the rotating part faces the tracking target, which is either the target hand or the target object, and the target object is the object to which the target hand belongs. Among them, the rotating part is at least one of the head and the torso. The head facing the tracking target means that the facial area of ​​the head faces the tracking target, and the torso facing the tracking target means that the front of the torso faces the tracking target. During or after the rotation of the rotating part, control the eyeball to move from the boundary position to the reference display position.

[0154] The "front" of the body can refer to the side of the body that corresponds to a preset body orientation. The preset body orientation can be determined in advance based on the structure of the companion robot, for example, it can be the direction in which the facial area of ​​the head faces when the companion robot is in its initial state. For a body with a planar frontal structure, the front of the body can be the side containing that planar structure; for a cylindrical, arc-shaped, or other body that does not have a clear front, the front of the body can be a preset side area centered on the preset body orientation.

[0155] Optionally, the preset duration can be set based on the allowed dwell time of the eye at the boundary position, the response speed of the companion robot, or the expected head-eye coordination effect. The duration for which the eye remains displayed at the boundary position can be calculated from the moment the eye begins to remain displayed at the boundary position. If the eye leaves the boundary position before the preset duration is reached, the timer can be stopped or the recorded duration can be reset to zero.

[0156] In some embodiments, "head facing the tracking target" can mean that the angle between the direction in which the facial region of the head faces and the direction from the head to the tracking target is less than a preset facing angle threshold. "Torso facing the tracking target" can mean that the angle between a preset torso orientation and the direction from a preset reference point on the torso to the tracking target is less than or equal to a preset facing angle threshold. For a torso that can only rotate horizontally, the angle between the preset torso orientation and the projection of the direction from the torso to the tracking target onto the horizontal plane can be used to determine whether the front of the torso is facing the tracking target.

[0157] In some embodiments, head rotation may refer to the head rotating relative to the torso in a horizontal and / or pitch direction, while the torso remains stationary relative to the ground. Torso rotation may refer to the torso rotating horizontally relative to the ground, while the head remains stationary relative to the torso. The head and torso may also rotate simultaneously or sequentially to align the rotating parts toward the tracking target. The head and torso may be driven by corresponding drive motors or other drive mechanisms.

[0158] When the eyeball remains displayed at the boundary position for a duration exceeding the preset duration, it indicates that the eyeball display position corresponding to the location of the target hand or the target object is continuously at the boundary position of the movable range, or the corresponding mapped display position is continuously outside the movable range. At this time, the rotating part of the companion robot can be controlled to rotate, changing the orientation of the rotating part relative to the tracking target, so that the orientation of the tracking target relative to the companion robot changes in the direction corresponding to the interior of the movable range of the eyeball.

[0159] In some embodiments, the reference display position of the eyeball can correspond to a preset reference orientation of the rotating part. When the head and / or torso rotates towards the tracking target, the preset reference orientation of the rotating part changes accordingly, and the orientation deviation of the tracking target relative to the preset reference orientation decreases accordingly, causing the eyeball display position corresponding to the orientation of the tracking target to move closer to the reference display position from the boundary position of the movable range. During the rotation of the rotating part, the eyeball can be gradually controlled to move from the boundary position to the reference display position according to the real-time rotation angle of the rotating part or the real-time orientation of the tracking target relative to the companion robot. Alternatively, after the rotating part completes its rotation, the orientation of the tracking target relative to the companion robot can be re-determined, and the eyeball can be controlled to move from the boundary position to the reference display position according to the re-determined orientation. The reference display position can be the display position where the eyeball is in a forward-looking state. In embodiments where the reference display position is located at the center of the eye image, controlling the eyeball to move to the reference display position can restore the eyeball to the center position of the eye image; in embodiments where the reference display position is not located at the center of the eye image, the eyeball is controlled to return to the preset reference display position.

[0160] In some embodiments, after controlling the eye to move from the boundary position to the reference display position, it is possible to re-identify whether a hand is making any preset gesture from the specified gesture set. If a hand making any preset gesture is identified, it can be determined as the target hand, and step S201 and subsequent steps are executed. If multiple hands making any preset gesture are identified, one hand can be selected as the target hand according to the aforementioned target hand selection process. After re-identifying the target hand, the eye can be controlled to continue tracking based on the re-identified target hand.

[0161] In this embodiment, by triggering the rotation of the rotating part when the eyeball is in the boundary position for a long time, the tracking target can be brought back into the movable range of the eyeball, reducing the risk of visual stiffness caused by the eyeball being in the boundary position for a long time. At the same time, it can make the gaze behavior of the companion robot closer to the head-eye coordination mode of real humans, and improve the stability and human-like performance of the companion robot during long-term tracking.

[0162] In some embodiments of this application, the companion robot also includes a camera for detecting and tracking a target. Before controlling the rotation of the rotating parts of the companion robot, the control device 103 further performs the following steps: If the deflection angle between the direction the camera points to the target and the camera's shooting direction is within the first angle range, then the head is identified as the rotating part. If the deflection angle is within the second angle range, then the torso is identified as the rotating part; Among them, the deflection angles within the first angle range are all smaller than the deflection angles within the second angle range.

[0163] In some embodiments, the camera's shooting direction can be the direction of the camera's optical axis or the direction corresponding to the center of the camera's field of view. The direction from the camera to the tracking target can be represented by the direction from the camera's optical center to the tracking target reference point. When the tracking target is a hand, the tracking target reference point can be the center of the hand detection frame, the center of the palm, or a key point of the hand; when the tracking target is an object, the tracking target reference point can be the center of the object's face, the center of its body, or the center of its human body detection frame.

[0164] Given the three-dimensional coordinates of the target reference point in the camera coordinate system, the direction from the camera to the target can be determined based on the direction from the origin of the camera coordinate system to the corresponding position of the three-dimensional coordinates. Alternatively, given the two-dimensional coordinates of the target reference point in the image captured by the camera, the camera ray passing through the target reference point from the camera's optical center can be determined based on the camera's intrinsic parameters and the two-dimensional coordinates, and the direction of this ray can be taken as the direction from the camera to the target.

[0165] The deflection angle can be the angle between the direction the camera points towards the target and the camera's shooting direction. When the body can only rotate horizontally, the deflection angle can be the angle between the projections of the two directions onto the horizontal plane, i.e., the deflection angle in the horizontal direction.

[0166] Optionally, if the deflection angles within the first angle range are all smaller than the deflection angles within the second angle range, the first angle range and the second angle range can be set according to actual needs or empirical values.

[0167] In some embodiments, a first angle threshold and a second angle threshold can be set, where the first angle threshold is less than the second angle threshold. The first angle range can be an angle range where the deflection angle is greater than the first angle threshold and less than or equal to the second angle threshold, and the second angle range can be an angle range where the deflection angle is greater than the second angle threshold. When the deflection angle is less than or equal to the first angle threshold, head or body rotation may not be controlled, or the target may be tracked simply by adjusting the display position of the eyeballs.

[0168] The first and second angle thresholds can be set based on the camera's field of view, the maximum head rotation angle, the body's rotation capability, the head and body rotation speed, and / or the expected tracking response speed.

[0169] In some embodiments of this application, by selecting the head or torso as the rotation part according to the size of the deflection angle, only the head can be rotated when the tracking target deviates slightly from the shooting direction, so as to quickly align with the tracking target in a lightweight and flexible manner. When the tracking target deviates significantly from the shooting direction, the torso can be rotated to achieve a wide range of orientation adjustment, reducing the risk of excessive twisting or tracking failure caused by relying solely on head rotation. This improves the rationality and naturalness of the companion robot's orientation adjustment while ensuring tracking continuity.

[0170] In some embodiments, a third angle range can be set, where the deflection angles within the third angle range are all greater than those within the second angle range. If the deflection angle falls within the third angle range, the head and torso can be identified as rotating parts, allowing them to rotate together. This enables rapid, large-scale orientation adjustments when the target deviates significantly from the field of view, reducing the risk of tracking interruption due to insufficient rotation range or slow rotation speed of a single rotating part, and further improving the tracking response capability of the companion robot in scenarios with large angle deviations. When both the head and torso are identified as rotating parts, the head and torso can be controlled to rotate simultaneously, or they can be controlled to rotate sequentially according to a preset order.

[0171] Optionally, if the deflection angles within the third angle range are all greater than the deflection angles within the second angle range, the third angle range can be set according to actual needs or empirical values.

[0172] Optionally, a first angle threshold, a second angle threshold, and a third angle threshold can be set in ascending order, and the head and / or torso can be selected as the rotation part based on the range of the deflection angle. When the deflection angle is greater than the first angle threshold and less than or equal to the second angle threshold, the head can be identified as the rotation part; when the deflection angle is greater than the second angle threshold and less than or equal to the third angle threshold, the torso can be identified as the rotation part; when the deflection angle is greater than the third angle threshold, both the head and the torso can be identified as the rotation part.

[0173] In some embodiments, during the process of controlling the eyeball to move accordingly in the eye image, the eye tracking method can be adjusted according to the movement state of the target hand, so that the eyeball moves towards the target hand at a restricted speed, reducing the risk of the eyeball frequently swaying at large angles while following the target hand. The control device 103 can adjust the eye tracking method according to the movement state of the target hand by performing the following steps: Determine the target's hand movement speed; If the movement speed exceeds a preset speed threshold, the eyeball will be controlled using any of the following methods: The first predicted hand position of the target hand is determined based on the movement trajectory of the target hand, the first predicted display position of the eyeball in the eye image is determined based on the first predicted hand position, and the eyeball is controlled to move towards the first predicted display position. Alternatively, reduce the speed at which the eye moves in the eye image and control the eye to move at the reduced speed.

[0174] Optionally, a preset speed threshold can be set according to actual needs or experience. The movement trajectory of the target hand can consist of multiple target hand positions arranged according to the acquisition time.

[0175] In some embodiments, multiple two-dimensional positions of the target hand in continuous images can be determined based on images continuously captured by a camera on the companion robot. Then, based on these multiple two-dimensional positions and the corresponding image acquisition times, the moving speed and trajectory of the target hand in the image coordinate system can be determined. The moving speed of the target hand in the image coordinate system can be represented by the pixel distance moved per unit time or by a normalized image distance.

[0176] In other embodiments, multiple three-dimensional positions of the target hand in the camera coordinate system or robot coordinate system can be determined based on depth information. Then, based on these multiple three-dimensional positions and their corresponding acquisition times, the spatial movement velocity and trajectory of the target hand can be determined. The spatial movement velocity of the target hand can be represented by the spatial distance traveled per unit time. For example, if the target hand's position at a first moment is the first hand position, and its position at a second moment is the second hand position, the movement velocity of the target hand can be determined based on the distance between the first and second hand positions and the time interval between the first and second moments. The average movement velocity, direction of movement, and / or acceleration of the target hand can also be determined based on the target hand positions at three or more consecutive moments.

[0177] The first predicted hand position can refer to the position that the target hand may reach at the first prediction time, predicted based on the historical movement trajectory of the target hand. The first prediction time can be the next image acquisition time, the next eyeball display position update time, or a preset time after the current time.

[0178] In some embodiments, the first predicted hand position can be determined by linear extrapolation based on the position and movement speed of the target hand at the two most recent moments. Alternatively, the first predicted hand position can be determined by linear fitting, polynomial fitting, Kalman prediction, or other trajectory prediction methods based on the position of the target hand at multiple historical moments.

[0179] When the first predicted hand position is represented by two-dimensional coordinates in the target image, the first predicted hand orientation in the robot coordinate system can be determined based on the first predicted hand position, the depth information of the target hand, and the camera calibration parameters. When the first predicted hand position is represented by three-dimensional coordinates in the camera coordinate system, the first predicted hand position can be transformed to the robot coordinate system based on the camera extrinsic parameters, and the first predicted hand orientation can be determined accordingly. When the first predicted hand position is already represented by three-dimensional coordinates in the robot coordinate system, the first predicted hand orientation can be directly determined based on the first predicted hand position, and the first predicted display position can be determined based on the first predicted hand orientation. The implementation method of determining the first predicted display position based on the first predicted hand orientation is the same as the implementation method of determining the target display position based on the target hand orientation, and will not be described again in this application.

[0180] In some embodiments, the control device 103 controls the eye movement speed to decrease from a preset normal movement speed to a preset limit movement speed, or limits the eye movement speed to no more than the preset limit movement speed, and controls the eye to move at the reduced movement speed. This reduces frequent and significant changes in display position caused by the eye following a rapidly moving target hand. After the target hand's movement stabilizes, the target hand's direction and target display position can be redefined based on the target hand's current position, and the eye movement towards the redefined target display position can be resumed using the preset normal movement speed.

[0181] In this embodiment, by determining the first predicted hand position based on the movement trajectory of the target hand and controlling the eyeball to move towards the corresponding first predicted display position, the lag generated when the eyeball tracks the target hand can be reduced. By reducing or limiting the movement speed of the eyeball, the frequent and large-amplitude movements of the eyeball in order to follow the fast-moving target hand can be reduced, thereby reducing the risk of visual tremors or rapid jumps and improving the stability and anthropomorphic performance of the interaction process.

[0182] In some embodiments of this application, during the process of controlling the eyeball to move accordingly in the eye image, the control device 103 also performs the following steps: If the target hand is detected to be at least partially obscured, the second predicted hand position of the target hand is determined based on the movement trajectory of the target hand before it was obscured. The second predicted display position of the eyeball in the eye image is determined based on the second predicted hand position, and the eyeball is controlled to move towards the second predicted display position.

[0183] The target hand is at least partially occluded, which can mean that the corresponding image area is occluded by other objects.

[0184] In some embodiments, it can be determined whether the target hand is partially occluded based on the proportion of the visible area of ​​the target hand, the confidence level of hand detection, the degree of overlap between the image area corresponding to the target hand and the image area corresponding to the occluded object, or the number of detected key points of the hand.

[0185] In other embodiments, except where the target hand is at least partially obscured, when the target hand is not detected for a short period, a second predicted hand position can be determined based on the movement trajectory of the target hand before it was not detected. A second predicted display position is then determined based on the second predicted hand position, and the eye is controlled to move towards the second predicted display position. A short period of non-detection of the target hand may mean that the duration of continuous non-detection of the target hand since the most recent detection has not reached a preset prediction duration. During the period before the preset prediction duration, the position of the target hand at different prediction times can be predicted based on the movement trajectory of the target hand before the most recent detection, and the second predicted display position is continuously updated based on the predicted second predicted hand position. When the target hand is detected again, the target hand direction and target display position can be re-determined based on the actual position of the re-detected target hand, and the use of the second predicted hand position to control the eye is stopped.

[0186] In some embodiments, if the target hand is detected to be at least partially obscured, the eye can be controlled to maintain the most recent gaze direction for a preset time. When the target hand is detected again within the preset time, tracking of the target hand can continue.

[0187] The second predicted hand position can refer to the position that the target hand may reach at the second prediction time, predicted based on the movement trajectory of the target hand before it is occluded. The second prediction time can be the next image acquisition time after the target hand is occluded, the next eyeball display position update time, or a preset time after the target hand is occluded.

[0188] In some embodiments, when the target hand is detected to be at least partially obscured or is not detected for a short period of time, the historical movement trajectory of the target hand before it was obscured or not detected can be obtained, and the second predicted hand position can be determined according to the trajectory prediction method described above for determining the first predicted hand position. The difference between the first and second predicted hand positions is that the first predicted hand position is the predicted position determined based on the movement trajectory of the target hand when the movement speed of the target hand exceeds a preset speed threshold, while the second predicted hand position is the predicted position determined based on the historical movement trajectory of the target hand before it is obscured or not detected when the target hand is at least partially obscured or is not detected for a short period of time.

[0189] The coordinate representation of the second predicted hand position, and the method for determining the second predicted hand direction and the second predicted display position based on the second predicted hand position, can be the same as the aforementioned implementation methods of the first predicted hand position, the first predicted hand direction, and the first predicted display position, and will not be repeated here.

[0190] When a hand making the designated gesture that triggers eye control is re-identified, the use of the second predicted hand position to control the eyes can be stopped, and the target hand and target display position can be re-identified according to the aforementioned target hand identification method. The re-identified hand does not need to be the same hand as the target hand before occlusion.

[0191] In some embodiments of this application, by predicting the position of the target hand based on its movement trajectory before occlusion when the target hand is partially occluded, and controlling the eye to move toward the predicted display position, the gaze direction during occlusion can be effectively connected, reducing the risk of the eye losing the target or swinging back and forth due to the target hand being temporarily invisible, and improving the continuity and stability of eye tracking in occluded scenarios.

[0192] In some embodiments of this application, the specified gesture is any preset gesture from a preset gesture set. After the eyeball moves from the current display position to the target display position in the eye image, the control device 103 further performs the following steps: If the duration of a hand making any preset gesture from the preset gesture set is not continuously recognized reaches the first preset duration, then the eyeball is controlled to remain at the first display position where the eyeball is when the duration reaches the first preset duration; If, after a second preset time has elapsed since the eye has been held in the first display position, no hand making any preset gesture from the preset gesture set has been detected, the eye is moved to the reference display position, or, after the target face is identified, the eye is moved to the second display position corresponding to the location of the target face.

[0193] The preset gesture set may include at least one preset gesture. Any preset gesture in the preset gesture set can be used as the designated gesture to trigger eye control.

[0194] The first display position is the actual display position of the eyeball in the eye image when the first preset time period expires. Depending on the eyeball control method within the first preset time period, the first display position can be the target display position, the second predicted display position, the intermediate display position during eyeball movement, the boundary position of the movable range, or other actual display positions.

[0195] The first preset duration can be the duration for continuing detection or prediction after failing to recognize a hand making any preset gesture. The second preset duration can be the duration for keeping the eye in the first display position while waiting for the preset gesture to be recognized again. The first and second preset durations are two consecutive time stages. Optionally, the first and second preset durations can be set according to actual needs or empirical values. The first and second preset durations can be the same or different, and this application does not limit this.

[0196] If a hand making any preset gesture from the preset gesture set is re-identified before the first preset time expires, the timer can be stopped, and the target hand and target display position can be determined based on the re-identified hand, while eye movement continues. If no hand making any preset gesture is identified before the first preset time expires, the eye remains displayed in the first display position, and a second preset time is calculated from the moment the display in the first display position begins. If a hand making any preset gesture is re-identified before the second preset time expires, that hand can be designated as the target hand; if multiple hands making any preset gesture are re-identified, one hand can be selected as the target hand according to the aforementioned target hand selection process. The re-identified hand does not need to be the same hand as the previous target hand. After re-identifying the target hand, the corresponding target display position can be determined based on the re-identified target hand's position relative to the companion robot, and the eye moves from the first display position to the target display position. If no hand making any preset gesture is recognized when the second preset time expires, the eyeball can be controlled to move from the first display position to the reference display position; alternatively, the target face can be identified, and the second display position can be determined based on the position of the target face relative to the companion robot, and the eyeball can be controlled to move from the first display position to the second display position.

[0197] The target face can be the face of the same target object as the previously determined target hand, i.e., the face of the target object making the specified gesture, or the face of the object currently being interacted with by the companion robot, or the face determined based on the distance between the face and the companion robot, the confidence level of face recognition, the position of the face in the image, or other preset face selection information.

[0198] The location of the target face can be represented by the direction from a preset reference point on the companion robot to a target face reference point. The target face reference point can be the center of the face detection box, the center of the facial features, or other preset facial key points. The second display position can be determined based on the location of the target face according to the preset orientation mapping relationship between the spatial orientation and the eye display orientation. For example, when the target face is above the companion robot, the second display position can be located above the reference display position; when the target face is to the left of the companion robot, the second display position can be located to the left of the reference display position. Figure 7 The image shown is an example of a target face being positioned to the left of the companion robot, with the eyes located to the left of the baseline display position.

[0199] In this embodiment of the application, by keeping the eyeball in the first display position when the preset gesture is not recognized continuously, and then restoring it to the reference display position or turning it to the target face after a second preset time, the risk of the eyeball immediately resetting or repeatedly swinging due to the instantaneous loss of the target hand can be reduced, and the continuity of the gaze state during the gesture interruption can be maintained.

[0200] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0201] Corresponding to the eye control method for the companion robot described in the above embodiments, Figure 8 A schematic diagram of the eye control device for a companion robot according to an embodiment of this application is shown. The companion robot includes an eye screen for displaying an eye image, which includes the eyeball. For ease of explanation, only the parts relevant to the embodiment of this application are shown.

[0202] Reference Figure 8 The device includes: An eye control module 801 is used to control the eyeball to move from the current display position to the target display position in the eye image if a specified gesture is recognized. Wherein, when the eyeball is located at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot, and the target hand is the hand that makes the specified gesture.

[0203] In some embodiments, the eye control module 801 is further configured to: When the target hand moves and the orientation of the target hand relative to the companion robot changes, the eyeball is controlled to move accordingly in the eye image so that the display orientation of the eyeball relative to the reference display position changes accordingly with the orientation of the target hand relative to the companion robot.

[0204] In some embodiments, the eye control module 801 is further configured to: During the process of controlling the eyeball to move accordingly in the eye image, if the eyeball moves to the boundary position of the movable range in the eye image, and the target hand continues to move to the side corresponding to the boundary position, then the eyeball is controlled to remain displayed at the boundary position.

[0205] In some embodiments, the eye control module 801 is further configured to: After the eyeball remains displayed at the boundary position, if the duration of the eyeball remaining displayed at the boundary position exceeds a preset duration, the rotating part of the companion robot is controlled to rotate so that the rotating part faces the tracking target, the tracking target is the target hand or the target object, and the target object is the object to which the target hand belongs; Wherein, the rotating part is at least one of the head and the torso, the head facing the tracking target means that the facial area of ​​the head faces the tracking target, and the torso facing the tracking target means that the front of the torso faces the tracking target; During the rotation of the rotating part or after the rotating part has completed its rotation, the eyeball is controlled to move from the boundary position to the reference display position.

[0206] In some embodiments, the companion robot further includes a camera for detecting the tracked target, and the eye control module 801 is further configured to: Before controlling the rotation of the companion robot's rotating part, if the deflection angle between the direction the camera points to the tracking target and the camera's shooting direction is within a first angle range, then the head is identified as the rotating part; If the deflection angle is within the second angle range, then the body is identified as the rotating part; Wherein, the deflection angles within the first angle range are all smaller than the deflection angles within the second angle range.

[0207] In some embodiments, the eye control module 801 is further configured to: During the process of controlling the eyeball to move accordingly in the eye image, the movement speed of the target hand is determined; If the movement speed exceeds a preset speed threshold, the eyeball will be controlled in any of the following ways: The first predicted hand position of the target hand is determined based on the movement trajectory of the target hand, the first predicted display position of the eyeball in the eye image is determined based on the first predicted hand position, and the eyeball is controlled to move towards the first predicted display position; Alternatively, the movement speed of the eyeball in the eye image can be reduced, and the eyeball can be controlled to move at the reduced movement speed.

[0208] In some embodiments, the eye control module 801 is further configured to: During the process of controlling the eyeball to move accordingly in the eye image, if it is detected that the target hand is at least partially obscured, then the second predicted hand position of the target hand is determined based on the movement trajectory of the target hand before it is obscured. The second predicted display position of the eyeball in the eye image is determined based on the second predicted hand position, and the eyeball is controlled to move toward the second predicted display position.

[0209] In some embodiments, the specified gesture is any preset gesture in a preset gesture set, and the eye control module 801 is further configured to: If the duration for which no hand making any preset gesture in the preset gesture set is continuously not recognized reaches a first preset duration, then the eyeball is controlled to remain at the first display position where the eyeball is when the duration reaches the first preset duration; If, after a second preset time has elapsed since the eyeball has been held at the first display position, no hand making any of the preset gestures in the preset gesture set has been identified, then the eyeball is controlled to move to the reference display position, or after the target face is determined, the eyeball is controlled to move to the second display position corresponding to the location of the target face.

[0210] In some embodiments, the companion robot further includes a camera, and the aforementioned device further includes: The location acquisition module 802 is used to acquire the hand position information of the target hand, the hand position information including the two-dimensional coordinates of the target hand in the target image or the three-dimensional coordinates of the target hand in the camera coordinate system, and the target image is an image captured by the camera that recognizes the specified gesture; The orientation determination module 803 is used to determine the orientation of the target hand in the robot coordinate system based on the camera calibration parameters and the hand position information; The position determination module 804 is used to determine the target display position based on the direction of the target's hand.

[0211] In some embodiments, the position determination module 804 is specifically used for: Determine the current gaze direction corresponding to the current display position in the robot coordinate system; Calculate the horizontal and / or vertical angle difference between the current gaze direction and the direction of the target hand; The horizontal angle difference is converted into the horizontal displacement of the eyeball in the eye image, and / or the vertical angle difference is converted into the vertical displacement of the eyeball in the eye image; The target display position is determined based on the current display position and the horizontal displacement and / or the vertical displacement.

[0212] In some embodiments, the eye control module 801 is specifically used for: Interpolate the current display position and the target display position to determine at least one intermediate display position between the current display position and the target display position; Control the eyeball to move from the current display position to the target display position via at least one intermediate display position.

[0213] In some embodiments, the above-described apparatus further includes: The candidate determination module 805 is used to determine the multiple hands as candidate hands when multiple hands making the specified gesture are identified. The target determination module 806 is used to acquire target selection information for each of the candidate hands, and select one of the candidate hands as the target hand based on the target selection information; The target selection information includes at least one of the following: the position information of the candidate hand, the confidence level of the gesture recognition of the candidate hand, the duration of the candidate hand making the specified gesture, and the judgment result of whether the candidate hand belongs to the current interactive object.

[0214] In some embodiments, the companion robot further includes a camera, and the target determination module 806 is specifically used for: Acquire target images including multiple candidate hands captured by the camera; Based on the position information of each candidate hand, a first distance between the image reference point of each candidate hand in the target image and the image center of the target image, and a second distance between each candidate hand and the companion robot are determined. A first selection score is determined based on the first distance, and a second selection score is determined based on the second distance, wherein the first selection score increases as the first distance decreases, and the second selection score increases as the second distance decreases; The first selection score and the second selection score of the same candidate hand are weighted and summed to obtain the target selection score of each candidate hand; The candidate hand with the highest target selection score is determined as the target hand.

[0215] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0216] Figure 9 This is a schematic diagram of the structure of the companion robot provided in an embodiment of this application. Figure 9 As shown, the companion robot of this embodiment includes: at least one processor 90 ( Figure 9 The system includes, but is only shown in one example, a memory 91, a computer program 92 stored in the memory 91 and executable on the at least one processor 90, and an eye screen 93 for displaying an image of the eye, including the eyeball. When the processor 90 executes the computer program 92, it causes the companion robot to perform the steps in any of the above-described method embodiments.

[0217] The companion robot may include, but is not limited to, a processor 90, a memory 91, and an eye-screen 93. Those skilled in the art will understand that... Figure 9 This is merely an example of a companion robot and does not constitute a limitation on companion robots. It may include more or fewer parts than shown in the illustration, or a combination of certain parts, or different parts, such as input / output devices, network access devices, etc.

[0218] The processor 90 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0219] In some embodiments, the memory 91 may be an internal storage unit of the companion robot, such as a hard drive or memory. In other embodiments, the memory 91 may be an external storage device of the companion robot, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 91 may include both internal and external storage units of the companion robot. The memory 91 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0220] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0221] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the device / companion robot, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0222] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0223] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0224] In the embodiments provided in this application, it should be understood that the disclosed devices / companion robots and methods can be implemented in other ways. For example, the device / companion robot embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for controlling the eye movements of a companion robot, characterized in that, The companion robot includes an eye screen for displaying an eye image, the eye image including the eyeball, and the method includes: If a specified gesture is detected, the eyeball is controlled to move from the current display position to the target display position in the eye image; Wherein, when the eyeball is located at the target display position, the display orientation of the eyeball relative to the reference display position in the eye image is consistent with the orientation of the target hand relative to the companion robot, and the target hand is the hand that makes the specified gesture.

2. The method according to claim 1, characterized in that, After the eyeball moves from its current display position to the target display position in the eye image, the method further includes: When the target hand moves and the orientation of the target hand relative to the companion robot changes, the eyeball is controlled to move accordingly in the eye image so that the display orientation of the eyeball relative to the reference display position changes accordingly with the orientation of the target hand relative to the companion robot.

3. The method according to claim 2, characterized in that, The process of controlling the eyeball to move accordingly in the eye image also includes: If the eyeball moves to the boundary position of the movable range in the eye image, and the target hand continues to move to the side corresponding to the boundary position, then the eyeball is controlled to remain displayed at the boundary position.

4. The method according to claim 3, characterized in that, After the eyeball remains displayed at the boundary position, the method further includes: If the eyeball remains displayed at the boundary position for a duration exceeding a preset duration, the rotating part of the companion robot is controlled to rotate so that the rotating part faces the tracking target, the tracking target being the target hand or the target object, the target object being the object to which the target hand belongs; Wherein, the rotating part is at least one of the head and the torso, the head facing the tracking target means that the facial area of ​​the head faces the tracking target, and the torso facing the tracking target means that the front of the torso faces the tracking target; During the rotation of the rotating part or after the rotating part has completed its rotation, the eyeball is controlled to move from the boundary position to the reference display position.

5. The method according to claim 4, characterized in that, The companion robot also includes a camera for detecting the tracked target. Before controlling the rotation of the rotating parts of the companion robot, it also includes: If the deflection angle between the direction from which the camera points to the tracked target and the shooting direction of the camera is within a first angle range, then the head is identified as the rotating part; If the deflection angle is within the second angle range, then the body is identified as the rotating part; Wherein, the deflection angles within the first angle range are all smaller than the deflection angles within the second angle range.

6. The method according to claim 2, characterized in that, The process of controlling the eyeball to move accordingly in the eye image further includes: Determine the movement speed of the target hand; If the movement speed exceeds a preset speed threshold, the eyeball will be controlled in any of the following ways: The first predicted hand position of the target hand is determined based on the movement trajectory of the target hand, the first predicted display position of the eyeball in the eye image is determined based on the first predicted hand position, and the eyeball is controlled to move towards the first predicted display position; Alternatively, the movement speed of the eyeball in the eye image can be reduced, and the eyeball can be controlled to move at the reduced movement speed.

7. The method according to claim 2, characterized in that, The process of controlling the eyeball to move accordingly in the eye image further includes: If it is detected that the target hand is at least partially obscured, then a second predicted hand position of the target hand is determined based on the movement trajectory of the target hand before it is obscured; The second predicted display position of the eyeball in the eye image is determined based on the second predicted hand position, and the eyeball is controlled to move toward the second predicted display position.

8. The method according to claim 1, characterized in that, The specified gesture is any preset gesture from a preset gesture set. After the eyeball moves from the current display position to the target display position in the eye image, the following is also included: If the duration for which no hand making any preset gesture in the preset gesture set is continuously not recognized reaches a first preset duration, then the eyeball is controlled to remain at the first display position where the eyeball is when the duration reaches the first preset duration; If, after a second preset time has elapsed since the eyeball has been held at the first display position, no hand making any of the preset gestures in the preset gesture set has been identified, then the eyeball is controlled to move to the reference display position, or after the target face is determined, the eyeball is controlled to move to the second display position corresponding to the location of the target face.

9. The method according to any one of claims 1 to 4 or 6 to 8, characterized in that, The companion robot also includes a camera, and before controlling the eyeballs to move from the current display position to the target display position in the eye image, the method further includes: The hand position information of the target hand is obtained, including the two-dimensional coordinates of the target hand in the target image or the three-dimensional coordinates of the target hand in the camera coordinate system, and the target image is an image captured by the camera that recognizes the specified gesture; Based on the camera calibration parameters and the hand position information, the orientation of the target hand in the robot coordinate system is determined; The target display position is determined based on the direction of the target's hand.

10. The method according to claim 9, characterized in that, Determining the target display position based on the target hand direction includes: Determine the current gaze direction corresponding to the current display position in the robot coordinate system; Calculate the horizontal and / or vertical angle difference between the current gaze direction and the direction of the target hand; The horizontal angle difference is converted into the horizontal displacement of the eyeball in the eye image, and / or the vertical angle difference is converted into the vertical displacement of the eyeball in the eye image; The target display position is determined based on the current display position and the horizontal displacement and / or the vertical displacement.

11. The method according to any one of claims 1 to 8, characterized in that, The control of the eyeball to move from the current display position to the target display position in the eye image includes: Interpolate the current display position and the target display position to determine at least one intermediate display position between the current display position and the target display position; Control the eyeball to move from the current display position to the target display position via at least one intermediate display position.

12. The method according to any one of claims 1 to 4 or 6 to 8, characterized in that, Before controlling the eyeball to move from the current display position to the target display position in the eye image, the method further includes: If multiple hands making the specified gesture are identified, each of the multiple hands is identified as a candidate hand; Obtain target selection information for each of the candidate hands, and select one from the multiple candidate hands as the target hand based on the target selection information; The target selection information includes at least one of the following: the position information of the candidate hand, the gesture recognition confidence of the candidate hand, the duration of the candidate hand making the specified gesture, and the judgment result of whether the candidate hand belongs to the current interactive object.

13. The method according to claim 12, characterized in that, The companion robot also includes a camera, and the step of selecting one hand from a plurality of candidate hands as the target hand based on the target selection information includes: Acquire target images including multiple candidate hands captured by the camera; Based on the position information of each candidate hand, a first distance between the image reference point of each candidate hand in the target image and the image center of the target image, and a second distance between each candidate hand and the companion robot are determined. A first selection score is determined based on the first distance, and a second selection score is determined based on the second distance, wherein the first selection score increases as the first distance decreases, and the second selection score increases as the second distance decreases; The first selection score and the second selection score of the same candidate hand are weighted and summed to obtain the target selection score of each candidate hand; The candidate hand with the highest target selection score is determined as the target hand.

14. A companion robot, comprising an eye screen, a memory, a processor, and a computer program stored in the memory and executable on the processor, the eye screen being used to display an eye image including the eyeball; characterized in that, When the processor executes the computer program, it causes the companion robot to implement the method as described in any one of claims 1 to 13.

Images