Humanoid robot active following method and device, humanoid robot, electronic equipment and storage medium

Abstract

The invention provides a humanoid robot active following method and device, a humanoid robot, electronic equipment and a storage medium, and relates to the technical field of robot active following. Determining three-dimensional position information of a center point of a sensing target corresponding to a sensing target frame under a camera coordinate system, and further respectively calculating a first angle deviation, a second angle deviation and a distance deviation; the head of the humanoid robot is controlled to rotate through the first angle deviation, and the humanoid robot body is controlled to rotate and translate through the second angle deviation and the distance deviation. According to the method, the posture of the head of the humanoid robot can be adjusted, the anthropomorphic degree in the target tracking process is effectively improved, the robustness, naturalness and smoothness of target tracking are improved, the possibility of target missing is reduced, and the landing efficiency of the target tracking function of the humanoid robot is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of robot active following technology, and in particular to a method, apparatus, humanoid robot, electronic device and storage medium for humanoid robot active following. Background Technology

[0002] Active following technology for robots refers to the process by which robots perceive dynamic target information in real time through sensor data such as LiDAR and 3D cameras, and combine it with planning and control algorithms to achieve continuous tracking of the target. It is mainly used in scenarios such as home care, carrying large goods, following suitcases, guiding in public places, and medical collaboration.

[0003] Existing robot active following solutions typically fuse LiDAR point cloud data collected by sensors with image data collected by depth cameras, and use the fused data to track the target, output the tracking target bounding box, and simultaneously output the robot's linear velocity and angular velocity in real time to achieve real-time continuous following.

[0004] However, in the above solutions, the depth camera is usually fixed to the robot's chassis, which results in a low degree of anthropomorphism and makes it easy to lose track of the target due to occlusion. Summary of the Invention

[0005] This invention provides a method, apparatus, humanoid robot, electronic device, and storage medium for active following of a humanoid robot, in order to overcome the deficiencies existing in related technologies.

[0006] This invention provides a method for a humanoid robot to actively follow, comprising: The position and depth information of the sensing target box closest to the 3D camera in the target image acquired by the 3D camera are acquired in real time. The 3D camera is fixed to the head of the humanoid robot. Based on the position and depth information of the sensing target bounding box, the three-dimensional position information of the center point of the sensing target corresponding to the sensing target bounding box in the camera coordinate system is determined. Based on the three-dimensional position information, calculate the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the orientation of the humanoid robot body in the world coordinate system. Based on the first angle deviation, the humanoid robot's head is controlled to rotate, and based on the second angle deviation and the distance deviation, the humanoid robot's body is controlled to rotate and translate.

[0007] The humanoid robot active following method provided by the present invention further includes: The position range of the perceived target bounding box in the target image is recorded in real time; the position range includes the range of a first side or a second side; the first side and the second side are the two sides of the target image in the horizontal direction; If the perceived target box disappears from the target image, the humanoid robot's head is controlled to rotate based on the position range of the perceived target box in the target image at the previous moment, until the perceived target box reappears in the target image.

[0008] According to the present invention, a method for active following of a humanoid robot, wherein the humanoid robot's head is controlled to rotate based on the position range of the perceived target box in the target image at a previous moment until the perceived target box appears in the target image, includes: If the perceived target box is within the range of the first side of the target image at the previous moment, then control the humanoid robot's head to rotate towards the second side until the perceived target box appears in the target image; If the perceived target box is within the range of the second side of the target image at the previous moment, then the humanoid robot's head is controlled to rotate in the direction of the first side until the perceived target box appears in the target image.

[0009] According to the present invention, a humanoid robot actively follows another robot, wherein the first angular deviation includes pitch angle and yaw angle. The pitch angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the yz plane and the z-axis. The yaw angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the xz plane and the z-axis. The second angular deviation is the angle between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body, and the orientation of the humanoid robot body. The distance deviation is the distance between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body.

[0010] According to the present invention, a method for active following of a humanoid robot, wherein controlling the humanoid robot body to rotate and translate based on the second angle deviation and the distance deviation includes: Based on the second angular deviation, a proportional control algorithm is applied to control the angular velocity of the humanoid robot body, causing the humanoid robot body to rotate. Based on the distance deviation, a proportional control algorithm is applied to control the linear velocity of the humanoid robot body, causing the humanoid robot body to translate.

[0011] According to the present invention, the rate of change of angular velocity of the humanoid robot head and the humanoid robot body, as well as the rate of change of linear velocity of the humanoid robot body, are all within the corresponding preset rate of change ranges. And / or, The angular velocities of the humanoid robot's head and body, as well as the linear velocity of the humanoid robot's body, are all within their respective preset amplitude ranges.

[0012] According to the present invention, a method for active following of a humanoid robot, wherein the real-time acquisition of the position and depth information of the nearest perception target bounding box to the 3D camera in the target image acquired by the 3D camera specifically includes: Receive user voice and parse the user voice to determine the user's intent; If the user's intention is for the robot to follow, then calculate the area ratio of the perceived target box in the target image; If the area ratio is greater than or equal to the ratio threshold, the position and depth information of the perceived target box are obtained in real time.

[0013] The humanoid robot active following method provided by the present invention further includes: The status information of the humanoid robot is broadcast to the user.

[0014] The present invention also provides a humanoid robot active following device, comprising: The information acquisition module is used to acquire in real time the position and depth information of the sensing target box that is closest to the 3D camera in the target image captured by the 3D camera, wherein the 3D camera is fixed to the head of the humanoid robot. The position determination module is used to determine the three-dimensional position information of the center point of the sensing target corresponding to the sensing target box in the camera coordinate system based on the position information and depth information of the sensing target box. The deviation calculation module is used to calculate, based on the three-dimensional position information, the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the humanoid robot body in the world coordinate system. The follow control module is used to control the humanoid robot's head to rotate based on the first angle deviation, and to control the humanoid robot's body to rotate and translate based on the second angle deviation and the distance deviation.

[0015] The present invention also provides a humanoid robot, comprising: a humanoid robot head, a humanoid robot body, a 3D camera, and a decision-making module; The humanoid robot head is mounted on the humanoid robot body, and the 3D camera is fixed to the humanoid robot head; The 3D camera is connected to the decision module; The decision-making module is used to execute the above-described humanoid robot active following method.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the humanoid robot active following method as described above.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the humanoid robot active following method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the humanoid robot active following method as described above.

[0019] This invention provides a method, apparatus, humanoid robot, electronic device, and storage medium for active tracking of a humanoid robot. The method involves fixing a 3D camera to the head of the humanoid robot and using the target image acquired by the 3D camera to determine the 3D position information of the center point of the target corresponding to the target frame in the camera coordinate system. Then, it calculates a first angular deviation, a second angular deviation, and a distance deviation. By controlling the rotation of the humanoid robot's head through the first angular deviation, the head can rotate in real-time to follow the target. By controlling the rotation and translation of the humanoid robot body through the second angular deviation and the distance deviation, the robot body can rotate and translate in real-time to follow the target. This method can adjust the posture of the humanoid robot's head, effectively improving the anthropomorphism during target tracking, enhancing the robustness, naturalness, and smoothness of target tracking, reducing the possibility of losing the target, and effectively improving the efficiency of the humanoid robot's target tracking function. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the humanoid robot active following method provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the target image in the humanoid robot active following method provided by the present invention.

[0023] Figure 3 This is a structural schematic diagram of the wheeled humanoid robot provided by the present invention.

[0024] Figure 4 This is a structural schematic diagram of the bipedal humanoid robot provided by the present invention.

[0025] Figure 5 This is a schematic diagram showing the three-dimensional position information of the center point of the target perceived in the camera coordinate system and the first angle deviation in the humanoid robot active following method provided by the present invention.

[0026] Figure 6 This is a schematic diagram showing the three-dimensional position information of the center point of the perceived target and the first angle deviation in the new coordinate system after the camera coordinate system has been rotated and transformed in the humanoid robot active following method provided by the present invention.

[0027] Figure 7 This is a schematic diagram showing the second angular deviation and distance deviation of the center point of the perceived target relative to the orientation of the humanoid robot body in the world coordinate system in the humanoid robot active following method provided by the present invention.

[0028] Figure 8 This is a schematic diagram of the structure of the humanoid robot active following device provided by the present invention.

[0029] Figure 9 This is a structural schematic diagram of the humanoid robot provided by the present invention.

[0030] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] In existing robot active following solutions, the depth camera is usually fixed to the robot's chassis, resulting in low anthropomorphism and a tendency to lose track of the target due to occlusion. Therefore, this invention provides a humanoid robot active following method.

[0033] Figure 1 This is a flowchart illustrating a humanoid robot's active following method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: S1, real-time acquisition of the position and depth information of the perception target box closest to the 3D camera in the target image captured by the 3D camera, wherein the 3D camera is fixed to the head of the humanoid robot; S2, Based on the position information and depth information of the sensing target box, determine the three-dimensional position information of the center point of the sensing target corresponding to the sensing target box in the camera coordinate system; S3, based on the three-dimensional position information, calculate the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the orientation of the humanoid robot body in the world coordinate system; S4, based on the first angle deviation, control the humanoid robot head to rotate, and based on the second angle deviation and the distance deviation, control the humanoid robot body to rotate and translate.

[0034] Specifically, the humanoid robot active following method provided in this embodiment of the invention is executed by a humanoid robot active following device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0035] First, step S1 is executed, where a 3D camera is used to acquire target images in real time and identify them to obtain the position and depth information of the nearest perception target bounding box to the 3D camera. Here, the 3D camera can be an RGB-D camera, capable of simultaneously acquiring color and depth information. The 3D camera can integrate an image recognition algorithm to identify the target image. Alternatively, the 3D camera can send the target image to a humanoid robot active following device, which then uses its built-in image recognition algorithm to identify the target image.

[0036] The image recognition algorithm can be the YOLOv8+SAM2 model, or other image recognition algorithms; no specific limitations are made here.

[0037] The target image can be an image containing at least one or more targets, which can be people or other objects that the humanoid robot needs to follow. For example... Figure 2 As shown, the target image includes three targets. The largest target is the closest perceptual target to the 3D camera, and the four corner points of the perceptual target bounding box are divided into P0-P3.

[0038] Next, the perceived target needs to be tracked. To ensure stable tracking of the perceived target in complex situations such as multiple targets, the target tracking algorithm used here can be the Cutie algorithm to achieve target tracking through real-time video segmentation.

[0039] The positional information of the target bounding box in the target image can be represented by the positional information of the four corner points of the target bounding box, which can be represented by the number of pixels. The depth information of the target bounding box in the target image can be the distance between the target and the 3D camera, in meters.

[0040] In this embodiment of the invention, a 3D camera can be fixed to the head of the humanoid robot. The humanoid robot can be a wheeled humanoid robot or a bipedal humanoid robot. The main difference between wheeled humanoid robots and bipedal humanoid robots lies in their mode of locomotion. If a humanoid robot moves using a wheeled chassis and has two arms to perform different tasks, it is a wheeled humanoid robot; while if the robot moves using humanoid legs, it is a bipedal humanoid robot.

[0041] Wheel-arm humanoid robots, such as Figure 3 As shown, the robot includes a humanoid robot head 11 and a humanoid robot body. The humanoid robot body includes a humanoid robot arm 12 and a mobile chassis 13. The humanoid robot head 11 contains a control motor 111. A 3D camera 14 can be fixed inside the humanoid robot head 11.

[0042] Bipedal humanoid robots, such as Figure 4 As shown, the robot includes a humanoid robot head 21 and a humanoid robot body. The humanoid robot body includes a humanoid robot arm 22 and a bipedal actuator 23. The humanoid robot head 21 contains a control motor 211. A 3D camera 24 can be fixed inside the humanoid robot head 21.

[0043] Next, step S2 is executed, using the position and depth information of the sensing target bounding box to determine the 3D position information of the center point of the sensing target corresponding to the sensing target bounding box in the camera coordinate system. The camera coordinate system can be a 3D right-handed Cartesian coordinate system constructed with the optical center of the camera as the origin, the left and right directions of the 3D camera as the x-axis, and the optical axis direction of the 3D camera as the z-axis.

[0044] By utilizing the positional information of the perceived target bounding box in the target image, the positional information of the center point of the perceived target bounding box can be determined. For example, the positional information of the center point of the perceived target bounding box can be calculated based on the positional information of any diagonal point of the perceived target bounding box, i.e.: ; in, Let P0 be the x-coordinate of the target image. Let P2 be the x-coordinate of the target image. Let P0 be the ordinate of the target image. Let P2 be the ordinate of the target image. Let x be the x-coordinate of the center point of the perceived target bounding box in the target image. The ordinate is the center point of the perceived target bounding box in the target image.

[0045] Using the intrinsic parameters of the 3D camera, combined with the position information of the center point of the perceived target bounding box in the target image and the depth information of the perceived target bounding box in the target image, the 3D position information of the center point of the perceived target corresponding to the perceived target bounding box in the camera coordinate system is calculated, that is: ; Where (X,Y,Z) is the center point of the perceived target in the camera coordinate system. Pxyz Three-dimensional position information, Z It is the depth information of the perceived target bounding box in the target image. f x and f y These are the focal lengths of the 3D camera in the x and y directions, respectively, in the camera coordinate system, corresponding to the intrinsic parameters of the 3D camera. K 11 and K 22 , c x and c yThese are the x and y coordinates of the principal point of the 3D camera in the camera coordinate system, corresponding to the intrinsic parameters of the 3D camera. K 13 and K 23 .

[0046] Subsequently, step S3 is executed, using the three-dimensional position information of the center point of the perceived target in the camera coordinate system to calculate the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the orientation of the humanoid robot body in the world coordinate system.

[0047] like Figure 5 As shown, the first angular deviation includes the pitch angle ∆pitch and the yaw angle ∆yaw, where the pitch angle ∆pitch is the center point of the perceived target in the camera coordinate system. Pxyz The angle between the projection O1-Pyz of the line connecting the optical center O1 of the 3D camera onto the yz plane and the z-axis, where Pyz is the center point of the perceived target. Pxyz The projection point on the yz plane. Given that the coordinates of Pyz are (Y, Z), then: .

[0048] The yaw angle ∆aw is the center point of the perceived target in the camera coordinate system. Pxyz The angle between the projection O1-Pxz of the line connecting the optical center O1 of the 3D camera onto the xz plane and the z-axis, where Pxz is the center point of the perceived target. Pxyz The projection point of Pxz onto the xz plane. The coordinates of Pxz are (X, Z), then: .

[0049] Here, the world coordinate system can be a three-dimensional right-handed Cartesian coordinate system constructed with the initial position O2 of the humanoid robot as the origin, the ground as the xy plane, and the vertical direction as the z-direction. By combining the Unified Robot Description Format (URDF) of the humanoid robot and performing a TF transformation, the three-dimensional position information of the perceived target's center point in the camera coordinate system can be converted to the world coordinate system. Projecting the center point of the perceived target in the world coordinate system onto the xy plane yields the second angular deviation.

[0050] In the process of converting the 3D position information of the center point of the perceived target in the camera coordinate system to that in the world coordinate system, the camera coordinate system can first be rotated and transformed, such as... Figure 6As shown, the z-axis of the camera coordinate system becomes the x-axis of the new coordinate system after the rotation transformation, the x-axis of the camera coordinate system becomes the negative direction of the y-axis of the new coordinate system after the rotation transformation, and the y-axis of the camera coordinate system becomes the negative direction of the z-axis of the new coordinate system after the rotation transformation. In the new coordinate system after the rotation transformation, we have: .

[0051] Therefore, the second angular deviation can be the angle between the line connecting the projection point Pxy of the center point of the perceived target in the xy plane in the world coordinate system and the center point o3 of the humanoid robot body, and the orientation of the humanoid robot body. For example... Figure 7 As shown, ∆θ is the second angular deviation.

[0052] The distance deviation is the distance between the projection point Pxy of the center point of the perceived target in the xy plane in the world coordinate system and the center point o3 of the humanoid robot body, i.e. .

[0053] Finally, step S4 is executed. By utilizing the first angular deviation, the rotation of the motor installed inside the humanoid robot's head is controlled, so that the first angular deviation approaches 0. This ensures that the tracked target is always in the center of the 3D camera. From the perspective of the humanoid robot from the outside, it appears that the humanoid robot's head is always turning its neck to follow the moving target, ensuring that the center of the 3D camera is always facing the target, thus achieving a very human-like target tracking effect.

[0054] By utilizing the second angle deviation, the humanoid robot body is controlled to rotate, making the second angle deviation approach 0. By utilizing the distance deviation, the humanoid robot body is controlled to translate, making the distance deviation approach 0. In this way, stable tracking of the target can be achieved.

[0055] Since the target tracking in this embodiment of the invention ultimately calculates the angle deviation and distance deviation, and the control commands for both wheeled humanoid robots and bipedal humanoid robots are rotation commands and translation commands, this embodiment of the invention can support both types of humanoid robots.

[0056] In addition, since the humanoid robot head follows the target movement in order to achieve the target tracking process in the embodiments of the present invention, it is suitable for humanoid robots whose heads contain motors, but not for humanoid robots whose heads do not contain motors.

[0057] The humanoid robot active following method provided in this embodiment of the invention involves fixing a 3D camera to the head of the humanoid robot and determining the 3D position information of the center point of the target corresponding to the target frame in the camera coordinate system using the target image acquired by the 3D camera. Then, a first angle deviation, a second angle deviation, and a distance deviation are calculated. The first angle deviation is used to control the rotation of the humanoid robot's head, allowing the head to rotate in real-time to follow the target. The second angle deviation and the distance deviation are used to control the rotation and translation of the humanoid robot's body, enabling the body to rotate and translate in real-time to follow the target. This method can adjust the posture of the humanoid robot's head, effectively improving the anthropomorphism during target tracking, enhancing the robustness, naturalness, and smoothness of target following, reducing the possibility of losing the target, and effectively improving the implementation efficiency of the humanoid robot's target following function.

[0058] Considering the anomaly of the target disappearing from the field of view, traditional methods rely on historical trajectory prediction, which is prone to cumulative errors and results in weak anomaly resistance. Based on the above embodiments, the humanoid robot active following method provided in this invention further includes: The position range of the perceived target bounding box in the target image is recorded in real time; the position range includes the range of a first side or a second side; the first side and the second side are the two sides of the target image in the horizontal direction; If the perceived target box disappears from the target image, the humanoid robot's head is controlled to rotate based on the position range of the perceived target box in the target image at the previous moment, until the perceived target box reappears in the target image.

[0059] Specifically, in this embodiment of the invention, the position range of the perceived target bounding box in the target image can be recorded in real time. This position range may include the range of a first side or a second side; the first side and the second side are respectively the two sides of the target image in the horizontal direction, for example, the first side is the left side and the second side is the right side.

[0060] If the target bounding box disappears from the target image, the humanoid robot's head can be rotated using the previous position range of the target bounding box in the target image until the target bounding box reappears in the target image. This process is the target search process.

[0061] If the two corner points of the perceived target bounding box on the second side are located at a preset proportion in the target image on the first side, then the perceived target bounding box is considered to be within the range of the first side. If the two corner points of the perceived target bounding box on the first side are located at a preset proportion in the target image on the second side, then the perceived target bounding box is considered to be within the range of the second side. The preset proportion can be set as needed, for example, it could be 1 / 3.

[0062] If the previously perceived target bounding box's position in the target image falls within the range of the first side, the humanoid robot's head can be controlled to rotate towards the first side at a fixed angular velocity until the target bounding box appears in the target image. If the previously perceived target bounding box's position in the target image falls within the range of the second side, the humanoid robot's head can be controlled to rotate towards the second side at a fixed angular velocity until the target bounding box appears in the target image.

[0063] Here, the fixed angular velocity can be set as needed, for example, it can be 0.3 rad / s or other values. During the rotation of the humanoid robot's head, its linear velocity is 0.

[0064] In this embodiment of the invention, by recording the relative positional relationship between the perceived target and the 3D camera before the perceived target disappears, a search control strategy is implemented after the perceived target disappears. The process from the disappearance of the perceived target to the re-finding of the perceived target is smoothly transitioned without interruption, which improves the robustness and smoothness of target following and solves the pain points that will inevitably be encountered in the target following process, such as the disappearance of the perceived target and perception anomalies.

[0065] Based on the above embodiments, controlling the humanoid robot body to rotate and translate based on the second angle deviation and the distance deviation includes: Based on the second angular deviation, a proportional control algorithm is applied to control the angular velocity of the humanoid robot body, causing the humanoid robot body to rotate. Based on the distance deviation, a proportional control algorithm is applied to control the linear velocity of the humanoid robot body, causing the humanoid robot body to translate.

[0066] Specifically, in engineering practice, It cannot directly approach 0; instead, it needs to stop within a small range to ensure that the humanoid robot does not collide with the target it is sensing, and when When the size is relatively large, it is also necessary to ensure that the turn is made first. The humanoid robot should only move when the temperature is low. Otherwise, it will cause abnormal movement of the humanoid robot, and the control law can be implemented using a proportional control algorithm.

[0067] Therefore, the formulas for calculating the linear velocity and angular velocity of the humanoid robot tracking and sensing the target are as follows: ; in, d safe It refers to the safe distance from the target being sensed. K v and K wThese are the proportional control coefficients for the linear velocity and angular velocity of the humanoid robot, respectively. The threshold angle θ0 can be set as needed, for example, it can be set to 1 / 2π, or it can be adjusted according to the actual situation.

[0068] Based on the above embodiments, the rate of change of angular velocity of the humanoid robot head and the humanoid robot body, as well as the rate of change of linear velocity of the humanoid robot body, are all within the corresponding preset rate of change ranges. And / or, The angular velocities of the humanoid robot's head and body, as well as the linear velocity of the humanoid robot's body, are all within their respective preset amplitude ranges.

[0069] Specifically, in order to achieve seamless switching between the target tracking and search processes, it is necessary to ensure that the linear velocity and angular velocity issued to the humanoid robot do not have significant abrupt changes. Therefore, this can be achieved by limiting the range of changes in linear velocity and angular velocity. Ultimately, the target tracking and search processes are imperceptible to the humanoid robot, thereby improving the robustness and consistency of the humanoid robot's active following method.

[0070] Based on this, a preset rate of change range can be set. The preset rate of change range may include a first angular velocity rate of change range, a second angular velocity rate of change range, and a linear velocity rate of change range. The angular velocity rate of change of the humanoid robot head is limited to the first angular velocity rate of change range, the angular velocity rate of change of the humanoid robot body is limited to the second angular velocity rate of change range, and the linear velocity rate of change of the humanoid robot body is limited to the linear velocity rate of change range.

[0071] If the rate of change of the angular velocity of the humanoid robot's head is higher than the upper limit of the first angular velocity change rate range, then the rate of change of the angular velocity of the humanoid robot's head is set to the upper limit of the first angular velocity change rate range. If the rate of change of the angular velocity of the humanoid robot's head is lower than the lower limit of the first angular velocity change rate range, then the rate of change of the angular velocity of the humanoid robot's head is set to the lower limit of the first angular velocity change rate range.

[0072] If the rate of change of the humanoid robot's angular velocity is higher than the upper limit of the second angular velocity change range, then the rate of change of the humanoid robot's angular velocity is set to the upper limit of the second angular velocity change range. If the rate of change of the humanoid robot's angular velocity is lower than the lower limit of the second angular velocity change range, then the rate of change of the humanoid robot's angular velocity is set to the lower limit of the second angular velocity change range.

[0073] If the rate of change of the linear velocity of the humanoid robot body is higher than the upper limit of the range of linear velocity change rates, then the rate of change of the linear velocity of the humanoid robot body is set to the upper limit of the range of linear velocity change rates. If the rate of change of the linear velocity of the humanoid robot body is lower than the lower limit of the range of linear velocity change rates, then the rate of change of the linear velocity of the humanoid robot body is set to the lower limit of the range of linear velocity change rates.

[0074] Similarly, a preset amplitude range can be set, which may include a first angular velocity amplitude range, a second angular velocity amplitude range, and a linear velocity amplitude range. The angular velocity amplitude of the humanoid robot's head is limited to the first angular velocity amplitude range, the angular velocity amplitude of the humanoid robot's body is limited to the second angular velocity amplitude range, and the linear velocity amplitude of the humanoid robot's body is limited to the linear velocity amplitude range.

[0075] If the angular velocity amplitude of the humanoid robot's head is higher than the upper limit of the first angular velocity amplitude range, then the angular velocity amplitude of the humanoid robot's head is set to the upper limit of the first angular velocity amplitude range. If the angular velocity amplitude of the humanoid robot's head is lower than the lower limit of the first angular velocity amplitude range, then the angular velocity amplitude of the humanoid robot's head is set to the lower limit of the first angular velocity amplitude range.

[0076] If the angular velocity amplitude of the humanoid robot body is higher than the upper limit of the second angular velocity amplitude range, then the angular velocity amplitude of the humanoid robot body is set to the upper limit of the second angular velocity amplitude range. If the angular velocity amplitude of the humanoid robot body is lower than the lower limit of the second angular velocity amplitude range, then the angular velocity amplitude of the humanoid robot body is set to the lower limit of the second angular velocity amplitude range.

[0077] If the linear velocity amplitude of the humanoid robot body is higher than the upper limit of the linear velocity amplitude range, then the linear velocity amplitude of the humanoid robot body is set to the upper limit of the linear velocity amplitude range. If the linear velocity amplitude of the humanoid robot body is lower than the lower limit of the linear velocity amplitude range, then the linear velocity amplitude of the humanoid robot body is set to the lower limit of the linear velocity amplitude range.

[0078] Based on the above embodiments, the real-time acquisition of the position and depth information of the perception target bounding box closest to the 3D camera in the target image acquired by the 3D camera specifically includes: Receive user voice and parse the user voice to determine the user's intent; If the user's intention is for the robot to follow, then calculate the area ratio of the perceived target box in the target image; If the area ratio is greater than or equal to the ratio threshold, the position and depth information of the perceived target box are obtained in real time.

[0079] Specifically, in order to improve the human-computer interaction of the humanoid robot, this embodiment of the invention also provides a voice wake-up function for the humanoid robot. When a user speaks in front of the humanoid robot, the robot can receive the user's voice, analyze the user's voice, and determine the user's intention.

[0080] If the user's intention is for the robot to follow, for example, if the user says "Please follow me", then the area ratio of the perceived target box in the target image can be calculated. First, the area of ​​the perceived target box can be calculated based on the position information of the four corner points of the perceived target box. Then, the ratio of the area of ​​the perceived target box to the area of ​​the target image can be calculated to obtain the area ratio of the perceived target box.

[0081] Here, the relationship between the area ratio of the perceived target box and the ratio threshold is determined. If the area ratio of the perceived target box is greater than or equal to the ratio threshold, the perceived target in the target image is considered to be the user. Subsequently, the position and depth information of the perceived target box are acquired in real time. If the area ratio of the perceived target box is less than the ratio threshold, it is considered that there is no user in the target image, and the process of tracking the perceived target is not performed.

[0082] The percentage threshold can be set as needed, and no specific limit is specified here.

[0083] When the user says "Stop following", the humanoid robot can be controlled to stop following.

[0084] In this embodiment of the invention, by sensing the relationship between the area ratio of the target box and the ratio threshold, it is determined whether to execute the following process, which can avoid accidental triggering by user voice and improve the accuracy of target following.

[0085] Based on the above embodiments, the humanoid robot active following method provided in this embodiment of the invention further includes: The status information of the humanoid robot is broadcast to the user.

[0086] Specifically, to improve the human-computer interaction of the humanoid robot, this embodiment of the invention also provides a status broadcast function for the humanoid robot. In case of abnormal situations such as occlusion of the 3D camera, the humanoid robot can continue to broadcast the status "The camera is blocked, please check" during the target following process. At this time, the linear velocity and angular velocity are both 0 to ensure safety. In addition, during the target search process, status prompts such as "Look to the left" and "Look to the right" are broadcast. When the target is detected again in the search state, voice prompts such as "Found you" can be broadcast, which makes it easier for users to understand the current status of the humanoid robot. This improves the interactivity with humans during the following process, enhances the efficiency of human-computer collaboration, and thus improves the success rate of following and enhances the user experience.

[0087] like Figure 8As shown, based on the above embodiments, this embodiment of the invention provides a humanoid robot active following device, comprising: Information acquisition module 81 is used to acquire in real time the position information and depth information of the perception target box closest to the three-dimensional camera in the target image captured by the three-dimensional camera, wherein the three-dimensional camera is fixed to the head of the humanoid robot. The position determination module 82 is used to determine the three-dimensional position information of the center point of the sensing target corresponding to the sensing target box in the camera coordinate system based on the position information and depth information of the sensing target box. The deviation calculation module 83 is used to calculate, based on the three-dimensional position information, the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the humanoid robot body in the world coordinate system. The follow control module 84 is used to control the humanoid robot head to rotate based on the first angle deviation, and to control the humanoid robot body to rotate and translate based on the second angle deviation and the distance deviation.

[0088] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention further includes a target search module, used for: The position range of the perceived target bounding box in the target image is recorded in real time; the position range includes the range of a first side or a second side; the first side and the second side are the two sides of the target image in the horizontal direction; If the perceived target box disappears from the target image, the humanoid robot's head is controlled to rotate based on the position range of the perceived target box in the target image at the previous moment, until the perceived target box reappears in the target image.

[0089] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention, wherein the target search module is specifically used for: If the perceived target box is within the range of the first side of the target image at the previous moment, then control the humanoid robot's head to rotate towards the second side until the perceived target box appears in the target image; If the perceived target box is within the range of the second side of the target image at the previous moment, then the humanoid robot's head is controlled to rotate in the direction of the first side until the perceived target box appears in the target image.

[0090] Based on the above embodiments, the humanoid robot active following device provided in the embodiments of the present invention includes a pitch angle and a yaw angle as the first angular deviation; The pitch angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the yz plane and the z-axis. The yaw angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the xz plane and the z-axis. The second angular deviation is the angle between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body, and the orientation of the humanoid robot body. The distance deviation is the distance between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body.

[0091] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention, wherein the following control module is specifically used for: Based on the second angular deviation, a proportional control algorithm is applied to control the angular velocity of the humanoid robot body, causing the humanoid robot body to rotate. Based on the distance deviation, a proportional control algorithm is applied to control the linear velocity of the humanoid robot body, causing the humanoid robot body to translate.

[0092] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention has the angular velocity change rate of the humanoid robot head and the humanoid robot body and the linear velocity change rate of the humanoid robot body all within the corresponding preset change rate range. And / or, The angular velocities of the humanoid robot's head and body, as well as the linear velocity of the humanoid robot's body, are all within their respective preset amplitude ranges.

[0093] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention, wherein the information acquisition module is specifically used for: Receive user voice and parse the user voice to determine the user's intent; If the user's intention is for the robot to follow, then calculate the area ratio of the perceived target box in the target image; If the area ratio is greater than or equal to the ratio threshold, the position and depth information of the perceived target box are obtained in real time.

[0094] Based on the above embodiments, the humanoid robot active following device provided in this embodiment of the invention further includes a voice broadcast module, used for: The status information of the humanoid robot is broadcast to the user.

[0095] Specifically, the functions of each module in the humanoid robot active following device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0096] like Figure 9 As shown, based on the above embodiments, this embodiment of the invention also provides a humanoid robot, including: a humanoid robot head 91, a humanoid robot body 92, a three-dimensional camera 93, and a decision module 94; The humanoid robot head 91 is mounted on the humanoid robot body 92, and the three-dimensional camera 93 is fixed to the humanoid robot head 91; The 3D camera 93 is connected to the decision module 94; The decision module 94 is used to execute the humanoid robot active following method provided in the above embodiments.

[0097] The humanoid robot in this embodiment of the invention, by integrating a decision-making module and a 3D camera fixed on the head of the humanoid robot, can realize the tracking and search of perceived targets. By adjusting the posture of the humanoid robot's head, the anthropomorphism in the target tracking process is effectively improved, the robustness, naturalness and smoothness of target following are improved, the possibility of losing the target is reduced, and the implementation efficiency of the humanoid robot's target following function is effectively improved.

[0098] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the humanoid robot active following method provided in the above embodiments.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the humanoid robot active following method provided in the above embodiments.

[0101] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the humanoid robot active following method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for a humanoid robot to actively follow, characterized in that, include: The position and depth information of the sensing target box closest to the 3D camera in the target image acquired by the 3D camera are acquired in real time. The 3D camera is fixed to the head of the humanoid robot. Based on the position and depth information of the sensing target bounding box, the three-dimensional position information of the center point of the sensing target corresponding to the sensing target bounding box in the camera coordinate system is determined. Based on the three-dimensional position information, calculate the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the orientation of the humanoid robot body in the world coordinate system. Based on the first angle deviation, the humanoid robot's head is controlled to rotate, and based on the second angle deviation and the distance deviation, the humanoid robot's body is controlled to rotate and translate.

2. The humanoid robot active following method according to claim 1, characterized in that, Also includes: The position range of the perceived target bounding box in the target image is recorded in real time; the position range includes the range of a first side or a second side; the first side and the second side are the two sides of the target image in the horizontal direction; If the perceived target box disappears from the target image, the humanoid robot's head is controlled to rotate based on the position range of the perceived target box in the target image at the previous moment, until the perceived target box reappears in the target image.

3. The humanoid robot active following method according to claim 2, characterized in that, The step of controlling the humanoid robot's head to rotate until the perceived target box appears in the target image, based on the position range of the target box in the target image at the previous moment, includes: If the perceived target box is within the range of the first side of the target image at the previous moment, then control the humanoid robot's head to rotate towards the second side until the perceived target box appears in the target image; If the perceived target box is within the range of the second side of the target image at the previous moment, then the humanoid robot's head is controlled to rotate in the direction of the first side until the perceived target box appears in the target image.

4. The humanoid robot active following method according to claim 1, characterized in that, The first angular deviation includes pitch angle and yaw angle; The pitch angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the yz plane and the z-axis. The yaw angle is the angle between the projection of the line connecting the center point of the perceived target and the optical center of the 3D camera in the camera coordinate system onto the xz plane and the z-axis. The second angular deviation is the angle between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body, and the orientation of the humanoid robot body. The distance deviation is the distance between the projection point of the center point of the perceived target in the xy plane in the world coordinate system and the center point of the humanoid robot body.

5. The humanoid robot active following method according to claim 4, characterized in that, The step of controlling the humanoid robot body to rotate and translate based on the second angle deviation and the distance deviation includes: Based on the second angular deviation, a proportional control algorithm is applied to control the angular velocity of the humanoid robot body, causing the humanoid robot body to rotate. Based on the distance deviation, a proportional control algorithm is applied to control the linear velocity of the humanoid robot body, causing the humanoid robot body to translate.

6. The humanoid robot active following method according to any one of claims 1-5, characterized in that, The rate of change of angular velocity of the humanoid robot head and the humanoid robot body, as well as the rate of change of linear velocity of the humanoid robot body, are all within the corresponding preset rate of change ranges. And / or, The angular velocities of the humanoid robot's head and body, as well as the linear velocity of the humanoid robot's body, are all within their respective preset amplitude ranges.

7. The humanoid robot active following method according to any one of claims 1-5, characterized in that, The real-time acquisition of the position and depth information of the perception target bounding box closest to the 3D camera in the target image acquired by the 3D camera specifically includes: Receive user voice and parse the user voice to determine the user's intent; If the user's intention is for the robot to follow, then calculate the area ratio of the perceived target box in the target image; If the area ratio is greater than or equal to the ratio threshold, the position and depth information of the perceived target box are obtained in real time.

8. The humanoid robot active following method according to any one of claims 1-5, characterized in that, Also includes: The status information of the humanoid robot is broadcast to the user.

9. A humanoid robot active following device, characterized in that, include: The information acquisition module is used to acquire in real time the position and depth information of the sensing target box that is closest to the 3D camera in the target image captured by the 3D camera, wherein the 3D camera is fixed to the head of the humanoid robot. The position determination module is used to determine the three-dimensional position information of the center point of the sensing target corresponding to the sensing target box in the camera coordinate system based on the position information and depth information of the sensing target box. The deviation calculation module is used to calculate, based on the three-dimensional position information, the first angular deviation of the center point of the perceived target relative to the optical axis of the three-dimensional camera in the camera coordinate system, and the second angular deviation and distance deviation of the center point of the perceived target relative to the humanoid robot body in the world coordinate system. The follow control module is used to control the humanoid robot's head to rotate based on the first angle deviation, and to control the humanoid robot's body to rotate and translate based on the second angle deviation and the distance deviation.

10. A humanoid robot, characterized in that, include: Humanoid robot head, humanoid robot body, 3D camera, and decision-making module; The humanoid robot head is mounted on the humanoid robot body, and the 3D camera is fixed to the humanoid robot head; The 3D camera is connected to the decision module; The decision module is used to execute the humanoid robot active following method as described in any one of claims 1-8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the humanoid robot active following method as described in any one of claims 1-8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the humanoid robot active following method as described in any one of claims 1-8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the humanoid robot active following method as described in any one of claims 1-8.

Images