Robot control method, robot and electronic equipment

Abstract

The invention provides a robot control method, a robot and electronic equipment, and the method comprises the steps: determining a first target angle of a robot head in a horizontal direction and a second target angle of the robot head in a vertical direction through the deviation between the position of a target object in an image and the central position of the image; and controlling the robot head to rotate according to the first target angle and the second target angle. Wherein any one of the first target angle and the second target angle is used for representing the angle needing to be adjusted in the direction corresponding to the target angle when the sight line of the robot is adjusted to the position facing the target object under the condition that the sight line orientation of the robot is the central position of the image. According to the method, the deviation between the position of the target object in the image and the central position of the image can be quantified into the first target angle and the second target angle to which the head of the robot should be adjusted in the horizontal dimension and the vertical dimension, and the interaction quality is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a robot control method, a robot, and an electronic device. BACKGROUND

[0002] With the development of science and technology and the development of intelligence, robots are increasingly widely used in people's lives. One of the common scenarios is that a robot interacts with a user.

[0003] Currently, when a robot interacts with a user, the robot typically adjusts the angle of the robot head based on the center of an image collected by an image sensor arranged on the robot head and the distance between a target object included in the image, so that the target object coincides with the center of the image. However, when the position of the image sensor is not on the robot head, the robot is difficult to interact with the user. SUMMARY

[0004] In view of this, embodiments of the present application provide a robot control method, a robot, and a robot electronic device, which can improve the interaction quality of the robot.

[0005] In a first aspect, embodiments of the present application provide a robot control method, a robot is provided with an image sensor, the image sensor is arranged in a non-head region of the robot, comprising: determining a target object in an image collected by the image sensor, determining a first target angle of the robot head in a horizontal direction and a second target angle of the robot head in a vertical direction according to a deviation between a position of the target object in the image and a center position of the image, wherein any one of the first target angle and the second target angle is used to represent the angle required to adjust the line of sight of the robot to the position of the target object in the case that the line of sight of the robot is toward the center position of the image, and controlling the robot head to rotate according to the first target angle and the second target angle.

[0006] In a second aspect, embodiments of the present application provide a robot control apparatus, the robot being provided with an image sensor, the image sensor being arranged at a non-head region of the robot, the robot control apparatus comprising: a target object determining module configured to determine a target object in an image captured by the image sensor; an angle determining module configured to determine a first target angle of a head of the robot in a horizontal direction and a second target angle of the head of the robot in a vertical direction according to a deviation between a position of the target object in the image and a center position of the image, wherein any one of the first target angle and the second target angle is used to represent an angle required to be adjusted in a direction corresponding to the target angle when a line of sight of the robot is adjusted to be directed to the position of the target object in a case that the line of sight of the robot is directed to the center position of the image; and a control module configured to control the head of the robot to rotate according to the first target angle and the second target angle.

[0007] In a third aspect, embodiments of the present application provide a robot, comprising a robot head, an image sensor and a control module, the image sensor being arranged at a non-head region of the robot, the control module being configured to perform the robot control method of the first aspect.

[0008] In a fourth aspect, embodiments of the present application provide an electronic device, comprising: a processor; and a memory configured to store processor-executable instructions, wherein the processor is configured to perform the robot control method of the first aspect.

[0009] In a fifth aspect, embodiments of the present application provide a computer-readable storage medium, the storage medium storing a computer program, the computer program being configured to perform the robot control method of the first aspect.

[0010] In a sixth aspect, embodiments of the present application provide a computer program product, the computer program product comprising a computer program, the computer program being configured to cause a computer device to perform the robot control method of the first aspect when the computer program is executed by a processor of the computer device.

[0011] In a seventh aspect, embodiments of the present application provide a chip, comprising: a processor; and a memory configured to store processor-executable instructions, wherein the processor is configured to perform the robot control method of the first aspect.

[0012] The embodiment of the present application provides a robot control method, a robot and an electronic device, the deviation between the position of the target object in the image and the center position of the image is used to determine a first target angle of the robot head in the horizontal direction and a second target angle of the robot head in the vertical direction respectively, and the robot head is controlled to rotate according to the first target angle and the second target angle. Any one of the first target angle and the second target angle is used to represent the angle required to adjust the line of sight of the robot to the position of the target object in the direction corresponding to the target angle when the line of sight of the robot is directed to the center position of the image.

[0013] The present application can quantize the deviation between the position of the target object in the image and the center position of the image into the first target angle and the second target angle to which the robot head should be adjusted in the horizontal and vertical dimensions when the image sensor is arranged in the non-head region of the robot, and the interaction quality is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 Fig. 1 shows a system architecture schematic diagram of a robot control system provided by an exemplary embodiment of the present application.

[0015] Figure 2 Fig. 2 shows a flowchart of a robot control method provided by an exemplary embodiment of the present application.

[0016] Figure 3A Fig. 3 shows a structural schematic diagram of the maximum horizontal angle of the robot head provided by an exemplary embodiment of the present application.

[0017] Figure 3B Fig. 4 shows a structural schematic diagram of the maximum vertical angle of the robot head provided by an exemplary embodiment of the present application.

[0018] Figure 4 Fig. 5 shows a flowchart of a robot control method provided by an exemplary embodiment of the present application.

[0019] Figure 5 Fig. 6 shows a flowchart of a robot control method provided by an exemplary embodiment of the present application.

[0020] Figure 6 Fig. 7 shows a flowchart of a robot control method provided by an exemplary embodiment of the present application.

[0021] Figure 7 Fig. 8 shows a structural schematic diagram of a robot control device provided by an exemplary embodiment of the present application.

[0022] Figure 8 Fig. 9 shows a block diagram of an electronic device for executing a robot control method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0024] SUMMARY

[0025] With the continuous development of computer technology, robots are increasingly widely used in people's life, and interaction between a robot and a target object (such as human-robot interaction) has become a relatively common scenario.

[0026] Taking human-robot interaction as an example, a relatively common interaction method at present is achieved by relying on an image sensor arranged on the head of a robot, which assumes that the actual line-of-sight of the robot (i.e., the head direction) is always directed to the center position of an image collected by the image sensor. Thus, after the image sensor collects the image, the head of the robot can be controlled to rotate based on the difference between the center position of the image and the position of a target object in the image, so that the center position of the image and the position of the target object in the image coincide, thereby achieving the purpose of enabling the robot to interact with the target object (user).

[0027] However, under the influence of various factors, the image sensor cannot be arranged on the head of the robot, such as that the head region of a medical robot needs to integrate too many modules, so the image sensor cannot be arranged, the carrying robot needs to detect the height of a cargo stack and surrounding obstacles in real time, and if the image sensor is arranged on the head, its monitoring angle of view can be blocked by the cargo being carried by itself, and the like. When the position of the image sensor is a non-head region of the robot, the actual line-of-sight of the robot no longer points to the center of the field of view of the image sensor, that is, no longer points to the center position of the image collected by the image sensor. In this case, if the above-mentioned method of controlling the head of the robot to rotate based on the difference between the center position of the image and the position of the target object in the image is still used, the actual line-of-sight of the robot after rotation can still not be aligned with the target object, and the interaction quality is low.

[0028] In summary, for the scenario in which the image sensor is arranged on a non-head region of the robot, that is, the position of the collection range is fixed and does not change with the movement of the head of the robot, the current method of controlling the robot has the technical problem of low interaction quality.

[0029] The embodiment of the present application provides a robot control method, deviation between the position of a target object in an image and the central position of the image is used to determine a first target angle of a robot head in a horizontal direction and a second target angle of the robot head in a vertical direction respectively, and the robot head is controlled to rotate according to the first target angle and the second target angle. Any one of the first target angle and the second target angle is used to represent an angle required to be adjusted in a direction corresponding to the target angle when a line of sight of the robot is adjusted to be directed to the position of the target object under the condition that the line of sight of the robot is directed to the central position of the image.

[0030] The robot control method in the present application can quantize the deviation between the position of the target object in the image and the central position of the image into the first target angle and the second target angle to which the robot head should be adjusted in the horizontal and vertical dimensions, and control the robot head to rotate according to the first target angle and the second target angle, so that the robot head can be accurately controlled to rotate based on the determined target angle under the condition that the image sensor is arranged in a non-head region of the robot (that is, the collection range and position of the image sensor do not change with the rotation of the robot head), and the line of sight of the robot can be accurately aligned to the target object, thereby ensuring the interaction quality.

[0031] Exemplary system

[0032] Figure 1 Fig. 1 shows a system architecture schematic diagram of a robot control system provided by an exemplary embodiment of the present application, as shown in the figure, Figure 1 As shown in the figure, the robot control system 100 can include a robot 110 and a target object 120. The robot 110 can be a humanoid robot, and the head of the robot 110 can be rotated to realize observation and interaction with the surrounding environment. The target object 120 can be an object (such as furniture, tools, etc.), can also be a human (that is, a user), and can also be a part of a human (such as a face, a hand, etc.).

[0033] When the target object 120 is a user or a part of the user, the robot 110 can identify and track the target object 120 through the image sensor arranged therein, and realize interaction with the user, such as providing services, having a conversation, etc. When the target object 120 is an object, the robot 110 can perform detection, positioning, etc. on the object. The image sensor 111 is arranged in the robot 110, and the image sensor 111 is fixedly installed at a non-head position of the robot 110. The collection range and physical position of the image sensor 111 are independent of the rotation of the robot head, that is, do not change with the rotation of the robot head.

[0034] The robot 110 can acquire the image captured by the image sensor 111, and identify the target object in the image, and determine a first target angle of the robot head in a horizontal direction and a second target angle of the robot head in a vertical direction according to a deviation between a position of the target object in the image and a center position of the image. The first target angle is a theoretical angle value of the robot head in the horizontal direction required to make the line of sight of the robot head coincide with the position of the target object, assuming that the line of sight of the robot head is directed to the center position of the image. The second target angle is a theoretical angle value of the robot head in the vertical direction required to make the line of sight of the robot head coincide with the position of the target object, assuming that the line of sight of the robot head is directed to the center position of the image. That is, for each of the first target angle and the second target angle, the target angle is used to represent the angle required to adjust the line of sight of the robot to be directed to the position of the target object in the direction of the target angle, when the line of sight of the robot is directed to the center position of the image.

[0035] The robot 110 can control the robot head to rotate according to the determined first target angle and second target angle.

[0036] It should be understood that the above application scenario examples are only shown for the purpose of facilitating understanding of the spirit and principles of the present application, and embodiments of the present application are not limited thereto. On the contrary, embodiments of the present application can be applied to any scenario that can be applicable.

[0037] Exemplary method

[0038] Figure 2 A flowchart of a robot control method provided by an exemplary embodiment of the present application is shown. Figure 2 The method can be performed by the robot 110 in Figure 1 The robot 110 in Figure 2 As shown, the robot control method can include the following.

[0039] 210: Determine the target object in the image captured by the image sensor.

[0040] In an example, the image sensor is fixedly arranged at a non-head region of the robot, which can be a chest region or a waist region, and the specific position of the image sensor can be set as required, which is not limited in the present specification.

[0041] In an example, the robot has a pre-stored identification model, and the robot can input the image captured by the image sensor into the identification model as input to obtain the target object contained in the image sensor.

[0042] In an example, the output of the recognition model is a position of the bounding box of the target object in the image, which can be a rectangular box corresponding to a smallest rectangle containing the target object.

[0043] In an example, the output of the recognition model is a coordinate of the bounding box, which can be (x, y, w, h), where x and y can be coordinates of the top-left corner of the bounding box, and w and h can be width and height of the bounding box, respectively.

[0044] In an example, the output of the recognition model is a coordinate of the bounding box, which can be (x, y), where x and y can be coordinates of the center point of the bounding box.

[0045] In an example, the output of the recognition model is a coordinate of the bounding box, which can be (x1, y1), (x2, y2), (x3, y3), (x4, y4).

[0046] In an example, the robot can send the image sensor to another electronic device, and the other electronic device can identify the image and return the position of the target object to the robot after determining the target object in the image.

[0047] 220: determining a first target angle of the head of the robot in the horizontal direction and a second target angle of the head of the robot in the vertical direction according to the deviation of the position of the target object in the image and the center position of the image, wherein any one of the first target angle and the second target angle represents an angle needed to be adjusted in the direction corresponding to the target angle when the line of sight of the robot is adjusted to be directed to the position of the target object from the center position of the image.

[0048] In an example, the robot can determine the first target angle of the head of the robot in the horizontal direction according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and determine the second target angle of the head of the robot in the vertical direction according to the deviation of the position of the target object in the image and the center position of the image in the vertical direction.

[0049] In an example, the first target angle represents an angle needed to be adjusted in the horizontal direction (i.e., an angle needed to be adjusted for the heading angle) when the line of sight of the robot is adjusted to be directed to the position of the target object from the center position of the image. The second target angle represents an angle needed to be adjusted in the vertical direction (i.e., an angle needed to be adjusted for the pitch angle) when the line of sight of the robot is adjusted to be directed to the position of the target object from the center position of the image.

[0050] In an example, the robot can determine the relative positional relationship between the target object and the robot head in the three-dimensional space (e.g., the depth information between the target object and the robot head, the height information of the target object and the height information of the robot head, etc.) according to the deviation between the position of the target object in the image and the central position of the image, and then determine the target angle of the robot head according to the relative positional relationship.

[0051] It should be noted that the position of the target object in the image in the present specification is the position of the center of the target object in the image, which can be represented by the center of the detection frame of the target object, or can be determined based on the relative positional relationship between the preset center of the target object and the detection frame of the target object. How to determine the position of the center of the target object in the image and how to represent the center of the target object can be set as needed, and the present specification does not limit this.

[0052] 230: controlling the robot head to rotate according to the target angle.

[0053] In an example, the robot can determine the current horizontal angle of the robot head, and control the robot head to rotate in the horizontal direction according to the difference between the current horizontal angle and the first target angle, so as to adjust the heading angle of the robot head to the first target angle.

[0054] In an example, the robot can determine the current vertical angle of the robot head, and control the robot head to rotate in the vertical direction according to the difference between the current vertical angle and the second target angle, so as to adjust the pitch angle of the robot head to the second target angle.

[0055] In an example, the robot can predict the first specific angle and the second specific angle at which the robot head should rotate at the next time according to the determined first target angle and second target angle, and the position information of the target object in the historical image collected by the image sensor, and control the robot head to rotate based on the first specific angle and the second specific angle.

[0056] In an example, for any of the first target angle and the second target angle, the target angle is an angle to which the robot head needs to be adjusted, rather than an angle to which the robot head needs to be rotated in this time. The robot can determine a current angle of the robot head, and control the robot head to rotate according to a difference between the current angle and the target angle, so that the angle of the robot head is adjusted to be the same as the target angle, and the robot head stops rotating. Taking left rotation as negative and right rotation as positive as an example, assuming that the current horizontal angle of the robot head is 15 degrees, and the first target angle of the robot head is 30 degrees, the robot head needs to be adjusted by 30-15=15 degrees in this time, and the adjustment direction is right rotation. If the current horizontal angle of the robot head is 10 degrees, and the first target angle of the robot head is -30 degrees, the robot head needs to be adjusted by -30-15=-45 degrees in this time, and the adjustment direction is left rotation.

[0057] The robot control method provided in the embodiments of the present application can quantize the deviation between the position of the target object in the image and the center position of the image into the first target angle and the second target angle to which the robot head should be adjusted in the horizontal and vertical dimensions, and control the robot head to rotate according to the first target angle and the second target angle, so that the robot head can be accurately controlled to rotate based on the determined target angle even when the image sensor is arranged in a non-head region of the robot (i.e., the collection range and position of the image sensor do not change with the rotation of the robot head), and the line of sight of the robot can be accurately aligned to the target object, thereby ensuring the interaction quality.

[0058] The robot control method in the present application can quantize the deviation between the position of the target object in the image and the center position of the image into the first target angle and the second target angle to which the robot head should be adjusted in the horizontal and vertical dimensions, and control the robot head to rotate according to the first target angle and the second target angle, so that the robot head can be accurately controlled to rotate based on the determined target angle even when the image sensor is arranged in a non-head region of the robot (i.e., the collection range and position of the image sensor do not change with the rotation of the robot head), and the line of sight of the robot can be accurately aligned to the target object, thereby ensuring the interaction quality.

[0059] According to an embodiment of the present application, the robot head can also be provided with a limit angle, including a horizontal limit angle and a vertical limit angle. In a case where the first target angle is not greater than the horizontal limit angle, and the second target angle is not greater than the vertical limit angle, the robot head can be controlled to rotate according to the determined first target angle and second target angle, otherwise, a warning information is sent.

[0060] Wherein, when the robot head needs to be controlled, the mechanical limiting structure of the robot head also needs to be considered. Thus, the limiting angle of the robot head can be determined, which can be pre-stored for the robot.

[0061] In an example, after the limiting angle of the robot head is determined, it can be judged whether the first target angle is not greater than the horizontal limiting angle, and whether the second target angle is not greater than the vertical limiting angle. If the judgment result is yes, the rotation of the robot head can be controlled according to the first target angle and the second target angle, and if any judgment result is no, a warning information is sent, wherein the warning information is used to prompt that the determined target angle is too large.

[0062] In the embodiments of the present application, by limiting the first target angle and the second target angle by the limiting angle within the mechanical limiting range, the faults such as gear wear and motor overload caused by the rotation of the robot head exceeding the mechanical limit are avoided, and the service life of the mechanical parts of the robot head is ensured.

[0063] According to an embodiment of the present application, step 220 comprises: determining the maximum horizontal angle of the robot head, wherein when the angle of the robot head exceeds the maximum horizontal angle, the target object will remove the collection range of the image sensor, determining the first proportional relationship between the pixels in the image and the rotation angle of the robot head in the horizontal direction according to the maximum horizontal angle and the width of the image, and determining the first target angle of the robot head in the horizontal direction according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the first proportional relationship.

[0064] In an example, when the projection of the installation position of the image sensor and the rotation center of the robot head in the vertical direction coincides, half of the horizontal field of view angle of the image sensor can be directly taken as the maximum angle of the robot head in the horizontal direction, that is, the maximum horizontal angle.

[0065] In an example, the maximum horizontal angle is used to limit the robot head no matter how to rotate, the line of sight in the horizontal direction is always within the collection range of the image sensor.

[0066] In an example, by pre-determining the maximum angle threshold that can make the line of sight of the robot always remain within the collection range of the image sensor in the horizontal direction when the robot head rotates, as the maximum horizontal angle of the robot head, the mapping relationship between the image pixel range in the horizontal direction and the rotation angle of the robot head can be established, that is, how many pixels the target object deflects, how many degrees the robot head needs to rotate in the horizontal direction. Then, according to the deviation of the position of the target object and the center position of the image in the horizontal direction, and the mapping relationship (i.e., the first proportional relationship) constructed, the first target angle of the robot head can be determined.

[0067] In an example, the maximum horizontal angle can be a maximum angle threshold that can keep the target object in the image sensor's capturing range in the horizontal direction when the robot head rotates after the image sensor is fixedly installed in the robot.

[0068] In an example, the maximum horizontal angle can be determined by a mapping relationship between the image pixel range and the rotation angle of the robot head in the horizontal direction, which is specifically that when the rotation angle of the robot head in the horizontal direction exceeds the maximum horizontal angle, the target object will exceed the imaging range of the image sensor, i.e., the position of the target object in the image exceeds the pixel coordinates of the image edge.

[0069] In an example, the maximum horizontal angle of the robot head can be determined in various ways, such as installing a laser emitter on the robot head, moving the position of the target object to the image edge, controlling the robot head to rotate, and determining the rotation angle of the robot head at this time as the maximum horizontal angle of the robot head. Alternatively, the maximum horizontal angle of the robot can be deduced by moving the target object to the laser drop point when the pixel position of the target object in the image. How to determine the maximum horizontal angle can be set as needed, and the present specification does not limit this.

[0070] In an example, half of the horizontal field of view angle of the image sensor can be directly taken as the maximum horizontal angle of the robot head. Thus, based on the pinhole imaging principle, the relationship between the focal length of the image sensor, the image width, and the horizontal field of view angle of the image sensor can be where HFOV is the horizontal field of view angle of the image sensor, which is twice the maximum horizontal angle of the robot head, f cam is the focal length of the image sensor, I w is the width of the image captured by the image sensor.

[0071] In an example, by the horizontal field of view angle of the image sensor can be determined, and thus the maximum horizontal angle of the robot head can be determined.

[0072] In an example, assuming HFOV=(―θ1,θ1), when the target object is located at the left edge of the image, the rotation angle of the robot head should be ―θ1, and when the target object is located at the right edge of the image, the rotation angle of the robot head should be θ1.

[0073] In an example, the focal length of the image sensor can be pre-stored data or obtained by calibrating with a chessboard calibration tool.

[0074] In an example, the width of the image can be pre-stored data of the robot, or can be data acquired in real time by the robot through the acquisition interface of the image sensor.

[0075] In an example, the first proportional relationship is used to represent the mapping relationship between the pixels and the rotation angle of the robot head in the horizontal direction, for example, 3 pixels correspond to 1 radian.

[0076] In an example, half of the horizontal field of view angle of the image sensor can be used as the maximum horizontal angle of the robot head. Then, according to the maximum horizontal angle and the width of the image acquired by the image sensor, the proportion between the pixels and the rotation angle of the robot head in the horizontal direction can be determined. For example, if the width of the image is 500 and the maximum horizontal angle of the robot head is 50, the first proportional relationship can be (500 / 2):50, that is, 5:1, that is, the distance of five pixels in the horizontal direction of the image can control the heading angle adjustment of the robot head by 1 degree.

[0077] In an example, the width of the image is w and the maximum horizontal angle is θ1, and the first proportional relationship is

[0078] In the embodiments of the present application, by constructing the optical geometric constraint of , the first proportional relationship between the width of the image and the rotation angle of the robot head in the horizontal direction is determined, so as to convert the pixel deviation into the first target angle in the horizontal direction according to the first proportional relationship, and the accuracy of the first target angle is ensured.

[0079] According to an embodiment of the present application, the projection of the rotation center of the robot head and the mounting position of the image sensor in the vertical direction does not coincide, and step 220 comprises: determining the horizontal region corresponding to the target object according to the position of the target object in the image, the horizontal region comprising one of the left region and the right region of the image, in the case that the horizontal region corresponding to the target object is the left region, determining the first target angle according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the second proportional relationship corresponding to the left region, and in the case that the horizontal region corresponding to the target object is the right region, determining the first target angle according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the third proportional relationship corresponding to the right region.

[0080] In an example, when the projection of the rotation center of the robot head and the mounting position of the image sensor in the vertical direction does not coincide, the angle of the line of sight of the robot turned from the left boundary of the image sensor to the right boundary is different from the angle corresponding to the horizontal field of view angle of the image sensor, and therefore, the horizontal field of view angle of the image sensor cannot be directly used as the rotation range of the heading angle of the robot head, that is, half of the horizontal field of view angle cannot be directly used as the maximum horizontal angle of the robot head. When the robot head is horizontally rotated, the spatial offset of the two will cause the mapping relationship of the pixels and angles of the left and right regions of the image to be asymmetric. Therefore, when the target object is in different regions of the image, the first target angle of the robot head needs to be accurately determined based on the proportional relationship of different regions.

[0081] In an example, in the horizontal direction, the initial line of sight of the robot head is directed from the direction of the robot head to the center of the horizontal field of view of the image sensor.

[0082] In an example, according to the position of the target object in the image, the horizontal region corresponding to the target object can be determined, the horizontal region including one of a left region and a right region of the image, and the first proportional relationship corresponding to the left region and the second proportional relationship corresponding to the right region have different numerical values.

[0083] In an example, when the horizontal region corresponding to the target object is the left region, the first target angle can be determined according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the second proportional relationship corresponding to the left region. The rotation direction of the robot head is to the left, and the angle after rotation is the first target angle.

[0084] In an example, when the horizontal region corresponding to the target object is the right region, the first target angle can be determined according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the third proportional relationship corresponding to the right region. The rotation direction of the robot head is to the right, and the angle after rotation is the first target angle, and the numerical values of the second proportional relationship and the third proportional relationship are different.

[0085] In the embodiments of the present application, for the case that the rotation center of the robot head and the mounting position of the image sensor in the vertical direction do not coincide, the horizontal rotation will cause the mapping relationship of the pixels and angles of the left and right regions of the image to be asymmetric, by dividing the image into a left region and a right region, and using the proportional relationship of the corresponding region (the second proportional relationship for the left region and the third proportional relationship for the right region) to calculate the first target angle (left or right turning angle), the mapping asymmetry problem caused by the spatial offset is effectively solved, the accuracy of the rotation angle of the robot head is improved, and the accuracy and reliability of the visual tracking or perception of the robot to the target object are improved.

[0086] According to an embodiment of the present application, the robot control method further comprises: adjusting an initial line of sight of the robot head to be towards a center of a horizontal field of view of the image sensor, respectively determining a first angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view to a left boundary of the horizontal field of view, and a second angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view to a right boundary of the horizontal field of view, determining a second proportional relationship according to the first angle and a width of the image, and determining a third proportional relationship according to the second angle and the width of the image.

[0087] In an example, as shown in FIG. 2, the robot control method further comprises: adjusting an initial line of sight of the robot head to be towards a center of a horizontal field of view of the image sensor, respectively determining a first angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view to a left boundary of the horizontal field of view, and a second angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view to a right boundary of the horizontal field of view, determining a second proportional relationship according to the first angle and a width of the image, and determining a third proportional relationship according to the second angle and the width of the image. Figure 3A Figure 3A A structural diagram of a maximum horizontal angle of a robot head provided by an example embodiment of the present application is shown in FIG. 2, in which point A is a projection of a mounting position of the image sensor in a vertical direction, point B is a projection of a rotation center of the robot head in the vertical direction, points C and D are located on an imaging plane with a specific depth from the image sensor and correspond to left and right boundaries of a horizontal field of view of the image sensor respectively, and point E is a midpoint of line segment CD, that is, AE points to a center of the horizontal field of view of the image sensor. Obviously, the sizes of ∠CAD and ∠CBD are different, and the sizes of ∠CBE and ∠EBD are also different. Therefore, it is necessary to determine a second proportional relationship corresponding to the left region and a third proportional relationship corresponding to the right region respectively based on ∠CBE and ∠EBD and the width of the image.

[0088] In an example, the angle of the robot head can be adjusted first, so that the initial line of sight of the robot is towards the center of the horizontal field of view of the image sensor, that is, the heading angle of the robot head is adjusted to the BE direction in FIG. 2. In order to facilitate subsequent control of the rotation of the robot head based on the initial line of sight of the robot. Figure 3A

[0089] In an example, a first angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view of the image sensor to the left boundary of the horizontal field of view of the image sensor, and a second angle of rotation of the robot head when the line of sight of the robot is adjusted from the center of the horizontal field of view of the image sensor to the right boundary of the horizontal field of view of the image sensor can be respectively determined, that is, ∠EBC when the line of sight of the robot is adjusted from the BE direction to the BC direction in FIG. 2 is determined as the first angle, and ∠EBD when the line of sight of the robot is adjusted from the BE direction to the BD direction in FIG. 2 is determined as the second angle.

[0090] In an example, ∠EBC and ∠EBD can be determined by calculation or calibration.

[0091] ​​In an example, the calculation manner can be that, given the depth of the CD (i.e., a specific depth), based on the known horizontal field of view angle of the image sensor (of ∠CAD), the deviation between the projection of the installation position of the image sensor and the rotation center of the robot head in the vertical direction (the distance between AB), ∠EBC and ∠EBD respectively corresponding angle values can be calculated.

[0092] In an example, the calibration manner can be that, when the laser emitter is installed on the robot head, the heading angle when the laser landing point is point E is determined as the initial angle of the robot head in the horizontal direction, and the first angle of the robot head in the horizontal direction when the laser landing point is adjusted from point E to point C is determined, and the second angle of the robot head in the horizontal direction when the laser landing point is adjusted from point E to point D is determined, so that ∠EBC and ∠EBD respectively corresponding angle values are obtained.

[0093] In the embodiments of the present application, by adjusting the initial line of sight of the robot to the center of the horizontal field of view of the image sensor, and respectively calibrating / calculating the angles of the head from the center to the left and right boundaries of the field of view, the different proportional relationships corresponding to the left and right regions are determined, and the problem of asymmetric mapping of pixels and angles in the left and right regions of the image caused by the fact that the projection of the rotation center of the robot head and the installation position of the image sensor in the vertical direction does not coincide is effectively solved, which provides a basis for subsequent accurate calculation of the rotation angle of the head.

[0094] According to an embodiment of the present application, step 220 comprises: determining a fourth proportional relationship according to the size of the target object in the image and the preset size of the target object, and determining the height information of the target object according to the position of the target object in the image and the deviation of the center position of the image in the vertical direction, and the fourth proportional relationship, determining the depth information of the target object, and determining the second target angle of the robot head in the vertical direction according to the depth information and the height information of the target object, and the height information of the robot head.

[0095] When the image sensor is arranged in the non-head region of the robot, the height of the image sensor and the height of the robot head are often inconsistent, at this time, the height information and the depth information of the target object can be determined based on the existing information, and then the relative position relationship between the target object and the robot head in the three-dimensional space is determined according to the determined height information and depth information and the height information of the image sensor, and the second target angle of the robot head is determined.

[0096] In an example, the actual height of the target object in the three-dimensional space, i.e., the height information of the target object, can be determined according to the height information of the image sensor and the relative height between the image sensor and the target object. Wherein, the height information of the image sensor is known, and the relative height between the image sensor and the target object can be determined according to the deviation of the position of the target object in the image and the center position of the image in the vertical direction.

[0097] In an example, the relative height between the target object and the image sensor can be determined according to the deviation of the position of the target object in the image and the center position of the image in the vertical direction, and a fourth proportional relationship. Wherein, the fourth proportional relationship is used to represent the proportional relationship between the size of the target object in the image and the actual size of the target object, and thus the fourth proportional relationship can also represent the proportional relationship between the pixel in the image and the actual height or actual width in the three-dimensional space at the depth corresponding to the target object.

[0098] In an example, for different types of target objects, the preset size (i.e., the size that the target object should actually correspond to) of each type of target object can be pre-stored, and the preset size includes at least one of the preset height and the preset depth. For example, if the target object is a human face, the corresponding preset width can be 0.15m, and if the target object is a traffic light, the corresponding actual width can be 0.2m. The actual size of each type of target object can be set as needed, and the present application does not limit this.

[0099] In an example, taking the determination of the fourth proportional relationship according to the width of the target object in the image and the preset width in the preset size of the target object as an example, assuming that the width of the target object is w, and the preset width of the target object is w obj , then the fourth proportional relationship is Wherein, the width of the target object can be the width of the detection frame of the target object. The fourth proportional relationship can be used to represent the proportional relationship between the size of the target object in the image and the actual size of the target object in the three-dimensional space.

[0100] In an example, after determining the fourth proportional relationship, the relative height between the target object and the image sensor can be determined according to the fourth proportional relationship and the deviation of the position of the target object in the image and the center position of the image in the vertical direction, and the height information of the target object can be determined according to the determined relative height and the height information of the image sensor. Taking the fourth proportional relationship of 1:10 as an example, the actual height or actual width of one pixel in the image in the three-dimensional space can be 10 cm, assuming that the deviation of the position of the target object in the image and the center position of the image in the vertical direction is 5 (assuming that the position of the target object in the image is above the center position of the image, the deviation is positive, and the position of the target object in the image is below the center position of the image, the deviation is negative), and the height of the image sensor is 150 cm, then the height information of the image sensor is 150+5x10=200 cm.

[0101] In an example, the depth information of the target object and the image sensor can be determined, which can be achieved by a depth camera or other electronic devices that can be used to measure depth information arranged on the robot.

[0102] In an example, the fourth proportional relationship can also represent the proportional relationship between the width of the image and the width of the horizontal field of view of the image sensor at the depth of the target object, and based on the imaging principle, the fourth proportional relationship can also be used to represent the proportional relationship between the focal length of the image sensor and the focal length between the target object and the image sensor, so that based on the fourth proportional relationship, the depth information between the target object and the image sensor can be determined.

[0103] In an example, after determining the depth information and height information of the target object, the second target angle of the head of the robot in the vertical direction can be determined according to the height information of the head of the robot and the depth information and height information of the target object: wherein a is within the vertical limiting angle, H robot H is the height information of the head of the robot, obj D is the height information of the target object, that is, the actual height of the target object in the three-dimensional space, obj D is the depth information of the target object.

[0104] In an example, the foregoing is described by taking the limit angle in the vertical direction determined according to the difference between the target object and the robot head in the three-dimensional space as an example. In actual application, the robot can also determine the limit angle in the horizontal direction according to the difference between the target object and the robot head in the three-dimensional space. Specifically, the depth information of the target object can be determined by the size of the target object in the image and the preset size of the target object, a fourth proportional relationship is determined according to the fourth proportional relationship, the distance of the target object offset from the center of the horizontal field of view of the image sensor is determined according to the difference between the position of the target object in the image and the center position of the image in the horizontal direction, and the fourth proportional relationship, and finally the corresponding target angle of the robot head in the horizontal direction is determined according to the depth information and the distance of the target object offset from the center of the horizontal field of view of the image sensor.

[0105] It should be noted that the height information of the image sensor, the height information of the robot head, and the height of the image can be set as needed, and the present application does not limit this.

[0106] In the embodiment of the present application, in order to solve the problem that the image sensor is set in the non-head region of the robot, which causes the height difference between the sensor and the head, the fourth proportional relationship determined by the image size of the target object and the preset size is combined with the position deviation of the target object in the image and the depth information to accurately calculate the height and depth information of the target object, and then the second target angle of the robot head in the vertical direction is determined, thereby effectively solving the three-dimensional positioning error caused by the height difference between the sensor and the head, improving the accuracy and reliability of the visual tracking or interaction of the robot head to the target object, and further ensuring the interaction quality.

[0107] According to an embodiment of the present application, step 220 comprises: determining the corresponding vertical region of the target object according to the position of the target object in the image, the vertical region comprising one of the upper region and the lower region of the image, in the case that the corresponding vertical region of the target object is the upper region, determining the second target angle according to the position deviation of the target object in the image and the center position of the image in the vertical direction, and the fifth proportional relationship corresponding to the upper region, and in the case that the corresponding vertical region of the target object is the lower region, determining the second target angle according to the position deviation of the target object in the image and the center position of the image in the vertical direction, and the sixth proportional relationship corresponding to the lower region.

[0108] In an example, when the mounting positions of the robot head and the image sensor are inconsistent in height, the line of sight of the robot is directed at an angle different from the angle corresponding to the vertical field of view angle of the image sensor when the line of sight is turned from the upper boundary to the lower boundary of the vertical field of view of the image sensor, and therefore, the vertical field of view angle of the image sensor cannot be directly used as the rotation range of the pitch angle of the robot head. When the robot head is rotated in the vertical direction, the spatial offset between the two causes the mapping relationship between the pixels and the angle of the upper and lower regions of the image to be asymmetric, and therefore, different scaling relationships can be set for different regions of the image, and the second target angle of the robot head can be accurately determined according to the scaling relationship corresponding to the region where the image is located.

[0109] In an example, in the vertical direction, the initial line of sight of the robot head is directed from the direction of the robot head to the direction of the center of the vertical field of view of the image sensor.

[0110] In an example, the vertical region corresponding to the target object can be determined according to the position of the target object in the image, the vertical region including one of the upper region and the lower region of the image, and the values of the fifth scaling relationship corresponding to the upper region and the sixth scaling relationship corresponding to the lower region being different values.

[0111] In an example, when the vertical region corresponding to the target object is the upper region, the second target angle can be determined according to the deviation of the position of the target object in the image from the center position of the image in the vertical direction and the fifth scaling relationship corresponding to the upper region, wherein the rotation direction of the robot head is upward rotation, and the angle after rotation is the second target angle.

[0112] In an example, when the vertical region corresponding to the target object is the lower region, the second target angle can be determined according to the deviation of the position of the target object in the image from the center position of the image in the vertical direction and the sixth scaling relationship corresponding to the lower region, wherein the rotation direction of the robot head is downward rotation, and the angle after rotation is the second target angle.

[0113] In the embodiments of the present application, for the case where the height of the robot head is different from the height of the image sensor, the image is divided into an upper region and a lower region, and the second target angle (upward rotation or downward rotation angle) is calculated using the scaling relationship corresponding to the region (the fifth scaling relationship for the upper region and the sixth scaling relationship for the lower region), thereby ensuring the accuracy of the determined second target angle, improving the precision of the rotation angle of the robot head, and further improving the accuracy and reliability of the visual tracking or perception of the robot on the target object, and further ensuring the interaction quality.

[0114] According to an embodiment of the present application, the robot control method further comprises: adjusting the initial line of sight of the robot head to be towards the center of the vertical field of view of the image sensor, respectively determining a third angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view to the upper boundary of the vertical field of view, and a fourth angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view to the lower boundary of the vertical field of view, determining a fifth proportional relationship according to the third angle and the height of the image, and determining a sixth proportional relationship according to the fourth angle and the height of the image.

[0115] In an example, as shown in Figure 3B , Figure 3B A structural diagram of the maximum vertical angle of the robot head is provided for an example embodiment of the present application. In the diagram, point P is used to represent the height of the installation position of the image sensor, point O is used to represent the height of the robot head, points N and Q are located on the imaging plane with a specified depth from the image sensor, and correspond to the upper boundary and the lower boundary of the vertical field of view thereof respectively, and point M is the midpoint of line segment QN, that is, OM points to the center of the vertical field of view of the image sensor. Obviously, ∠NPQ and ∠NOQ are different in size, and ∠MOQ and ∠MON are also different in size.

[0116] In an example, the angle of the robot head can be adjusted first, so that the initial line of sight of the robot is towards the center of the vertical field of view of the image sensor, that is, the pitch angle of the robot head is adjusted to the OM direction as shown in Figure 3B , so that the subsequent rotation of the robot head can be controlled based on the initial line of sight of the robot head.

[0117] In an example, a third angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view of the image sensor to the upper boundary of the vertical field of view of the image sensor, and a fourth angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view of the image sensor to the lower boundary of the vertical field of view of the image sensor, can be respectively determined. That is, ∠MON at which the robot head turns in the process of adjusting the direction of the line of sight of the robot from the OM direction to the ON direction is determined as the third angle, and ∠MOQ at which the robot head turns in the process of adjusting the line of sight of the robot from the OM direction to the OQ direction is determined as the fourth angle.

[0118] In an example, ∠MON and ∠MOQ can be determined by calculation or by calibration.

[0119] In an example, the calculation manner can be that, given the depth of NQ (the specified depth, where the specified depth and the particular depth can be the same depth or different depths), based on the known vertical field of view angle (∠NPQ) of the image sensor, the height information of the image sensor and the height information of the robot head, the angle values corresponding to ∠MON and ∠MOQ can be calculated.

[0120] In an example, the calibration manner can be that, when the laser emitter is installed on the robot head, the pitch angle when the laser landing point is point M is determined as the initial angle of the robot head in the vertical direction, and the third angle when the laser landing point is adjusted from point M to point N and the robot head is rotated in the vertical direction, and the fourth angle when the laser landing point is adjusted from point M to point Q and the robot head is rotated in the vertical direction are determined respectively, so that the angle values corresponding to ∠MON and ∠MOQ are obtained.

[0121] In the embodiments of the present application, by adjusting the initial line of sight of the robot to the center of the vertical field of view of the image sensor, and respectively calibrating / calculating the angles of the head from the center of the vertical field of view to the upper and lower boundaries of the vertical field of view, and then determining the different proportional relationships of the upper and lower regions, the problem of inconsistent mapping of pixels in the upper and lower regions of the image and angles caused by the inconsistency between the height of the robot head and the height of the image sensor is effectively solved.

[0122] According to an embodiment of the present application, the target object is a face, and the robot control method further comprises: determining the type of the target object, different types of target objects having different age stages, and determining the preset size of the target object according to the type of the target object.

[0123] In an example, among various scenes in which the robot interacts with the target object, human-computer interaction is the main application scene of the robot interaction, and in the human-computer interaction scene, the line of sight of the robot needs to be controlled to be directed to the face direction of the user, and therefore, the target object in the image can be a face.

[0124] In an example, when the target object is a face, the size of the face is usually quite different for different age stages, for example, the width of the face of a user aged 0-5 years is less than the width of the face of a user aged 35-45 years. Therefore, when the target object is determined, the target object can be divided into multiple types based on the age stage, and the size of the target object of each type is preset as the preset size of the target object of the type.

[0125] In an example, the faces of users in different age stages can be regarded as different types of target objects, and the types of the target objects contained in the image can be: the face of a user aged 0-5 years, the face of a user aged 35-45 years, a cat, goods, etc.

[0126] In an example, the robot can store preset sizes corresponding to different types of target objects, so that the robot can determine the actual width of the target object according to the type of the target object.

[0127] In the embodiments of the present application, different preset sizes are set for different types of target objects, so that the robot can determine the preset size of the target object according to the type of the target object, and then determine a more accurate fourth proportional relationship according to the preset size, thereby ensuring the accuracy of the determined second target angle and the interaction quality.

[0128] According to an embodiment of the present application, step 230 includes: when the first target angle is greater than the first preset angle, controlling the robot head to rotate at a first speed, otherwise, controlling the robot head to rotate at a second speed, wherein the first speed is greater than the second speed, when the second target angle is greater than the second preset angle, controlling the robot head to rotate at a third speed, otherwise, controlling the robot head to rotate at a fourth speed, wherein the third speed is greater than the fourth speed.

[0129] In an example, when the target angle is too large, the robot may need a long time to adjust the robot head to the position corresponding to the target object, therefore, the first preset angle and the second preset angle can be set respectively to control the robot head to rotate at a faster speed when the first target angle exceeds the first preset angle or the second target angle exceeds the second preset angle.

[0130] In an example, the first preset angle is less than the maximum horizontal angle, the second preset angle is less than the third angle or the fourth angle, the first speed is greater than the second speed, and the third speed is greater than the fourth speed.

[0131] In an example, when the first target angle is greater than the first preset angle, the robot head can be controlled to rotate in the horizontal direction at the first speed to adjust the heading angle. When the first target angle is less than or equal to the first preset angle, the robot head can be controlled to rotate in the horizontal direction at the second speed to adjust the heading angle.

[0132] In an example, when the second target angle is greater than the second preset angle, the robot head can be controlled to rotate in the vertical direction at the third speed to adjust the pitch angle. When the second target angle is less than or equal to the second preset angle, the robot head can be controlled to rotate in the vertical direction at the fourth speed to adjust the pitch angle.

[0133] In the embodiments of the present application, the rotation speed of the robot head when the target angle is too large and the positioning accuracy of the robot head when the target angle is not too large can be ensured, thereby further ensuring the interaction quality.

[0134] According to an embodiment of the present application, the target object in the image is multiple, and step 210 includes: determining multiple candidate target objects in the image, and determining a weight of each candidate target object in the multiple candidate target objects; and determining the target object corresponding to the image from the multiple candidate target objects according to the weights corresponding to the multiple candidate target objects respectively.

[0135] In an example, there can be multiple target objects in the image, but the robot can only interact with one target object at the same time, and therefore, the weights corresponding to the target objects respectively can be determined to select the target object that the robot needs to interact with.

[0136] In an example, the robot can identify the image to determine multiple candidate target objects. The multiple candidate target objects can include a pet, a baby face, an adult face, and the like.

[0137] In an example, for each candidate target object, the robot can determine the weight of the candidate target object according to the appearance frequency of the candidate target object in a specified number of continuous frame images before the current frame image, and according to the appearance frequency.

[0138] In an example, the appearance frequency and the weight are inversely proportional.

[0139] In an example, the weight of the candidate target object can be determined according to the appearance frequency of the candidate target object, and the deviation of the position of the candidate target object from the center of the image, wherein the deviation is inversely proportional to the weight.

[0140] In an example, the target object with the highest weight can be selected from the candidate target objects as the target object corresponding to the image.

[0141] In an example, a candidate target object with a weight higher than a preset weight threshold can be randomly determined from the candidate target objects as the target object corresponding to the image.

[0142] In the embodiments of the present application, the weights of multiple candidate target objects in the image are determined based on the historical appearance frequencies of the candidate target objects, and the target object is selected according to the weights, so that the robot can efficiently and accurately screen out a priority interaction target in a multi-target scene, and the interaction pertinence and scene adaptability are improved.

[0143] Figure 4 Fig. 3 shows a flowchart of a robot control method provided by another example embodiment of the present application. Figure 4 The embodiments are Figure 2 For the sake of brevity, the same parts can be referred to the description of the above embodiments, and will not be described here. For example, Figure 4 As shown in the figure, the robot control method can include the following contents.

[0144] 400: determining a target object in the image captured by the image sensor.

[0145] 410: determining a second target angle.

[0146] 420: determining whether the projection of the installation position of the image sensor and the rotation center of the robot head in the vertical direction coincides. If yes, step 430 is performed, and if no, step 450 is performed.

[0147] 430: determining a first proportional relationship according to the maximum horizontal angle of the robot head and the width of the image.

[0148] 440: determining a first target angle according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the first proportional relationship.

[0149] 450: determining a horizontal region of the target object in the image, the horizontal region including one of a left region and a right region.

[0150] 460: determining a first target angle according to the deviation of the position of the target object in the image and the center position of the image in the horizontal direction, and the proportional relationship corresponding to the horizontal region of the target object in the image.

[0151] 470: determining whether the first target angle is not greater than a horizontal limit angle and the second target angle is not greater than a vertical limit angle, if yes, step 480 is performed, and if any result is no, step 490 is performed.

[0152] 480: controlling the robot head to rotate according to the first target angle and the second target angle.

[0153] 490: sending a warning information.

[0154] In the embodiment, the contents described in 410-490 can be referred to the description of steps 210-230, which will not be repeated here.

[0155] In the embodiment, by determining whether the projection of the installation position of the image sensor and the rotation center of the robot head in the vertical direction coincides, different strategies are selected to determine the first target angle, and the second target angle is determined, and the first target angle and the second target angle are limited according to the limit angle. The embodiment improves the accuracy of the first target angle calculation in the horizontal direction by processing the projection coincidence and the regional proportional relationship, and guarantees the safety of the robot head rotation through the limit determination, finally improves the accuracy and reliability of the visual tracking or interaction of the robot to the target object, and guarantees the interaction quality.

[0156] Figure 5Fig. 1 shows a flowchart of a robot control method according to an example embodiment of the present application. Figure 5 An example embodiment is Figure 2 Another example embodiment is provided to avoid repetition. The same parts can refer to the description of the above example embodiments, which will not be repeated here. For example, Figure 5 As shown in the figure, the robot control method can include the following.

[0157] 510: Determine the target object of the image collected by the image sensor.

[0158] 520: Determine the first target angle.

[0159] 530: Determine the fourth proportional relationship according to the size of the target object in the image and the preset size of the target object.

[0160] 540: Determine the height information of the target object according to the fourth proportional relationship, and determine the second target angle according to the height information of the target object, the depth information of the target object, and the height information of the robot head.

[0161] 550: Determine whether the first target angle is not greater than the horizontal limit angle and the second target angle is not greater than the vertical limit angle. If yes, execute step 560, and if any result is no, execute step 570.

[0162] 560: Control the robot head to rotate according to the first target angle and the second target angle.

[0163] 570: Send a warning message.

[0164] In this embodiment, the contents described in 510-570 can refer to the description of steps 210-230 above, which will not be repeated here.

[0165] In this embodiment, the height information of the target object in the three-dimensional space and the depth information between the target object and the robot can be determined by calculation, and the second target angle can be determined at the same time. The first target angle is determined, and the first target angle and the second target angle are limited by the limit angle, which can ensure the safety of the robot head rotation while determining the second target angle in different ways, thereby ensuring the accuracy of the determined second target angle and the interaction quality.

[0166] Figure 6 Fig. 1 shows a flowchart of a robot control method according to an example embodiment of the present application. Figure 6 An example embodiment is Figure 2 Another example embodiment is provided to avoid repetition. The same parts can refer to the description of the above example embodiments, which will not be repeated here. For example, Figure 6 As shown in the figure, the robot control method can include the following.

[0167] 610: determine a target object in an image captured by the image sensor.

[0168] 620: determine a first target angle.

[0169] 630: determine a vertical region of the target object in the image, the vertical region including one of an upper region and a lower region.

[0170] 640: determine a second target angle according to a deviation of a position of the target object in the image and a central position of the image in a vertical direction, and a proportional relationship corresponding to the vertical region of the target object in the image.

[0171] 650: determine whether the first target angle is not greater than a horizontal limiting angle and the second target angle is not greater than a vertical limiting angle, if yes, execute step 660, and if any result is no, execute step 670.

[0172] 660: control the robot head to rotate according to the first target angle and the second target angle.

[0173] 670: send a warning information.

[0174] In the embodiment, the contents described in steps 610-670 can be referred to the description of steps 210-230, which will not be repeated herein.

[0175] In the embodiment, the second target angle can be determined according to the proportional relationship corresponding to the vertical region of the target object in the image, and the first target angle is determined at the same time, and the first target angle and the second target angle are limited according to the limiting angle, which can ensure the safety of the rotation of the robot head, and determine the second target angle in different ways to ensure the accuracy of the determined second target angle and the interaction quality.

[0176] It should be understood that the execution order of the above steps can be adjusted according to actual needs.

[0177] Exemplary device

[0178] The embodiment of the application further provides a robot, which comprises a robot head, an image sensor and a control module, the control module is configured to execute the robot control method provided in any of the above embodiments. Figure 2 、 Figure 4 、 Figure 5 、 Figure 6

[0179] The specific functions and effects of the robot provided in the embodiment of the application can be referred to the description of the above method embodiments, which will not be repeated herein to avoid repetition. ​

[0180] Figure 7 Fig. 1 shows a structural schematic diagram of a task arrangement device provided by an example embodiment of the present application, and the robot control device can be applied to a robot, and the robot is provided with an image sensor, and the image sensor is arranged at a non-head region of the robot. As shown in Fig. 1, the robot control device 100 includes a target determination module 110, an angle determination module 120 and a control module 130. Figure 7

[0181] The target determination module 110 is configured to determine a target in an image collected by the image sensor.

[0182] The angle determination module 120 is configured to determine a first target angle of the head of the robot in a horizontal direction and a second target angle of the head of the robot in a vertical direction according to a deviation between a position of the target in the image and a center position of the image, and any one of the first target angle and the second target angle is used to represent an angle required to be adjusted in a direction corresponding to the target angle when a line of sight of the robot is adjusted to be directed to the position of the target under a condition that the line of sight of the robot is directed to the center position of the image.

[0183] The control module 130 is configured to control the head of the robot to rotate according to the first target angle and the second target angle.

[0184] Optionally, the angle determination module 120 is configured to determine a maximum horizontal angle of the head of the robot, and when an angle of a heading angle of the head of the robot exceeds the maximum horizontal angle, the target will move out of a collection range of the image sensor, determine a first proportional relationship between a pixel in the image and a rotation angle of the head of the robot in the horizontal direction according to the maximum horizontal angle and a width of the image, and determine the first target angle of the head of the robot in the horizontal direction according to a deviation between the position of the target in the image and the center position of the image in the horizontal direction and the first proportional relationship.

[0185] Optionally, a projection of a rotation center of the head of the robot and a mounting position of the image sensor in the vertical direction does not coincide; and the angle determination module 120 is configured to determine a horizontal region corresponding to the target according to the position of the target in the image, the horizontal region includes one of a left region and a right region of the image, in a case that the horizontal region corresponding to the target is the left region, determine the first target angle according to the deviation between the position of the target in the image and the center position of the image in the horizontal direction and a second proportional relationship corresponding to the left region, and in a case that the horizontal region corresponding to the target is the right region, determine the first target angle according to the deviation between the position of the target in the image and the center position of the image in the horizontal direction and a third proportional relationship corresponding to the right region.

[0186] ​Optionally, the angle determining module 720 is further configured to: adjust the initial line of sight of the robot head to be towards a center of a horizontal field of view of the image sensor, respectively determine a first angle at which the line of sight of the robot is adjusted from the center of the horizontal field of view to a left boundary of the horizontal field of view and a second angle at which the line of sight of the robot is adjusted from the center of the horizontal field of view to a right boundary of the horizontal field of view, determine a second proportional relationship according to the first angle and a width of the image, and determine a third proportional relationship according to the second angle and the width of the image.

[0187] Optionally, the angle determining module 720 is configured to: determine a fourth proportional relationship according to a size of the target object in the image and a preset size of the target object, determine the height information of the target object according to a position of the target object in the image and a deviation of a center position of the image in a vertical direction and the fourth proportional relationship, determine the depth information of the target object, and determine the second target angle of the robot head in the vertical direction according to the depth information and the height information of the target object and height information of the robot head.

[0188] Optionally, the angle determining module 720 is configured to: determine a vertical region corresponding to the target object according to the position of the target object in the image, the vertical region including one of an upper region and a lower region of the image, in a case where the vertical region corresponding to the target object is the upper region, determine the second target angle according to the position of the target object in the image and the deviation of the center position of the image in the vertical direction and a fifth proportional relationship corresponding to the upper region, and in a case where the vertical region corresponding to the target object is the lower region, determine the second target angle according to the position of the target object in the image and the deviation of the center position of the image in the vertical direction and a sixth proportional relationship corresponding to the lower region.

[0189] Optionally, the angle determining module 720 is further configured to: adjust the initial line of sight of the robot head to be towards a center of a vertical field of view of the image sensor, respectively determine a third angle at which the line of sight of the robot is adjusted from the center of the vertical field of view to an upper boundary of the vertical field of view and a fourth angle at which the line of sight of the robot is adjusted from the center of the vertical field of view to a lower boundary of the vertical field of view, determine a fifth proportional relationship according to the third angle and a height of the image, and determine a sixth proportional relationship according to the fourth angle and the height of the image.

[0190] Optionally, the target object is a human face, and the target object determining module 710 is configured to: determine a type of the target object, different types of target objects having different age stages, and determine a preset size of the target object according to the type of the target object.

[0191] Optionally, the target object determining module 710 is configured to determine a plurality of candidate target objects in the image, determine a weight of each candidate target object in the plurality of candidate target objects, and determine the target object corresponding to the image from the plurality of candidate target objects according to the weights corresponding to the plurality of candidate target objects respectively.

[0192] Optionally, the target object determining module 710 is configured to determine the weight of the candidate target object according to the frequency of occurrence of the candidate target object in images collected by the image sensor historically, and the frequency of occurrence and the weight are negatively correlated.

[0193] It should be understood that the operations and functions of the target object determining module 710, the angle determining module 720 and the control module 730 in the above embodiments can be referred to the descriptions of the robot control method provided in the above embodiments. For the sake of brevity, they will not be repeated here. Figure 2

[0194] Figure 8 Fig. 8 shows a block diagram of an electronic device 800 for executing the robot control method according to an example embodiment of the present application. The electronic device 800 can be a robot, a server or other device interacting with the robot.

[0195] Referring to Figure 8 , the electronic device 800 includes a processing component 810, which further includes one or more processors, and a memory resource represented by a memory 820, for storing instructions executable by the processing component 810, such as an application program. The application program stored in the memory 820 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 810 is configured to execute the instructions to perform the robot control method described above.

[0196] The electronic device 800 can further include a power supply component configured to perform power management of the electronic device 800, a wired or wireless network interface configured to connect the electronic device 800 to a network, and an input / output (I / O) interface. The electronic device 800 can be operated based on an operating system stored in the memory 820, such as Windows Server TM , MacOSX TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0197] A non-transitory computer readable storage medium, when the instructions stored in the storage medium are executed by the processor of the above-mentioned electronic device 800, enable the above-mentioned electronic device 800 to perform a robot control method.

[0198] ​A computer program product comprising a computer program, when executed by a processor of a computer device, enables the computer device to perform the robot control method provided in any of the above embodiments.

[0199] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0200] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0202] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0203] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0204] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0205] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0206] It should be noted that in the description of the present application, the terms "first", "second", "third" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0207] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0208] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method characterized by, The robot is provided with an image sensor arranged at a non-head region of the robot, and the robot control method comprises: determining a target object in an image captured by the image sensor; determining a first target angle of the robot head in a horizontal direction and a second target angle of the robot head in a vertical direction according to a deviation between a position of the target object in the image and a center position of the image, wherein any one of the first target angle and the second target angle is used to represent an angle required to be adjusted in a direction corresponding to the target angle when a line of sight of the robot is adjusted to be directed to the position of the target object under a condition that the line of sight of the robot is directed to the center position of the image; controlling the robot head to rotate according to the first target angle and the second target angle.

2. The robot control method according to claim 1, characterized by, The determination of the first target angle of the robot head in the horizontal direction according to the deviation between the position of the target object in the image and the center position of the image comprises: determining a maximum horizontal angle of the robot head, wherein the target object will move out of a capturing range of the image sensor when a heading angle of the robot head exceeds the maximum horizontal angle; determining a first proportional relationship between a pixel in the image and a rotation angle of the robot head in the horizontal direction according to the maximum horizontal angle and a width of the image; determining the first target angle of the robot head in the horizontal direction according to the deviation between the position of the target object in the image and the center position of the image in the horizontal direction and the first proportional relationship.

3. The robot control method according to claim 1, wherein, A projection of a rotation center of the robot head and a mounting position of the image sensor in the vertical direction does not coincide; The determination of the first target angle of the robot head in the horizontal direction according to the deviation between the position of the target object in the image and the center position of the image comprises: determining a horizontal region corresponding to the target object according to the position of the target object in the image, wherein the horizontal region comprises one of a left region and a right region of the image; in a case where the horizontal region corresponding to the target object is the left region, determining the first target angle according to the deviation between the position of the target object in the image and the center position of the image in the horizontal direction and a second proportional relationship corresponding to the left region; in a case where the horizontal region corresponding to the target object is the right region, determining the first target angle according to the deviation between the position of the target object in the image and the center position of the image in the horizontal direction and a third proportional relationship corresponding to the right region.

4. The robot control method according to claim 3, wherein, The robot control method further comprises: adjusting an initial line of sight of the robot head to be directed to a center of a horizontal field of view of the image sensor; determining a first angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the horizontal field of view to the left boundary of the horizontal field of view, and a second angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the horizontal field of view to the right boundary of the horizontal field of view; determining a second proportional relationship according to the first angle and the width of the image, and determining a third proportional relationship according to the second angle and the width of the image.

5. The robot control method according to claim 1, wherein, determining a second target angle of the robot head in the vertical direction according to the deviation between the position of the target object in the image and the central position of the image, including: determining a fourth proportional relationship according to the size of the target object in the image and the preset size of the target object, and determining the height information of the target object according to the deviation between the position of the target object in the image and the central position of the image in the vertical direction and the fourth proportional relationship; determining the depth information of the target object, and determining the second target angle of the robot head in the vertical direction according to the depth information and the height information of the target object and the height information of the robot head.

6. The robot control method according to claim 1, wherein, determining a second target angle of the robot head in the vertical direction according to the deviation between the position of the target object in the image and the central position of the image, including: determining a vertical region corresponding to the target object according to the position of the target object in the image, the vertical region including one of an upper region and a lower region of the image; in a case where the vertical region corresponding to the target object is the upper region, determining the second target angle according to the deviation between the position of the target object in the image and the central position of the image in the vertical direction and a fifth proportional relationship corresponding to the upper region; in a case where the vertical region corresponding to the target object is the lower region, determining the second target angle according to the deviation between the position of the target object in the image and the central position of the image in the vertical direction and a sixth proportional relationship corresponding to the lower region.

7. The robot control method according to claim 6, wherein, The robot control method further includes: adjusting the initial line of sight of the robot head to be directed to the center of the vertical field of view of the image sensor; determining a third angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view to the upper boundary of the vertical field of view, and a fourth angle at which the robot head turns when the line of sight of the robot is adjusted from the center of the vertical field of view to the lower boundary of the vertical field of view; determining a fifth proportional relationship according to the third angle and the height of the image, and determining a sixth proportional relationship according to the fourth angle and the height of the image.

8. The robot control method according to claim 5, wherein, The target object is a human face. The robot control method further includes: determining the type of the target object, and the age stages of target objects of different types being different; determining the preset size of the target object according to the type of the target object.

9. The robot control method according to claim 1, wherein, The determination of the target object in the image collected by the image sensor includes: determining a plurality of candidate objects in the image, and determining a weight of each candidate object in the plurality of candidate objects; determining the object corresponding to the image from the plurality of candidate objects according to the weight corresponding to each candidate object.

10. The robot control method according to claim 9, wherein, The determining the weight of the candidate object comprises: determining the weight of the candidate object according to the frequency of appearance of the candidate object in the images collected by the image sensor in the past, the frequency of appearance and the weight being negatively correlated.

11. A robot, characterized in that The robot comprises a robot head, an image sensor and a control module, the image sensor is arranged in a non-head region of the robot, and the control module is used to execute the robot control method in any one of claims 1 to 10.

12. An electronic device, comprising: The electronic device comprises a processor and a memory for storing processor-executable instructions, wherein the processor is used to execute the robot control method in any one of claims 1 to 10.

13. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by a processor of a computer device, so that the computer device can execute the robot control method in any one of claims 1 to 10.

Images