Humanoid robot control method based on image segmentation

Abstract

The embodiment of the application provides a kind of based on image segmentation humanoid robot control method.Belongs to the field of robot technology.The method comprises: obtaining image information about target object;The image information of the target object is segmented based on instance segmentation algorithm, and the target region corresponding to the target object is obtained;According to the image information, the point cloud information corresponding to the target region is obtained;According to the coordinate information in the point cloud information, the position information of the target object is calculated;The point cloud information is analyzed by principal component, and the direction information of the target object is obtained;According to the position information and direction information of the target object, the grasping control information of the end effector is determined;According to the grasping control information, the end effector is controlled to grasp the target object.The method can improve the calculation accuracy of the position information of the target object in complex scene, so as to improve the grasping success rate in the scene.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a humanoid robot control method based on image segmentation. Background Technology

[0002] As robotics technology evolves towards intelligence and anthropomorphism, humanoid robots, with their unique advantage of being highly adaptable to human life and work scenarios, have become one of the core tracks for industry innovation and development.

[0003] Currently, in applications where humanoid robots grasp objects, feature extraction and analysis are performed on image data collected by visual sensors to identify the target object's spatial location and then grasp it. However, this method suffers from poor boundary segmentation accuracy, especially in complex scenes such as those with partial occlusion, shadows, or missing textures. Occluded parts may not be recognized, leading to poor accuracy in calculating the target object's location and potentially causing the grasping task to fail. Summary of the Invention

[0004] This application provides a humanoid robot control method based on image segmentation, which can improve the accuracy of calculating the position information of target objects in complex scenes, thereby improving the success rate of grasping in such scenes.

[0005] In a first aspect, embodiments of this application provide a humanoid robot control method based on image segmentation. The humanoid robot includes an end effector. The method includes: acquiring image information about a target object; segmenting the image information of the target object based on an instance segmentation algorithm to obtain a target region corresponding to the target object; acquiring point cloud information corresponding to the target region based on the image information corresponding to the target region; calculating the position information of the target object based on the coordinate information in the point cloud information; performing principal component analysis on the point cloud information to obtain the orientation information of the target object; determining grasping control information of the end effector based on the position information and orientation information of the target object; and controlling the end effector to grasp the target object based on the grasping control information.

[0006] In one possible implementation, the image information of the target object is segmented based on an instance segmentation algorithm to obtain the target region corresponding to the target object, including: associating the target object with different frame images based on a temporal tracking algorithm; and segmenting the target object after association between different frames based on the instance segmentation algorithm to obtain the target region corresponding to the target object.

[0007] In one possible implementation, the position information of the target object is calculated based on the coordinate information in the point cloud information, including: obtaining the offset between the instance mask output by the instance segmentation algorithm and the true boundary of the target object; if the offset is greater than a preset distance threshold, determining the point coordinate adjustment weight of the corresponding region of the point cloud based on the pixel depth value of the instance mask; wherein the point coordinate adjustment weight is negatively correlated with the pixel depth value of the instance mask; adjusting the coordinates in the point cloud information according to the point coordinate adjustment weight of the corresponding region of the point cloud to obtain the adjusted point cloud coordinate information; and calculating the position information of the target object based on the adjusted point cloud coordinate information.

[0008] In one possible implementation, the grasping control information of the end effector is determined based on the position and orientation information of the target object, including: determining the grasping contact area based on the position information of the target object; determining the mask area corresponding to the grasping contact area based on the instance mask output by the instance segmentation algorithm; and determining the gripping force of the end effector based on the mask area corresponding to the grasping contact area, wherein the grasping control information includes the gripping force; and wherein the gripping force is positively correlated with the mask area corresponding to the grasping contact area.

[0009] In one possible implementation, the position information of the target object is calculated based on the coordinate information in the point cloud information, including: obtaining the weight distribution information corresponding to the target object; whether the weight distribution information is uniform or non-uniform; if the weight distribution information corresponding to the target object is uniform, then the position information of the target object is calculated based on the point cloud information corresponding to all parts included in the target object; if the weight distribution information corresponding to the target object is non-uniform, then the weight concentration region corresponding to the target object is determined; and the position information of the target object is calculated based on the coordinates of the points located within the weight concentration region.

[0010] In one possible implementation, the weight concentration region corresponding to the target object is determined; the position information of the target object is calculated based on the coordinates of the points located within the weight concentration region, including: obtaining the material distribution of the target object; determining the weight concentration region corresponding to the target object based on the material distribution of the target object; the weight concentration region is a continuous region with the largest weight corresponding to a single material within the target object; extracting the point cloud corresponding to the weight concentration region from the point cloud information; and using the average coordinates of all points in the point cloud corresponding to the weight concentration region as the position information of the target object.

[0011] In one possible implementation, the position information of the target object is calculated based on the coordinate information in the point cloud information, including: determining, based on the image information, whether the target object consists of at least two non-overlapping parts, and at least one of the non-overlapping parts does not have a grasping position; if the target object consists of at least two non-overlapping parts, and at least one of the non-overlapping parts does not have a grasping position, then extracting the point cloud information containing the region corresponding to the grasping position from the point cloud information corresponding to the target region; and calculating the position information of the target object based on the coordinate information in the point cloud information containing the region corresponding to the grasping position.

[0012] In one possible implementation, the location information of the target object is calculated based on the point cloud information containing the region corresponding to the capture location, including: using the average value of the coordinates of all points in the point cloud corresponding to the capture location as the location information of the target object.

[0013] In one possible implementation, the location information includes a center point; based on the location and orientation information of the target object, the grasping control information of the end effector is determined, including: determining the grasping contact area of ​​the end effector based on the location information of the target object; the grasping contact area can encompass the center point; determining the grasping posture of the end effector based on the orientation information of the target object; and generating grasping control information based on the grasping contact area and the grasping posture.

[0014] In one possible implementation, the instance segmentation algorithm is the YOLACT algorithm.

[0015] Secondly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0016] The memory stores instructions that the computer executes;

[0017] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0019] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0020] This application provides a humanoid robot control method based on image segmentation. The method segments the image information of the target object based on an instance segmentation algorithm to obtain the target region of the target object. This can improve the accuracy of calculating the position information of the target object in complex scenes, thereby improving the success rate of grasping in such complex scenes. Furthermore, since only the point cloud information corresponding to the target region needs to be processed, the amount of point cloud information that needs to be processed is reduced, thus improving data processing efficiency. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 A schematic diagram of a scene for the humanoid robot control method based on image segmentation provided in this application;

[0023] Figure 2 A flowchart illustrating the humanoid robot control method based on image segmentation provided in this application embodiment;

[0024] Figure 3 This is a schematic diagram of the target region segmentation process provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the location information determination process provided in the embodiments of this application;

[0026] Figure 5 This is a schematic diagram of the process for generating capture control information provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the point cloud capture process provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] In the field of humanoid robots, object detection algorithms are the mainstream technology for object localization. They obtain the position of objects by analyzing image data from visual sensors, and then grasp the objects. However, the current object localization process requires processing a large amount of data, consumes a lot of computing resources, and has low data processing efficiency.

[0032] To address the aforementioned technical problems, the inventors propose the following technical concept: acquiring target point cloud information corresponding to the target object, performing instance segmentation on the target point cloud information to obtain the target region corresponding to the target object, using the point cloud information within the target region to determine the position and orientation information of the target object, determining the grasping control information based on the position and orientation information, and using the grasping control information to control the end effector to grasp the target object.

[0033] Figure 1 This is a schematic diagram illustrating a scenario for the humanoid robot control method based on image segmentation provided in this application. Figure 1 The scenario includes: a humanoid robot 10 and a target object 20 to be grasped.

[0034] The humanoid robot 10 includes an image acquisition device 101, at least one end effector 102, and a control unit 103. The image acquisition device 101 may include a depth camera, a vision sensor, etc. The end effector 102 may be a parallel gripper, a vertical gripper, a finger gripper, a dexterous hand, etc. The target object 20 includes the object to be grasped.

[0035] The control unit 103 is used to acquire image information of the current scene acquired by the image acquisition device 101. The image information acquired by the image acquisition device may include a two-dimensional RGB image and a three-dimensional depth map, where the depth map is the distance information of each pixel. The depth map can then be directly converted into point cloud information based on the SDK or hardware processing module built into the image acquisition device. The position and orientation of the target object 20 are determined based on the point cloud information, and the end effector 102 is controlled to grasp the target object 20 based on its position and orientation.

[0036] It is understood that the scenarios illustrated in the embodiments of this application do not constitute a specific limitation on the humanoid robot control method based on image segmentation. In other feasible embodiments of this application, the above scenarios may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and are not limited here. Figure 1 The scenario shown can be implemented by hardware, software, or a combination of both.

[0037] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0038] Figure 2 This is a flowchart illustrating the humanoid robot control method based on image segmentation provided in an embodiment of this application. The execution entity of this embodiment can be... Figure 1 The humanoid robot as a whole or the control unit within a humanoid robot. For example... Figure 2 As shown, the method includes steps S201 to S207.

[0039] S201: Obtain image information about the target object.

[0040] In this step, an image acquisition device can be used to periodically or continuously capture images of the target object to obtain image information.

[0041] The image information may include a single image, N images, or a new image obtained by combining N images, where N is a positive integer.

[0042] S202: Segment the image information of the target object based on the instance segmentation algorithm to obtain the target region corresponding to the target object.

[0043] In this step, the instance segmentation algorithm may include the Mask R-CNN model, the SOLO (Single-stage instance segmentation) model, or the YOLOCAT algorithm. By segmenting the image information of the target object using an instance segmentation algorithm, each independent object in the point cloud can be distinguished, thereby obtaining the category of the target object and the corresponding target region in the image information. The target region corresponding to the target object is typically defined by extracting the precise boundaries of the target object.

[0044] S203: Obtain the point cloud information corresponding to the target area based on the image information corresponding to the target area.

[0045] In this step, the point cloud information corresponding to the target area in the image information can be read directly.

[0046] Point cloud is a data set composed of a large number of discrete points in three-dimensional space. Point cloud information includes multiple points and the coordinate information corresponding to each point. It may also contain other information (such as color, intensity, normal vector, etc.).

[0047] S204: Calculate the position information of the target object based on the coordinate information in the point cloud information corresponding to the target area.

[0048] In this step, the average coordinates of all points in the point cloud information corresponding to the target area can be directly calculated as the center point of the target object, which is also the position information of the target object.

[0049] S205: Perform principal component analysis on the point cloud information to obtain the orientation information of the target object.

[0050] In this step, the process of performing principal component analysis on the point cloud information may include:

[0051] First, the average coordinates of all points in the target point cloud are taken as the coordinates of the centroid, which are expressed as: , The x-axis coordinates of the centroid are represented. This represents the y-axis coordinate corresponding to the centroid. This represents the z-axis coordinate corresponding to the centroid. That is, the centroid of the target point cloud is: The centroid coordinates serve as the position information of the target object, where N is the total number of points in the target point cloud. These are the three-dimensional coordinates of the i-th point. This represents the x-coordinate of the i-th point. This represents the y-coordinate of the i-th point. This represents the z-axis coordinate of the i-th point. Then, the z-axis coordinate of each point in the point cloud is calculated. Relative to the center of mass The deviation of the coordinates is calculated by subtracting the coordinate values ​​in each of the three directions respectively. The covariance matrix C of the point cloud data is then constructed.

[0052]

[0053] in, Represents the coordinates corresponding to the i-th point The resulting matrix is ​​3 rows and 1 column, that is, its size is 3×1. For example, if If the matrix is ​​(1,2,3), then the corresponding transformed matrix is... That is . Indicates the coordinates of the centroid The matrix formed by the transformation. This represents the transpose of the corresponding matrix. C is a 3×3 covariance matrix. The eigenvalues ​​are obtained by performing eigenvalue decomposition on the covariance matrix C. and the corresponding feature vector . and One-to-one correspondence. The larger the eigenvalue, the more important the direction of the eigenvector. Select the first eigenvector corresponding to the largest eigenvalue among the three eigenvalues. The second eigenvector corresponds to the second largest eigenvalue, which represents the object's principal direction (major axis). As the secondary direction (minor axis), then, based on the right-hand rule, the third eigenvector is obtained by multiplying the first and second eigenvectors. , that is , Represents the first eigenvector. This represents the second eigenvector. for one of the, for one of the.

[0054] S206: Determine the grasping control information of the end effector based on the position and orientation information of the target object.

[0055] In this step, the coordinates and orientation information (pitch angle, azimuth angle) of the target object are written into the grasping control information template to obtain grasping control information. It also includes using position and orientation information to determine the gripper opening degree, grasping point position, attitude, movement distance, movement direction, movement trajectory, rotation angle, etc., of the end effector, and writing these parameters into the grasping control information template to obtain grasping control information.

[0056] The center point of the aforementioned target object is located inside the area enclosed by the grasping posture of the end effector.

[0057] Specifically, after obtaining the target object's position and orientation information, a grasping strategy and grasping points can be determined. The grasping strategy can be determined based on task requirements and object attributes. For example, the finger spread and gripping force can be determined based on the object's size, weight, and material. Regarding grasping points, they can be generated based on offline-generated grasping templates, or their pose can be calculated in real-time using neural networks or optimization algorithms.

[0058] S207: Control the end effector to grasp the target object based on the grasping control information.

[0059] This step includes sending grasping control information to the end effector so that the end effector grasps the target object, or sending grasping control information to the controller of the end effector so that the controller of the end effector controls the end effector to grasp the target object.

[0060] As can be seen from the description of the above embodiments, the embodiments of this disclosure segment the image information of the target object based on the instance segmentation algorithm to obtain the target region of the target object, which can improve the accuracy of the calculation of the position information of the target object in complex scenes, thereby improving the capture success rate in such complex scenes; and since only the point cloud information corresponding to the target region needs to be processed, the amount of point cloud information that needs to be processed is reduced, thereby improving data processing efficiency.

[0061] Figure 3 This is a schematic diagram illustrating the target region segmentation process provided in an embodiment of this application. Figure 3 As shown, in some optional embodiments, based on any embodiment, step S202 includes: step S2021 and step S2022.

[0062] S2021, based on the time-series tracking algorithm, performs target object association on different frame images.

[0063] S2022, the target object associated between different frames is segmented based on the instance segmentation algorithm to obtain the target region corresponding to the target object.

[0064] For example, the aforementioned temporal tracking algorithm could be DeepSORT. This embodiment can associate objects between different images, avoid segmentation jitter, achieve temporal consistency, and improve the accuracy of the final determined target object's location information, thereby increasing the success rate of humanoid robots grasping objects.

[0065] In some alternative embodiments, based on any embodiment, step S202 includes:

[0066] The connectivity of the instance mask output by the instance segmentation algorithm is checked. If a disconnection is found, the side-view camera of the humanoid robot is controlled to acquire a second image of the target object. The instance mask results obtained from the segmentation of the two images are fused together to obtain the final output of the instance segmentation algorithm.

[0067] This embodiment can solve the problems of inaccurate masking and inaccurate calculation of target object location information caused by missing regions, thus ensuring the success rate and stability of the crawling process.

[0068] Figure 4This is a schematic diagram illustrating the location information determination process provided in an embodiment of this application. Figure 4 As shown, in some optional embodiments, based on any embodiment, step S204 includes steps S2041 to S2044.

[0069] S2041, Obtain the offset between the instance mask output by the instance segmentation algorithm and the actual boundary of the target object.

[0070] S2042, if the offset is greater than a preset distance threshold, determine the point coordinate adjustment weight for the corresponding region of the point cloud based on the instance mask pixel depth value. The point coordinate adjustment weight is negatively correlated with the instance mask pixel depth value. That is, the larger the instance mask pixel depth value, the smaller the point coordinate adjustment weight; the smaller the instance mask pixel depth value, the larger the point coordinate adjustment weight.

[0071] S2043, adjust the coordinates in the point cloud information according to the weights of the point coordinates in the corresponding region of the point cloud, and obtain the adjusted point cloud coordinate information. That is, multiply the coordinates in the point cloud information by the weights of the point coordinates to obtain the adjusted point cloud coordinate information.

[0072] S2044: Calculate the position information of the target object based on the adjusted point cloud coordinate information.

[0073] In step S2041 above, the comparison can be based on the coordinate offset between the mask edge and the real contour, or it can be based on the offset between the center point of the instance mask and the center point of the real boundary of the target object. Based on the above embodiment, when the depth value is large, i.e., the object surface is far from the camera, the impact on the center of gravity calculation is reduced after weighting; when the depth value is small, i.e., the object surface is close to the camera, some parts of the object surface may be missed in segmentation, and the missing areas can be partially compensated after weighting. The boundary offset problem is solved, which can avoid the problem of the grasping point deviating from the center of gravity of the object, resulting in slippage, and ensure the grasping success rate and grasping stability.

[0074] Figure 5 This is a schematic diagram of the process for generating capture control information provided in an embodiment of this application. In some optional embodiments, based on any embodiment, step S206 includes: steps S2061 to S2063.

[0075] S2061, Determine the grasping contact area based on the position information of the target object. Specifically, the position information of the target object can be used as the input parameter of the neural network model to calculate several grasping points. The continuous area on the object surface corresponding to the arc formed by connecting these grasping points is the aforementioned grasping contact area.

[0076] S2062, Based on the instance mask output by the instance segmentation algorithm, determine the mask area corresponding to the grasping contact area. That is, the area of ​​the mask region in the instance mask corresponding to the grasping contact area is the mask area.

[0077] S2063, determine the gripping force of the end effector based on the mask area corresponding to the gripping contact area. The gripping control information includes the gripping force. The gripping force is positively correlated with the mask area corresponding to the gripping contact area. That is, the smaller the mask area, the smaller the gripping force; the larger the mask area, the greater the gripping force.

[0078] In this embodiment, for example, a compliant gripping mode can be used for masks with smaller areas, resulting in a smaller gripping force. For masks with larger areas, a high-rigidity gripping mode can be used, resulting in a larger gripping force. This embodiment can solve the problem of insufficient contact surface of the gripper due to missing areas during the segmentation process. It can adjust the gripping force according to the mask area, so that the gripping force matches the contact area, ensuring the success rate and stability of the humanoid robot in grasping objects.

[0079] In one possible implementation, step S204, calculating the position information of the target object based on the coordinate information in the point cloud information, includes steps S204B1 to S204B3.

[0080] S204B1: Obtain the weight distribution information corresponding to the target object. The weight distribution information is either uniform or non-uniform.

[0081] This step may include determining the weight distribution information of the target object based on the density of points in the point cloud information. It may also include dividing the space containing the point cloud into multiple voxels (pixels in three-dimensional space), counting the number of points within each voxel as the density of points within the voxel, calculating the mean and variance of the point density, and determining whether the weight distribution is uniform or non-uniform based on the mean and variance. Alternatively, the uniformity of the weight distribution can be determined based on X-ray equipment or image information.

[0082] For example, if the density of some points in the point cloud is concentrated while the density of others is dispersed, then the weight distribution is determined to be uneven. As another example, if the variance of the density is less than a preset variance threshold, then the weight distribution is determined to be uniform; if the variance of the density is greater than or equal to the preset variance threshold, then the weight distribution is determined to be uneven.

[0083] S204B2: If the weight distribution information corresponding to the target object is uniform, then the position information of the target object is calculated based on the point cloud information corresponding to all parts included in the target object.

[0084] In this step, assuming uniform weight distribution, the average value of the coordinates of all point cloud information of the target object is calculated to obtain the position information of the target object.

[0085] S204B3: If the weight distribution information corresponding to the target object is uneven, then the weight concentration region corresponding to the target object is determined. Based on the coordinates of points located within the aforementioned weight concentration region, the position information of the target object is calculated. In this embodiment, the weight concentration region is a continuous region within the target object where the weight of a single substance is the largest.

[0086] This step includes identifying voxels whose number of points within a voxel is greater than the average density as target voxels, combining the target voxels to obtain a weight concentration region, and using the average coordinates of the points located within the weight concentration region as the location information of the target object.

[0087] As described in the above embodiments, in a first aspect, the embodiments of this disclosure determine whether the target object has a uniform weight distribution, and in the case of non-uniformity, determine a weight concentration area. Then, they use point cloud clusters of the weight concentration area to determine the target object's position information, thereby positioning the location information closer to the weight concentration area. Consequently, the subsequently generated gripping point will be closer to the weight concentration area, making the humanoid robot more stable during the gripping process. For example, when gripping a mineral water bottle, if the gripping position is located in a non-weight concentration area, i.e., a hollow area, the bottle may sway left and right in the air after gripping, meaning the bottle is in an unstable state, which could lead to the bottle falling and the task failing. This is especially true for end effectors that are parallel grippers or two-point grippers, where the likelihood of failure is even greater.

[0088] Secondly, if the centroid of the target object is calculated from all points corresponding to all parts of the target object, it may deviate significantly from the actual grasping position. Consequently, the grasping position calculated based on the target object's position may not match the actual grasping position, leading to grasping task failure. Therefore, in this embodiment, when the weight distribution is uneven, only the average coordinates of discrete points located within the weight concentration area are used as the target object's position information, instead of calculating the center point coordinates based on the coordinates of all discrete points in the target object and using this to determine the grasping point. This makes the final grasping point coordinates more consistent with the needs of the humanoid robot grasping scenario, and the grasping point more in line with actual operational requirements. This makes the calculated grasping point coordinates more accurate for this scenario, thereby improving the success rate of the humanoid robot grasping the target object and increasing the efficiency of target object position information calculation, thus improving task efficiency.

[0089] Figure 6This is a schematic diagram of the point cloud capture process provided in an embodiment of this application. Figure 6 As shown, in some embodiments, step S204B3 includes:

[0090] S204B31, If ​​the weight distribution information corresponding to the target object is uneven, then obtain the material distribution of the target object.

[0091] S204B32, Based on the material distribution of the target object, determine the weight concentration region corresponding to the target object. The weight concentration region is the continuous region within the target object where the weight of a single substance is the largest.

[0092] S204B33, extract the point cloud corresponding to the weight concentration region from the point cloud information.

[0093] And S204B34, which uses the average coordinates of all points in the point cloud corresponding to the weight concentration region as the location information of the target object.

[0094] In this embodiment, the material distribution of the target object can be determined by dividing the space where the target object is located into voxels, as described in step S204B1 above. Alternatively, the material distribution can be determined based on X-ray equipment or image information. Using X-ray equipment or image information, the distribution of various types of materials in each region of the target object (including material category, distribution area of ​​each material, and volume of each distribution area) can be identified. Based on the density and volume of different materials, the weight of each type of material can be calculated, thus determining the weight concentration area. Then, point cloud cropping can be performed based on the coordinate range of the points corresponding to the weight concentration area.

[0095] As can be seen from the description of the above embodiments, the embodiments of this disclosure are for objects with uneven weight distribution. The target point coordinates are calculated only based on the point cloud information of the weight concentration area. The target point coordinates can be used as the coordinates of the grasping point, instead of calculating the center point coordinates based on the coordinates of all discrete points in the target object. On the one hand, this makes the final target point coordinates more consistent with the grasping scenario of this special type of object, improves the accuracy of the position information calculation of this type of object, and thus helps to improve the success rate of humanoid robots grasping this type of object.

[0096] On the other hand, by making the positioning target point closer to the actual object of the target, the grasping point is also closer to the actual object. This makes the humanoid robot more stable during the grasping process and avoids the humanoid robot grasping point being located in an area where the object is not located, which would cause the center of gravity of the object to be unstable after grasping, resulting in left and right swaying and thus causing the task to fail.

[0097] In one possible implementation, step S203 above, obtaining point cloud information corresponding to the target area based on image information, includes steps S2031 to S2033.

[0098] S2031: Obtain point cloud information from the image.

[0099] This step includes reading point cloud information from image information.

[0100] S2032: Based on preset camera parameters, project each point of the point cloud information of the image information onto a two-dimensional plane to obtain the pixel coordinates of each point.

[0101] In this step, camera parameters such as focal length... , Optical center coordinates , The intrinsic parameter matrix constructed using the above camera parameter example is as follows:

[0102]

[0103] The intrinsic parameter matrix is ​​a parameter matrix used to describe the camera. It includes information such as the camera's focal length and optical center coordinates, and can convert the image coordinates acquired by the camera into real-world coordinates in the camera coordinate system. In this embodiment, the camera is a depth camera. Using the image and depth information obtained by the depth camera, point cloud information in the scene can be obtained through a 3D point cloud reconstruction algorithm.

[0104] Using an intrinsic parameter matrix, each point in the point cloud information can be projected onto a two-dimensional plane to obtain the pixel coordinates of each point.

[0105] S2033: If the pixel coordinates of the target point are not within the target area, remove the target point and obtain the point cloud information corresponding to the target area.

[0106] In this step, if the x-coordinate of a pixel is greater than the maximum x-coordinate of the target area but less than the minimum x-coordinate of the target area, the target point is removed; if the y-coordinate of a pixel is greater than the maximum y-coordinate of the target area but less than the minimum y-coordinate of the target area, the target point is removed, and only discrete points located within the target area are retained. The remaining points are the point cloud information corresponding to the target area.

[0107] As can be seen from the description of the above embodiments, the present disclosure embodiments combine camera parameters to project point cloud information onto a two-dimensional plane and remove points that do not belong to the target object to obtain point cloud information of the target area.

[0108] In some alternative embodiments, in the above Figure 2 Based on the corresponding embodiment, step S204 includes:

[0109] S2045, Based on the image information, determine whether the target object consists of at least two non-overlapping parts, and at least one of the non-overlapping parts does not have a grab position.

[0110] In this step, the target object category is determined based on an instance segmentation algorithm or any object recognition model. Then, combined with a pre-set object category feature library, it is determined whether the target object category contains a dedicated grasping position. If a dedicated grasping position is contained, a grasp detection model (such as GraspNet or GG-CNN) is used to identify whether each part of the target object contains a grasping position. The aforementioned pre-set object category feature library records the features of whether objects of each category have grasping positions.

[0111] For example, a glass water cup consists of a cup body and a handle, the orthographic projections of which do not overlap. The handle has a gripping position, while the cup body does not.

[0112] S2046 If the target object consists of at least two non-overlapping parts, and at least one of the non-overlapping parts does not have a grasping position, then extract the point cloud information containing the continuous region corresponding to the grasping position from the point cloud information corresponding to the target region.

[0113] This step may include determining the region corresponding to the capture position in the image information, removing points outside the region corresponding to the capture position in the point cloud information, and obtaining the point cloud information corresponding to the capture position.

[0114] S2047, calculate the position information of the target object based on the coordinate information in the point cloud information containing the continuous area corresponding to the capture position.

[0115] In this step, the average value of the coordinates of all points in the point cloud corresponding to the capture location will be used as the location information of the target object.

[0116] As described in the above embodiments, this disclosure is designed for a specific type of target object, namely, an object composed of at least two non-overlapping parts, where at least one of the non-overlapping parts does not have a grasping position. For this type of target object, if the centroid of the target object is calculated from all the points corresponding to all the parts of the target object, the centroid may deviate significantly from the actual grasping position. Consequently, the grasping position calculated from the target object's position may not match the actual grasping position, leading to grasping task failure. In this embodiment, when the target object meets the above-mentioned type conditions based on image information, the position information of the target object is determined only from the point cloud information containing the region corresponding to the grasping position, rather than calculating the center point coordinates based on the coordinates of all discrete points in the target object. This improves the accuracy of the grasping position calculation for this special type of target object, thereby increasing the success rate of the humanoid robot grasping this special type of target object. On the other hand, by calculating the position information of the target object only from a portion of the point cloud, computational efficiency can be improved, thereby increasing the operational efficiency of the humanoid robot.

[0117] In one possible implementation, the image information is acquired periodically. Step S201 is replaced by acquiring at least three sets of consecutive image information about the target object. Step S204 includes steps S240 and S241.

[0118] S240: Determine whether the target object is moving based on at least three sets of image information.

[0119] In this step, the methods of steps S202 to S204 above can be used to determine three location information based on three sets of image information. If the difference between the three location information is greater than a preset difference threshold, it is determined whether the target object is a dynamically moving object.

[0120] A set of images can consist of multiple images, a single image, or a combination of multiple images. The step of determining whether the target object is a dynamically moving object can be executed by the aforementioned processing unit. For example, if all images taken from the same shooting angle are identical, it indicates that the target object has not moved, i.e., it is a static object. Otherwise, it indicates that it is moving, i.e., a dynamic object.

[0121] S241: If the target object is moving, predict the position information of the target object in the next cycle based on at least three sets of image information, and determine the position information of the target object in the next cycle as the position information of the target object.

[0122] If the target object does not move, continue with steps S202 to S207.

[0123] This step may include determining location information using each set of image information, calculating the target object's moving speed based on the image acquisition period and location information, and predicting the target object's location information at the next moment based on the location information and moving speed. Alternatively, the target object's acceleration may be calculated based on this, and the velocity between the current moment and the next image acquisition moment may be determined by combining the acceleration, location information, and moving speed. Finally, the location information at the next image acquisition moment may be determined by combining the location information and the time interval between image acquisitions.

[0124] Regarding the orientation information of dynamic objects, multiple sets of orientation information can be calculated for each set of images based on step S205. Then, based on these multiple sets of orientation information, information such as angular velocity and angular acceleration during movement can be calculated, and the orientation information of the target object at the next moment can be predicted. Alternatively, the multiple sets of orientation information can be converted into rotation matrices, the translation vector can be solved by position changes, and the angular velocity can be estimated by changes in the rotation matrices, thereby deriving the three-dimensional linear velocity and angular velocity of the target object at the next moment.

[0125] As can be seen from the description of the above embodiments, the embodiments of this disclosure determine whether the target object is actually a dynamically moving object by combining at least three sets of images. This can accurately determine whether the target object is moving. Different position information calculation methods and direction information calculation methods are adopted for static objects and dynamic objects respectively. For the position information of dynamic objects, parameters such as speed and acceleration can be calculated based on multiple sets of image information, thereby making the determined new position information more accurate and increasing the success rate of humanoid robots in grasping target objects.

[0126] In one possible implementation, if the target object is in a static state after step S240, then steps S202 to S206 are continued.

[0127] In one possible implementation, the location information includes a center point. Step S206 above, determining the grasping control information of the end effector based on the target object's location and orientation information, includes steps S2064 to S2066.

[0128] S2064: Determine the gripping contact area of ​​the end effector based on the position information of the target object. The gripping contact area can cover the center point.

[0129] Specifically, the location information of the target object can be used as input parameters for the neural network model to calculate several grasping points. The continuous area on the object's surface corresponding to the arc formed by connecting these grasping points is the aforementioned grasping contact area. The grasping contact area can encompass the center point because the arc surface containing the grasping contact area can form a corresponding virtual closed shape. In other words, the arc surface containing the grasping contact area is considered part of a virtual closed shape, such as a circle or ellipse, and this virtual closed shape can encompass the center point.

[0130] S2065: Determine the grasping posture of the end effector based on the orientation information of the target object.

[0131] This step includes using the gripping angle corresponding to the orientation information, the rotation direction of the end effector, etc. The gripping posture of the end effector is matched with the orientation of the target object.

[0132] S2066: Generate gripping control information based on the gripping contact area and gripping posture.

[0133] In this step, the gripping contact area and gripping posture can be input into the gripping control information template to obtain gripping control information. Alternatively, an appropriate end effector type can be selected based on the gripping contact area to generate gripping control information corresponding to the end effector type and gripping posture.

[0134] As can be seen from the description of the above embodiments, the embodiments of this disclosure determine the grasping contact area by using position information, determine the grasping posture by using direction information, and generate grasping control information corresponding to the grasping contact area and grasping posture, so that the humanoid robot can stably grasp the target object.

[0135] In one possible implementation, the instance segmentation algorithm is the YOLACT algorithm. The idea behind the YOLACT algorithm is to decompose the instance segmentation task into two parallel subtasks:

[0136] Generate prototype masks: Predict a set of basic masks using a fully convolutional network (similar to semantic segmentation).

[0137] Predictive Mask Coefficients: Generate a set of linear combination coefficients for each detection instance, combining the prototype mask into the final instance mask.

[0138] YOLACT achieves real-time instance segmentation through a clever design of prototype mask and coefficient combination, balancing speed and accuracy, and is especially suitable for real-time computing scenarios such as humanoid robots performing grasping tasks.

[0139] By employing the YOLACT algorithm for instance segmentation, high-precision instance segmentation can be achieved. Furthermore, the YOLACT algorithm's adaptability to different types of objects and complex backgrounds increases the accuracy of instance segmentation. It should be noted that all the embodiments disclosed in this application can be freely combined, and the resulting technical solutions are also within the protection scope of this application.

[0140] To implement the above embodiments, this application also provides an electronic device.

[0141] refer to Figure 7 It shows a structural schematic diagram of an electronic device 700 suitable for implementing the embodiments of this application. The electronic device 700 can be a humanoid robot of any of the above embodiments. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0142] like Figure 7 As shown, the electronic device 700 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 701 and a memory 702 communicatively connected to the processor. The processor can perform various appropriate actions and processes based on programs stored in the memory 702, computer-executed instructions, or programs loaded from the storage device 708 into the random access memory (RAM) 703, thereby implementing the image segmentation-based humanoid robot control method in any of the above embodiments. The memory may be a read-only memory (ROM). The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing device 701, the memory 702, and the RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0143] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, speakers; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0144] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from memory 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.

[0145] It should be noted that the computer-readable storage medium described above in this application can be a computer-readable signal medium, a computer storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0146] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0147] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0151] This application also provides a computer-readable storage medium storing computer-executable instructions. When the processor executes the computer-executable instructions, it implements the technical solution of the humanoid robot control method based on image segmentation in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the humanoid robot control method based on image segmentation. Please refer to the implementation principle and beneficial effects of the humanoid robot control method based on image segmentation, which will not be repeated here.

[0152] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the humanoid robot control method based on image segmentation in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the humanoid robot control method based on image segmentation, and can be found in the implementation principle and beneficial effects of the humanoid robot control method based on image segmentation, which will not be repeated here.

[0154] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0155] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0156] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A humanoid robot control method based on image segmentation, characterized in that, The humanoid robot includes an end effector, and the method includes: Obtain image information about the target object; The image information of the target object is segmented based on the instance segmentation algorithm to obtain the target region corresponding to the target object; Based on the image information corresponding to the target region, obtain the point cloud information corresponding to the target region; The position information of the target object is calculated based on the coordinate information in the point cloud information; Principal component analysis is performed on the point cloud information to obtain the orientation information of the target object; Based on the position and orientation information of the target object, the grasping control information of the end effector is determined; The end effector is controlled to grasp the target object according to the grasping control information; The step of calculating the position information of the target object based on the coordinate information in the point cloud information includes: obtaining the offset between the instance mask output by the instance segmentation algorithm and the real boundary of the target object; if the offset is greater than a preset distance threshold, determining the point coordinate adjustment weight of the corresponding region of the point cloud based on the pixel depth value of the instance mask; wherein the point coordinate adjustment weight is negatively correlated with the pixel depth value of the instance mask; adjusting the coordinates in the point cloud information according to the point coordinate adjustment weight of the corresponding region of the point cloud to obtain the adjusted point cloud coordinate information; and calculating the position information of the target object based on the adjusted point cloud coordinate information.

2. The method according to claim 1, characterized in that, The instance-based segmentation algorithm segments the image information of the target object to obtain the target region corresponding to the target object, including: Target object association is performed on different frame images based on a time-series tracking algorithm; For target objects associated between different frames, segmentation is performed based on an instance segmentation algorithm to obtain the target region corresponding to the target object.

3. The method according to claim 1, characterized in that, The step of determining the grasping control information of the end effector based on the position and orientation information of the target object includes: The grasping contact area is determined based on the location information of the target object; Based on the instance mask output by the instance segmentation algorithm, determine the mask area corresponding to the grasping contact area; The gripping force of the end effector is determined based on the mask area corresponding to the gripping contact area, and the gripping control information includes the gripping force; wherein the gripping force is positively correlated with the mask area corresponding to the gripping contact area.

4. The method according to claim 1, characterized in that, The step of calculating the position information of the target object based on the coordinate information in the point cloud information includes: Obtain the weight distribution information corresponding to the target object; the weight distribution information is either uniform or non-uniform. If the weight distribution information corresponding to the target object is uniform, then the position information of the target object is calculated based on the point cloud information corresponding to all parts included in the target object; If the weight distribution information corresponding to the target object is uneven, then the weight concentration area corresponding to the target object is determined; based on the coordinates of the points located within the weight concentration area, the position information of the target object is calculated.

5. The method according to claim 4, characterized in that, The weight concentration region corresponding to the target object is determined; Based on the coordinates of points located within the weight concentration region, the position information of the target object is calculated, including: Obtain the material distribution of the target object; Based on the material distribution of the target object, the weight concentration region corresponding to the target object is determined; the weight concentration region is a continuous region with the largest weight corresponding to a single substance within the target object; Extract the point cloud corresponding to the weight concentration region from the point cloud information; The average coordinates of all points in the point cloud corresponding to the weight concentration region are used as the location information of the target object.

6. The method according to claim 1, characterized in that, The step of calculating the position information of the target object based on the coordinate information in the point cloud information includes: Based on the image information, determine whether the target object is composed of at least two non-overlapping parts, and whether at least one of the non-overlapping parts has a grasping position; If the target object consists of at least two non-overlapping parts, and at least one of the non-overlapping parts does not have a grasping position, then extract the point cloud information containing the region corresponding to the grasping position from the point cloud information corresponding to the target region. The position information of the target object is calculated based on the coordinate information in the point cloud information containing the region corresponding to the grasping position.

7. The method according to claim 6, characterized in that, The step of calculating the location information of the target object based on the point cloud information containing the region corresponding to the grasping location includes: The average value of the coordinates of all points in the point cloud corresponding to the capture location is used as the location information of the target object.

8. The method according to any one of claims 1 to 7, characterized in that, The location information includes the center point; The step of determining the grasping control information of the end effector based on the position and orientation information of the target object includes: Based on the location information of the target object, the grasping contact area of ​​the end effector is determined; the grasping contact area can encompass the center point; The grasping posture of the end effector is determined based on the orientation information of the target object; The grasping control information is generated based on the grasping contact area and the grasping posture.

9. The method according to any one of claims 1 to 7, characterized in that, The instance segmentation algorithm is the YOLACT algorithm.

Images