A method and system for folding clothes by a two-armed humanoid robot based on clothing pose estimation

Abstract

The present invention discloses a method and system for a two-armed humanoid robot to fold clothes based on clothing pose estimation. The method includes: obtaining a clothing image and segmenting it; extracting the clothing contour from the segmented image; fitting a simplified polygon to the extracted clothing contour graph to obtain a simplified fitted polygon; using a Gaussian model for modeling to obtain a clothing polygon model database; the polygon model corresponding to the clothing in the current image represents the pose corresponding to the current clothing; according to the pose corresponding to the current clothing and combining specific clothing folding operation steps, obtaining the next pose in sequence; updating the parameters of the next pose to obtain an updated pose; the two-armed robot uses an obstacle avoidance motion planning algorithm to complete the obstacle avoidance motion planning from the pose corresponding to the current clothing to the updated pose. The present invention performs robot operation planning and control according to the clothing pose.

Description

Technical Field

[0001] The present invention relates to the field of intelligent service robot operations, and particularly to a method and system for folding clothes by a two-armed humanoid robot based on clothing pose estimation. Background Art

[0002] With the development of technology, more and more robots have begun to enter people's production and living scenarios. Among them, two-armed humanoid service robots have received increasing attention because they can bring many conveniences to people's lives. For example, the robot can autonomously complete the folding and sorting of clothes. However, the operation of clothes is still a huge challenge for robots. The main reasons are as follows:

[0003] First, clothes have an infinite number of degrees of freedom, so it is difficult to characterize the current pose of the clothes;

[0004] Second, the dynamic / kinematic characteristics of clothes are extremely complex and difficult to model, so it is difficult to predict the pose of the clothes after operation;

[0005] Third, for a humanoid robot, the process of folding clothes is a continuous process with multiple grasping points and multiple steps, and it is necessary to plan each operation step in advance. Therefore, the folding algorithm requirements for the robot are relatively high.

[0006] Currently, there are mainly two solutions for the folding operation of robots. One is to specifically perform motion planning and program design for a specific type of robot and the corresponding types of clothes folding steps. However, this method does not model the pose of the clothes and does not plan the next operation according to the real-time pose of the clothes, so the versatility is poor. Changing a type of robot or clothes requires re-performing motion planning and program design; the other is to use machine learning to collect as many pose image data of clothes as possible to make up for the lack of the physics model of clothes. Generally, it is the "target-condition" paradigm, that is, a model is established through big data training. The input of this model is the image seen by the current robot, and the output is the robot motion trajectory corresponding to the current image. However, this method requires a large number of clothes pose images as training data, and since clothes have an infinite number of degrees of freedom, it is very difficult to collect images in the real environment. At the same time, the training process of the model also heavily depends on the hardware configuration of the computer and takes a long time.

[0007] Therefore, it is crucial to develop an efficient method for accurately estimating the pose of clothes for a two-armed humanoid robot to fold clothes. Summary of the Invention

[0008] The main objective of the present invention is to provide a method and system for folding clothes by a two-armed humanoid robot based on clothing pose estimation, so as to solve the problem that the existing technology of humanoid robots cannot accurately identify the pose during the clothing folding process and perform robot operation planning and control according to the clothing pose.

[0009] To achieve the above objective, the present invention adopts the following technical solutions.

[0010] A method for folding clothes by a two-armed humanoid robot based on clothing pose estimation includes the following specific steps:

[0011] 1) Obtain a clothing image, segment the clothing foreground and the desktop background in the clothing image to obtain a segmented image;

[0012] 2) Extract the clothing contour from the segmented image to obtain the contour graph of the clothing. The contour graph of the clothing has 500 to 5000 boundary points;

[0013] 3) Fit a simplified polygon to the contour graph of the clothing to obtain a simplified fitted polygon. The fitted polygon has 5 to 30 boundary points;

[0014] 4) According to the geometric characteristics of the clothing contour polygon, use the Gaussian model to model different types of clothing and clothing folding operation steps to obtain a clothing polygon model database;

[0015] 5) Specify specific clothing folding operation steps from the clothing polygon model database according to user requirements;

[0016] 6) Match the simplified fitted polygon obtained in step 3) with the specific clothing folding operation steps obtained in step 5) to obtain the polygon model corresponding to the clothing in the current image. The polygon model corresponding to the clothing in the current image represents the pose corresponding to the current clothing;

[0017] 7) According to the pose corresponding to the current clothing obtained in step 6) and the specific clothing folding operation steps obtained in step 5), obtain the pose corresponding to the next clothing folding operation step in sequence;

[0018] 8) Update the parameters of the pose corresponding to the next clothing folding operation step obtained in step 7) to obtain an updated pose;

[0019] 9) The two-armed robot uses an obstacle avoidance motion planning algorithm to complete the obstacle avoidance motion planning from the pose corresponding to the current clothing in step 6) to the updated pose in step 8).

[0020] In step 2), the Moore algorithm is used to extract the clothing contour from the segmented image.

[0021] In step 3), the Rame-Douglas-Peucker algorithm is used to fit a simplified polygon to the contour graph of the clothing.

[0022] In step 4), according to the geometric features of the clothing contour polygon, a Gaussian model is used to model different types of clothing and the clothing folding operation steps, and a clothing polygon model database is obtained, specifically including:

[0023] 4.1) For the outer contour polygon of the same type of clothing, the initial Gaussian model is trained by learning three geometric features of the known clothing contour polygon to obtain a trained Gaussian model, and the Gaussian model modeling is completed;

[0024] 4.2) Deleting or adding vertices of the trained Gaussian model to form the next polygon model;

[0025] 4.3) Summarizing the trained Gaussian model and the next polygon model to form a clothing polygon model database.

[0026] In step 4.1), the three geometric features are the distance between adjacent vertices of the simplified fitting polygon corresponding to the known clothing contour polygon, the ratio of the length of the line segment between vertices to the perimeter of the entire polygon, and the angle size of each vertex-corresponding angle.

[0027] In step 6), the simplified fitting polygon obtained in step 3) is matched with the specific clothing folding operation steps obtained in step 5), specifically including:

[0028] Each vertex of the simplified fitting polygon is sequentially matched and calculated with each vertex and side of the polygon model corresponding to the specific clothing folding operation steps. According to the three defined loss functions, a mapping f is found, and the mapping f makes the sum of the three loss functions the smallest and can ensure that each vertex p of the fitting polygon i will correspond to a certain vertex v of the polygon model m or a certain side s. The mapping f is obtained by using the dynamic programming algorithm to complete the matching.

[0029] In step 6), the mapping f is expressed as follows:

[0030]

[0031] In step 6), the three defined loss functions are as follows:

[0032] The first loss function is:

[0033] The first loss function represents on the fitting polygon with p iThe angular value of the angle with vertex |∠p h p i p j | with the vertex v of the polygon model m The distribution of the angular values of the corresponding angles The degree of conformity, where p h ,p i ,p j represent any three vertices on the fitted polygon, the mean μ of the distribution m and variance are obtained by learning from the training data, and λ V represents the weight value of vertex matching;

[0034] The second loss function is:

[0035] The second loss function represents the value of the edge on the fitted polygon The ratio value to the perimeter of the fitted polygon and the ratio value of the edge of the polygon model The distribution corresponding to the ratio value to the perimeter of the polygon model The degree of conformity, where p i ,p j represent two adjacent vertices on the fitted polygon, v m ,v m+1 represent two adjacent vertices on the polygon model, the mean υ of the distribution m and variance are obtained by learning from the training data, and λ E represents the weight value of edge ratio matching;

[0036] The third loss function is:

[0037] The third loss function represents the penalty term for the vertex p of the fitted polygon i not matching any vertex of the polygon model but matching as an edge of the polygon model, where |∠p i-1 p i p i+1 | represents the angular value of three adjacent vertices of the fitted polygon, the mean ξ = π of the distribution, and the variance φ 2 = π 2 / 16, and λ S represents the weight value of edge matching.

[0038] In step 8), update the parameters of the posture corresponding to the next clothing folding operation step obtained in step 7) to obtain the updated posture, specifically including:

[0039] Update the polygon model loss function corresponding to the posture of the next clothing folding operation step and the parameters of the Gaussian distribution model in the loss function and the μ m value and υ m value of the corresponding distribution in are updated to the geometric feature value data corresponding to the currently fitted polygon actually observed, and the variance of the newly added vertex distribution and are updated to twice the model variance in the clothing polygon model database.

[0040] A two - arm humanoid robot clothing folding system based on clothing pose estimation includes the following specific modules:

[0041] 1) An integrated control module, used to arrange the programs and data for implementing the method of the two - arm humanoid robot clothing folding based on clothing pose estimation;

[0042] 2) A vision module, installed on the head of the two - arm humanoid robot, directly above the clothing operation platform module, used to collect images of the initial pose of the clothing and the pose of the clothing after each operation is completed, and send the images to the integrated control module after image acquisition;

[0043] 3) A control module for the robotic arm and gripper, used to receive the motion trajectories of the robotic arm and the two - finger gripper sent by the integrated control module, and convert the trajectories into control instructions and send them to the robotic arm module and the end - effector module;

[0044] 4) A robotic arm module, used to receive the control instructions sent by the control module for the robotic arm and gripper, and complete the folding operation of the clothing;

[0045] 5) An end - effector module, used to receive the control instructions sent by the control module for the robotic arm and gripper, perform a clamping operation when the gripper reaches the initial picking point coordinates; perform a relaxation operation when the gripper reaches the target placement point coordinates;

[0046] 6) A clothing operation platform module, which is the platform for the clothing to be operated;

[0047] 7) A robot torso module, used to install the vision module, the robotic arm module and the clothing operation platform module.

[0048] A method for a two-armed humanoid robot to fold clothes based on clothing pose estimation. The method includes a clothing image segmentation method, a clothing contour extraction method, a polygon fitting method for clothing contours, a polygon modeling method for clothing, a method for generating clothing folding steps by expert teaching, a method for matching the clothing contour fitting polygon and the clothing polygon model, a method for updating the clothing polygon model, and an obstacle avoidance motion planning method for the two-armed robot.

[0049] The robot system obtains an RGB image of the currently unfolded clothes, and uses the clothing image segmentation method to segment the clothing foreground and the desktop background in the image. The clothing contour extraction method is used to extract the clothing contour from the segmented image, obtaining a contour graph of the clothing. At this time, the contour graph has a very large number of boundary points. The polygon fitting method is used to simplify the polygon fitting of the clothing contour graph. At this time, the obtained fitting polygon has fewer boundary points. The polygon modeling method for clothing is used to model the types of clothes such as long-sleeved T-shirts, short-sleeved T-shirts, shorts, long pants, and towels and their operation steps respectively, and a clothing polygon model database is established accordingly. Using the method for matching the clothing fitting polygon and the clothing polygon model, the simplified fitting polygon is matched with the polygon model in the database to obtain the polygon model corresponding to the clothes in the current image, that is, the pose corresponding to the current clothes. Using the method for updating the clothing polygon model, the polygon model is updated according to the specific folding operation steps of the matched model. According to the matched polygon model and the polygon model after the next operation is completed and updated, the obstacle avoidance motion planning method for the two-armed robot is used to perform obstacle avoidance motion planning for the two manipulators, and the obtained motion trajectory is sent to the robot controller to complete the operation steps of this step. After this step of operation is completed, the clothes will be in a new pose. The system obtains the RGB image of the current pose of the clothes again, and then performs multiple loops according to the above steps until the entire clothing folding process is completed.

[0050] The clothing image segmentation method includes an image foreground and background annotation method and an image pixel segmentation method. The Gaussian mixture model is used to model the color and texture of the operation desktop. Each pixel value of the current RGB image is brought into the Gaussian mixture model of the desktop color, and according to the obtained probability value, it is judged whether the current pixel value is the clothing pixel foreground or the desktop pixel background. The labeled pixel data is used as the input of the image pixel segmentation method, and the image pixel segmentation method uses the GrabCut algorithm, and its output is the segmentation mask of the image pixels. This method can avoid the work of manually annotating the image foreground and background, and the pixel classification and annotation are more accurate.

[0051] The contour extraction method uses the Moore algorithm, which can extract the contour graph of the clothing pixel area in the segmented image. The extracted contour graph may contain hundreds or thousands of contour boundary points.

[0052] The polygon fitting method for the clothing contour adopts the Ramer-Douglas-Peucker algorithm, which can simplify the extracted contour graph into a fitting polygon, and the fitting polygon has at most dozens of contour boundary points. At the same time, geometric feature values such as the relative position relationship between vertices, the size of the interior angle corresponding to each vertex, and the ratio of the length of the line segment between adjacent vertices to the perimeter of the entire polygon can be obtained through the fitted polygon. After this step, the clothing pose in the image is transformed into a simplified fitting polygon, greatly reducing the difficulty of clothing pose recognition.

[0053] The polygon modeling method for the clothing includes the polygon modeling method for a certain type of specific clothing and the polygon modeling method for the operation steps of a certain type of specific clothing. The polygon modeling method for a certain type of specific clothing refers to using the geometric features of the contour graphs corresponding to different types of clothing for modeling. Specifically, for the outer contour polygon of a certain type of specific clothing, the Gaussian model is used to model geometric features such as the relative position between adjacent vertices, the ratio of the length of the line segment between vertices to the perimeter of the entire polygon, and the size of the angle corresponding to each vertex. This model is obtained through learning and training on a certain amount of known geometric feature data of clothing contour polygons. That is, multiple Gaussian models are used to model a certain type of specific clothing polygon to obtain the polygon model corresponding to this type of clothing. The polygon modeling method for the operation steps of a certain type of specific clothing is realized in the form of deleting or adding vertices of the current polygon model to form the next polygon model. That is, multiple Gaussian models are also used to create polygon models for the clothing after each operation is completed. After the model is created, it will be stored in the polygon model database.

[0054] The method for generating the clothing folding steps by expert teaching refers to, based on the clothing polygon model already existing in the model database, using the human-computer interaction system, the user specifies the clothing folding steps, and the human-computer interaction system will generate the corresponding clothing folding step polygon model according to the user's specified steps. If the user does not specify specific operation steps, the system will give the default clothing folding operation steps.

[0055] The method for matching the clothing contour fitting polygon and the clothing polygon model is to perform sequential matching calculations on each vertex of the simplified fitting polygon with each vertex and edge of the polygon model corresponding to the specific clothing folding operation steps. According to the three defined loss functions, a mapping f is found, and the mapping f makes the sum of the values of the three loss functions the smallest and can ensure that each vertex p of the fitting polygon i will correspond to a certain vertex v of the polygon model mcorresponding to a certain edge s, the defined mapping f is expressed as follows:

[0056]

[0057] The defined mapping f needs to satisfy the following three conditions:

[0058] (1) For each vertex v of the clothing polygon model m , there exists a vertex p on the fitting polygon i corresponding to it;

[0059] (2) There cannot be two vertices p i , p j on the fitting polygon corresponding to the same vertex v on the polygon model m . However, multiple vertices on the fitting polygon can correspond to the same edge on the polygon model.

[0060] (3) This mapping can preserve the clockwise order of all vertices on the fitting polygon and also preserve the clockwise order of all vertices on the polygon model.

[0061] The three defined loss functions are as follows respectively:

[0062] The first loss function is:

[0063] The first loss function represents the degree of conformity of the angle value |∠p i p h p i p j | of the angle with vertex p on the fitting polygon and the distribution of the angle value corresponding to vertex v of the polygon model m , where p , p h , p i , p j represent any three vertices on the fitting polygon, and the mean μ m and variance are obtained through learning from the training data, and λ V represents the weight value of vertex matching;

[0064] The second loss function is:

[0065] The second loss function represents the value of the edge on the fitting polygon and the ratio value of the perimeter of the fitting polygon and the ratio value of the edge on the polygon model and the perimeter of the polygon model corresponding to the distribution , where pi , p j represents two adjacent vertices on the fitting polygon, v m , v m+1 represents the mean υ of the distribution of two adjacent vertices on the polygon model m and variance are obtained by learning from the training data, λ E represents the weight value of the edge ratio matching;

[0066] The third loss function is:

[0067] The third loss function represents the vertex p of the fitting polygon i not matching any vertex of the polygon model but matching as the edge penalty term of the polygon model, where |∠p i-1 p i p i+1 | represents the angle value of three adjacent vertices of the fitting polygon, the mean ξ of the distribution is π, and the variance φ 2 = π 2 / 16, λ S represents the weight value of the edge matching.

[0068] The above-defined mapping f can be obtained by using the dynamic programming algorithm, that is, a corresponding polygon model can be obtained according to the geometric features of the current fitting polygon. Thus, the posture corresponding to the current clothing is recognized in the form of a polygon model.

[0069] The update method of the clothing polygon model is to update the polygon model of the clothing posture after the next expert teaching operation corresponding to the polygon model of the current clothing posture. The operation steps of the expert teaching are realized by deleting or adding vertices of the current polygon model to form the form of the next polygon model. After the user specifies the operation steps, the clothing polygon model corresponding to each operation step is also preset. And the update method adjusts the loss function of the preset next polygon model and the parameters of the Gaussian distribution model in, so as to correspond the posture data of the clothing in the actually observed image and the polygon model data of the planned folding operation, thereby improving the accuracy of the robot operation. The μ m value and υ m value in the loss function of the next polygon model are updated to the geometric feature value data corresponding to the current fitting polygon actually observed. The variance and of the newly added vertex distribution is updated to twice the variance in the original next polygon model, which is also to improve the accuracy of the robot operation.

[0070] The obstacle avoidance motion planning method for the dual-arm robot includes three parts: the creation method of the motion planning simulation environment, the dual-arm obstacle avoidance motion planning method, and the robot trajectory generation method.

[0071] The proposed creation method of the motion planning simulation environment is to create various hardware systems with the corresponding relationships between the robot coordinate system, the camera coordinate system, and the image coordinate system calibrated in the real environment in the ROS simulation environment. That is, in the ROS environment, simulation models such as the dual-arm robot system, the camera module, the operation desktop, and the polygonal model of the clothing are established. And except for the polygonal model of the clothing, the physical parameters of other simulation models should correspond one by one to the physical parameters in the real environment, such as the kinematic parameters, dynamic parameters, coordinate system transformation parameters, etc. of the robot.

[0072] The proposed dual-arm obstacle avoidance motion planning method uses the RRT-Connect motion planning algorithm to achieve collision-free motion planning for the two arms. That is, according to the preset expert demonstration operation steps, two points are selected from the vertices of the currently matched polygonal model as the picking points of the two end effectors, and two points are selected from the vertices of the next polygonal model as the placing points of the two end effectors. The two sets of picking point - placing point coordinate value pairs respectively correspond to the initial point coordinates and target point coordinates of the two end effectors. According to these two sets of coordinate values, using the RRT-Connect motion planning algorithm, the collision-free motion trajectories of the two manipulators are completed.

[0073] The proposed robot trajectory generation method is to perform interpolation processing on the generated obstacle avoidance motion trajectories of the two manipulators using the Bezier curve algorithm. The transition between adjacent trajectory points of the obtained trajectory will be smoother, and the changes in the speed and acceleration of the robot executing the entire trajectory will also be more stable. According to the obtained smooth motion trajectory points, inverse kinematics solutions are performed on the two manipulators respectively to obtain the joint trajectories of each manipulator, and this trajectory is sent to the controller of the robot. The end effectors of the robot will respectively select the corresponding vertices on the fitted polygon corresponding to the picking point and the placing point selected in the polygonal model, and use these corresponding vertices on the fitted polygon as the picking point and the placing point corresponding to the real clothing, thereby completing the folding operation for the current clothing posture.

[0074] A humanoid robot clothing folding system for a clothing folding method based on clothing pose estimation. This system includes a vision module, a manipulator module, an end effector module, a manipulator - gripper control module, an integrated control module, a robot torso module, and a clothing operation platform module.

[0075] The visual module includes an RGBD camera module, which is installed on the head of the robot torso module and directly above the clothing operation platform module, and is used to collect images of the initial posture of the clothing and the posture of the clothing after each operation is completed. After the image acquisition is completed, it is sent to the integrated control module.

[0076] The robotic arm module includes two seven-axis redundant collaborative robotic arm units, one installed on each side (left and right) of the robot torso module, which are mainly used to receive control instructions sent by the robotic arm-gripper control module and complete the folding operation of the clothing. The degrees of freedom of the two seven-axis collaborative robotic arms are the same as those of a human arm, and the motion planning speed of the two arms is faster, and the execution of operation instructions is also easier.

[0077] The end effector module includes two-finger gripper units, one installed at the end of each of the left and right robotic arms, which are used to receive control instructions sent by the robotic arm-gripper control module. When the gripper reaches the initial pick-up point coordinates, it performs a clamping operation; when the gripper reaches the target placement point coordinates, it performs a relaxation operation.

[0078] The robotic arm-gripper control module is used to receive the motion trajectories of the robotic arm and the two-finger gripper sent by the integrated control module, and convert the trajectory into control instructions and send them to the robotic arm module and the end effector module.

[0079] The integrated control module includes a polygon model database unit, an expert teaching clothing folding step generation unit, a clothing image processing unit, a clothing fitting polygon and polygon model matching unit, a clothing polygon model update unit, and a two-arm motion planning unit.

[0080] The function of the proposed polygon model database unit is to store pre-trained clothing polygon models. This unit has two storage dimensions. One dimension stores polygon models according to different types of clothing, such as long-sleeved T-shirts, short-sleeved T-shirts, trousers, shorts, and towels; the second dimension stores polygon models according to different folding operation steps of a specific type of clothing.

[0081] The proposed expert teaching clothing folding step generation unit is used for users to specify the clothing folding steps they need. This unit includes a man-machine interaction system. Through this system, users can specify the specific operation steps of the clothing according to the polygon models corresponding to the operation steps preset in the polygon model database unit and store them in the polygon model database. If the user does not specify specific operation steps, the system will give default clothing folding operation steps.

[0082] The function of the proposed clothing image processing unit is to receive the images collected by the camera and sequentially complete the processing procedures such as image segmentation, clothing contour extraction, and clothing contour polygon fitting. The output of this unit is the fitted polygon corresponding to the clothing pose in the current image.

[0083] The function of the proposed clothing fitted polygon and polygon model matching unit is to match the fitted polygon output by the clothing image processing unit with the polygon models in the clothing polygon model database unit. According to the geometric feature information of the known vertices of the fitted polygon and the polygon models, a mapping f will be found using the dynamic programming algorithm. This mapping satisfies the condition that the sum of the values of the aforementioned three loss functions is minimized, and at the same time, it will ensure that each vertex p of the fitted polygon i will correspond to a certain vertex v m or a certain edge s of the polygon model. The output of this unit is the fitted polygon corresponding to the clothing pose in the current image and the matched clothing polygon model.

[0084] The proposed clothing polygon update unit is used to receive the current fitted polygon, the matched polygon model, and the polygon model data corresponding to the clothing pose after the next operation step. The polygon model data after the next operation is updated using the real geometric feature data of the currently obtained fitted polygon. The output of this unit is the updated polygon model after the next operation step.

[0085] The function of the proposed dual-arm motion planning unit is mainly to receive the matched polygon model and the updated polygon model after the next operation step. In the ROS / MoveIt environment, according to the geometric features of the vertices of these two models, the RRT-Connect algorithm is used to complete the obstacle avoidance motion planning of the dual arms, and the result is sent to the robotic arm-two-finger gripper control module. The output of this unit is the motion trajectory of the robotic arm-two-finger gripper.

[0086] The described robot torso module is used to integrate the vision module (head), the robotic arm module, and the clothing operation platform module into an anthropomorphic robot system, making the entire system more flexible in performing operations and more acceptable in appearance to humans.

[0087] The described clothing operation platform module is the platform on which the clothing is operated. The color and texture of the surface of this platform should have been modeled in advance through a Gaussian mixture model. The torso module of the robot is integrated above this module.

[0088] In summary, the present invention has the following beneficial effects:

[0089] (1) The vision module can accurately collect the images of the current clothing and the desktop, and perform a series of processing on this image, and can accurately extract the contour polygon of the clothing.

[0090] (2) The Gaussian model is used to create a polygon model of the clothing contour, and based on this, a polygon model database of the clothing is established, and it is matched with the fitting polygon of the clothing, so that the current posture of the clothing can be accurately recognized, and then the folding operation of the clothing can be completed by using the recognized clothing posture.

[0091] (3) The polygon model database also adds polygon model data of the expert teaching operation steps. The system can list various different folding methods of the same type of clothing, improving the convenience of user use.

[0092] (4) By using the expert teaching clothing folding step unit, the user can preset the clothing folding operation steps according to their own needs, and the robot will complete the clothing folding according to the expert teaching steps.

[0093] (5) The robotic arm used is a seven-axis collaborative robotic arm with a built-in compliant control algorithm, which will not cause harm to the human body when it touches the human body. The entire robot system is designed in a humanoid shape and is more easily accepted by users. Therefore, this system is more suitable for the application scenario of human-robot coexistence.

[0094] (6) By using the dual-arm obstacle avoidance motion planning unit, the obstacle avoidance motion planning of two seven-degree-of-freedom redundant robotic arms can be realized, which can prevent the self-collision of the two arms during the clothing operation process. Description of the Drawings

[0095] Figure 1 Schematic diagram of a humanoid robot clothing folding method based on clothing posture estimation according to an embodiment of the present invention.

[0096] Figure 2 Schematic diagram of clothing contour extraction and fitting according to an embodiment of the present invention.

[0097] Figure 3 Schematic diagram of clothing polygon model modeling according to an embodiment of the present invention.

[0098] Figure 4 Schematic diagram of polygon models of different clothing types according to an embodiment of the present invention.

[0099] Figure 5 Schematic diagram of the matching between the fitting polygon and the polygon model of the clothing according to an embodiment of the present invention.

[0100] Figure 6 Schematic diagram of the generation of expert teaching clothing folding steps according to an embodiment of the present invention.

[0101] Figure 7 Schematic diagram of a humanoid robot clothing folding system for a clothing folding method based on clothing posture estimation according to an embodiment of the present invention.

[0102] Reference numerals:

[0103] S101 - Modeling step of the polygonal model of the clothing; S102 - Modeling step of the surface of the clothing operation platform; S103 - Step of the user specifying the clothing folding operation sequence; S104 - Clothing image acquisition step; S105 - Pixel segmentation step of the clothing image; S106 - Clothing image contour extraction step; S107 - Polygonal fitting step of the clothing contour; S108 - Matching step between the fitted polygon of the clothing and the polygonal model; S109 - Retrieving and updating the polygonal model step; S110 - Dual-arm obstacle avoidance motion planning step; S111 - Trajectory point interpolation step; S112 - Inverse kinematics solution step; S113 - Robot system execution step.

[0104] 100 - Integrated control module, 200 - Manipulator-gripper control module, 300 - Vision module, 400 - Manipulator module, 500 - End effector module, 600 - Robot torso module, 700 - Clothing operation platform module. Detailed implementation manners

[0105] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as imposing any limitation on the protection scope of the present invention.

[0106] As Figure 1 shown, the present invention provides a humanoid robot clothing folding method based on clothing pose estimation, which solves the problem of recognizing the current pose of the clothing by fitting the image contour of the current clothing into a polygon and matching the fitted polygon with the polygonal model in the database, and solves the dual-arm obstacle avoidance motion planning problem by using the matched polygonal model and the polygonal model after the next operation to perform the motion planning of the dual arms. The obtained motion trajectory is sent to the humanoid robot system, and so on in a cycle until the clothing folding operation is finally completed.

[0107] Correspondingly, the present invention also provides a humanoid robot clothing folding system using the clothing folding method based on clothing pose estimation, as Figure 7As shown in the figure, the system includes: an integrated control module 100, which includes a polygon model database unit 101, an expert teaching clothing folding step generation unit 102, a clothing image processing unit 103, a clothing fitting polygon and polygon model matching unit 104, a clothing polygon model updating unit 105, and a two-arm motion planning unit 106; a robotic arm-gripper control module 200; a vision module 300; a robotic arm module 400, which includes a left robotic arm unit 401 and a right robotic arm unit 402; an end effector module 500, which includes a left two-finger gripper unit 501 and a right two-finger gripper unit 502; a robot torso module 600, and a clothing operation platform module 700.

[0108] The above-mentioned humanoid robot clothing folding method based on clothing pose estimation can be executed based on the humanoid robot clothing folding system of the clothing folding method based on clothing pose estimation. The humanoid robot clothing folding method provided by the embodiments of the present invention includes:

[0109] Step S101: Model different types of clothing and the folding steps of the clothing. This step includes two stages.

[0110] The first stage: First, collect a certain number of long-sleeved T-shirts, short-sleeved T-shirts, long pants, shorts, and towel clothes of different sizes respectively. Then, model each vertex of the outer contour polygon of a specific type of clothing, that is, use the Gaussian model to model the relative positions between adjacent two vertices, the ratio of the length of the line segment between vertices to the perimeter of the entire polygon, the angle size of the angle corresponding to each vertex, and other geometric features. The model thus established is the polygon model of this type of clothing. This model is trained through the geometric feature data of a certain number of collected clothing of the same type.

[0111] As Figure 3 shown is a schematic diagram of modeling the polygon model of a long-sleeved T-shirt, where α, β, γ, and δ are the interior angles corresponding to the contour vertices, and a, b, c, d, e, and f are the contour lengths between adjacent vertices. For each geometric feature of this type of clothing, the Gaussian model is used for modeling, that is, the modeling of this type of clothing such as long-sleeved T-shirts is completed using 11 trained Gaussian models.

[0112] As Figure 4 shown is a schematic diagram of polygon models of different clothing types, where the polygon model 401 of the towel has 4 vertices, the polygon model 402 of the short-sleeved T-shirt has 10 vertices, the polygon model 403 of the long-sleeved T-shirt has 10 vertices, the polygon model 404 of the shorts has 7 vertices, and the polygon model 405 of the long pants has 7 vertices.

[0113] The second stage: Different teaching operation steps are formed for different folding methods of a specific type of clothing. Each teaching operation step of different folding methods is realized in the form of deleting or adding vertices of the current polygon model to form the next polygon model. For example Figure 6 S601 in it is the initial pose polygon model of a long-sleeved T-shirt, and the polygon model S602 corresponding to the clothing pose after the next operation is formed by deleting point a and point b and adding point e and point f on the original model. That is, multiple Gaussian models are also used to create the polygon model of the clothing after each operation is completed.

[0114] After the above two stages are completed, all the created clothing polygon models are stored in the polygon model database unit 101 in the integrated control module 100.

[0115] Step S102: Model the surface of the clothing operation platform.

[0116] Collect the RGB images of three commonly used desktops in daily life, such as Figure 2 The desktop background of 201 in it is a common desktop type. According to the collected desktop image information, a Gaussian mixture model is used to realize the modeling of the color and texture of the operation desktop. This Gaussian mixture model can adopt three Gaussian distribution components. The established desktop Gaussian mixture model is stored in the clothing image processing unit 103 in the integrated control module 100. This operation platform corresponds to the clothing operation platform module 700.

[0117] Step S103: The user specifies the folding operation steps of the clothing.

[0118] According to the clothing polygon models already existing in the polygon model database unit 101 of the integrated control module 100, using the expert teaching clothing folding step generation unit 102, the user specifies the steps of the clothing folding operation, and the expert teaching clothing folding step generation unit 102 will generate the corresponding clothing folding step polygon model according to the user-specified steps. The generated polygon model of the specified steps will be stored in the real-time storage space of the polygon model database unit 101. If the user does not specify the specific operation steps, the expert teaching clothing folding step generation unit 102 will give the default clothing folding operation steps. For example, as Figure 6 shown is the schematic diagram of the folding steps of the long-sleeved T-shirt specified by the user, and this folding step includes a total of 7 operation steps.

[0119] Step S104: The vision module collects the image of the current clothing.

[0120] The robot system uses the RGBD camera in the vision module 300 to collect the RGB image of the current clothing, such as Figure 2The captured image shown as 201 in [the figure] is an image of a pair of trousers. The captured image data is transmitted to the clothing image processing unit 103 in the integrated control module 100.

[0121] Step S105: Perform pixel segmentation on the clothing image.

[0122] After receiving the RGB image of the current piece of clothing, the clothing image processing unit 103 in the integrated control module 100 substitutes the pixel data of this image into the Gaussian mixture model corresponding to the surface color and texture of the pre-established clothing operation platform module 300, and judges whether the current pixel value is the foreground (clothing pixel) or the background (desktop pixel) according to the obtained probability value. The labeled pixel data is used as the input of the image pixel segmentation method, and the image pixel segmentation method uses the GrabCut algorithm, and its output is the segmentation mask of the image pixels. This method can avoid the work of manually labeling the foreground and background of the image, and the pixel classification and labeling are more accurate. This step separates the pixels of the clothing foreground and the desktop background in the image.

[0123] Step S106: Extract the contour in the image of the clothing.

[0124] First, grayscale the image output in step S105, then select an appropriate threshold to perform binary processing on this grayscale image, and finally use the Moore algorithm to extract the boundary contour graph of the clothing foreground pixel area and the desktop background pixel area. The extracted contour graph may contain hundreds or thousands of contour boundary points, and the specific number is determined by the resolution of the used camera. As Figure 2 The red contour shown as 202 in [the figure] is the contour graph of the pair of trousers. Step S106 is performed in the clothing image processing unit 103.

[0125] Step S107: Perform polygon fitting on the extracted clothing contour.

[0126] According to the clothing contour graph obtained in step S106, perform polygon fitting using the Rame-Douglas-Peucker algorithm. This method can simplify the extracted contour graph into a fitted polygon, and this fitted polygon has at most only dozens of contour boundary points. As Figure 2As shown by 203 in [the figure], it is the fitted polygon of the long pants. The simplified fitted polygon has only 12 vertices. At this time, geometric eigenvalue such as the relative position relationship between vertices of the fitted polygon, the size of the interior angle corresponding to each vertex, and the ratio of the length of the line segment between adjacent vertices to the perimeter of the entire polygon can all be obtained through the fitted polygon. In this way, the clothing pose in the image is transformed into a simplified fitted polygon, greatly reducing the difficulty of clothing pose recognition. Step S107 is carried out in the clothing image processing unit 103. The established fitted polygon will be sent to the clothing fitted polygon and polygon model matching unit 104 in the integrated control module 100.

[0127] Step S108: Use the dynamic programming algorithm to match the fitted polygon of the clothing contour with the polygon model.

[0128] In the clothing fitted polygon and polygon model matching unit 104, use the dynamic programming algorithm to match the fitted polygon obtained in step S107 with the polygon models stored in the polygon model database unit 101.

[0129] Specifically, each vertex of the fitted polygon obtained by fitting the clothing contour polygon is sequentially matched and calculated with each vertex and edge of the polygon model corresponding to the specific clothing folding operation steps. According to the three defined loss functions, find a mapping f, and the mapping f makes the sum of the three loss functions the smallest and can ensure that each vertex p i of the fitted polygon m corresponds to a certain vertex v

[0130]

[0131] of the polygon model or a certain edge s. The defined mapping f is expressed as follows:

[0132] (1) For each vertex v m of the clothing polygon model, there exists a vertex p i on the fitted polygon corresponding to it;

[0133] (2) There cannot be two vertices p i , p j on the fitted polygon corresponding to the same vertex v m on the polygon model. However, multiple vertices on the fitted polygon can correspond to the same edge on the polygon model.

[0134] (3) This mapping can preserve the clockwise order of all vertices on the fitted polygon and also preserve the clockwise order of all vertices on the polygon model.

[0135] The three defined loss functions are as follows:

[0136] The first loss function is:

[0137] The first loss function represents the degree of conformity of the angle value |∠p i at the vertex of the fitting polygon to the distribution of the angle values of the corresponding angles of the vertices v h p i p j | of the polygon model, where p m The mean μ , p h , p i , p j represent any three vertices on the fitting polygon, and the variance m are obtained through learning from the training data, and λ represents the weight value of vertex matching; V

[0138] The second loss function is:

[0139] The second loss function represents the ratio value of the value of the edge on the fitting polygon to the perimeter of the fitting polygon and the ratio value of the edge on the polygon model to the perimeter of the polygon model, and the distribution of the corresponding values, where p i , p j represent two adjacent vertices on the fitting polygon, and v m , v m+1 represent two adjacent vertices on the polygon model. The mean υ m and variance are obtained through learning from the training data, and λ E represents the weight value of edge ratio matching;

[0140] The third loss function is:

[0141] The third loss function represents the penalty term for the vertex p i of the fitting polygon not matching any vertex of the polygon model but matching the edge of the polygon model. Here, |∠p i-1 p i p i+1 | represents the angle value of three adjacent vertices of the fitting polygon. The mean ξ = π, and the variance φ 2 = π 2 / 16, and λ S ​Represents the weight value for edge matching.

[0142] Using the dynamic programming algorithm, the mapping f defined above can be obtained, that is, according to the geometric features of the current fitted polygon, a corresponding polygon model can be obtained. Thus, the pose corresponding to the current clothing is recognized in the form of a polygon model.

[0143] According to experience, the weight values are set as follows: λ V = 1, λ E = 1 / 3, and λ S = 1.

[0144] Using the dynamic programming algorithm, the mapping f defined above can be obtained, that is, according to the geometric features of the current fitted polygon, a corresponding polygon model can be obtained. As Figure 5 shown, 501 is the fitted polygon of the towel, with a total of 7 vertices, and 502 is the polygon model of the towel, with a total of 4 vertices. After the matching is completed, vertex P1 corresponds to vertex V1, vertex P2 corresponds to V2, vertex P3 corresponds to edge V2 - V3, vertex P4 corresponds to vertex V3, vertex P5 corresponds to vertex V4, and vertices P6 and P7 correspond to edge V1 - V4.

[0145] Thus, the pose corresponding to the current clothing is recognized in the form of a polygon model. For example, as Figure 6 shown in S601, it is the polygon model of the initial state of the long-sleeved T-shirt recognized by the system. The matched polygon models will be sent to the clothing polygon model update unit 105 and the double-arm motion planning unit 106 respectively.

[0146] Step S109: Retrieve and update the polygon model corresponding to the next operation step after the matching polygon. This step includes two stages.

[0147] The first stage: The clothing polygon model update unit 105 will retrieve the clothing polygon model corresponding to the next operation after completion in the real-time storage space of the polygon model database unit 101 according to the polygon model matched in step S108. For example, Figure 6 as shown in S601, it is the polygon model of the initial state of the clothing, and S602 is the polygon model corresponding to the clothing state after the next operation retrieved.

[0148] The second stage: The clothing polygon model update unit 105 will update the retrieved clothing polygon model corresponding to the next operation. That is, update the μ m value and υ m value corresponding to the distribution in the loss function of the retrieved polygon model to the geometric feature value data corresponding to the currently observed fitted polygon. The variance of the newly added vertex distribution and Update it to twice the variance in the original next-step polygon model. The purpose of this update is to improve the accuracy of the robot's actual operation. The updated polygon model will be sent to the dual-arm motion planning unit 106.

[0149] Step S110: Dual-arm obstacle avoidance motion planning.

[0150] The dual-arm motion planning unit 106 will complete the motion planning of the two arms according to the matched polygon model sent in step S108 and the updated polygon model corresponding to the clothing posture after the next operation sent in step S109. This step includes two stages.

[0151] The first stage: Create various hardware systems that have calibrated the corresponding relationships between the robot coordinate system, camera coordinate system, and image coordinate system in the real environment in the ROS / MoveIt simulation environment. That is, establish simulation models such as the vision module, robotic arm module, end effector module, robot torso module, and clothing operation platform module in the ROS / MoveIt environment. And except for the polygon model of the clothing, the physical parameters of other simulation models should correspond one by one to the physical parameters in the real environment, such as the kinematic parameters, dynamic parameters, coordinate system conversion parameters, etc. of the robot.

[0152] The second stage: The dual-arm motion planning unit 106 will use the RRT-Connect motion planning algorithm to achieve collision-free motion planning of the two arms in the ROS / MoveIt environment. That is, according to the expert demonstration operation steps specified by the user, select two points from the vertices of the currently matched polygon model as the picking points of the two end effectors, and select two points from the vertices of the next-step polygon model as the placing points of the two end effectors. The two sets of picking point - placing point coordinate value pairs respectively correspond to the initial point coordinates and target point coordinates of the two end effectors. According to these two sets of coordinate values, use the RRT-Connect motion planning algorithm to complete the collision-free motion trajectories of the two robotic arms respectively. Such as Figure 6As shown in the figure, for the first step of clothing folding operation, points a and b in S601 respectively correspond to the grasping and picking points of the two end effectors, and points c and d in S602 are the corresponding placement points. The RRT-Connect algorithm can be used to simultaneously plan the collision-free motion trajectories of the left robotic arm from point a to point c and the right robotic arm from point b to point d. At the same time, when the left robotic arm reaches point a and the right robotic arm reaches point b, the clamping operation is performed; when the left robotic arm reaches point c and the right robotic arm reaches point d, the relaxation operation is performed. Similarly, for the second step of operation, points e and f in S602 are the picking points of the left and right arms, and points h and g in S603 are the placement points of the left and right arms; for the third step of operation, points a1 and b1 in S603 are the picking points of the left and right arms, and points c1 and d1 in S604 are the placement points of the left and right arms; for the fourth step of operation, points e1 and f1 in S604 are the picking points of the left and right arms, and points g1 and h1 in S605 are the placement points of the left and right arms; for the fifth step of operation, points i1 and j1 in S605 are the picking points of the left and right arms, and points k1 and l1 in S606 are the placement points of the left and right arms; for the sixth step of operation, points i and j in S606 are the picking points of the left and right arms, and points k and l (l1) in S607 are the placement points of the left and right arms; for the seventh step of operation, points m and m1 in S607 are the picking points of the left and right arms, and points n and n1 in S608 are the placement points of the left and right arms. After the above 7 steps of operations specified by the user are completed, the long-sleeved T-shirt can be folded by the humanoid robot.

[0153] Step S111: Trajectory point interpolation.

[0154] According to the motion trajectories of the two arms obtained in step S110, the two-arm motion planning unit 106 will respectively perform interpolation processing using the Bezier curve algorithm, and the transition between adjacent trajectory points of the obtained trajectory will be smoother, and the speed and acceleration changes of the robot when executing the entire trajectory will also be more stable.

[0155] Step S112: Inverse kinematics solution.

[0156] According to the smooth motion trajectory points obtained in step S111, the two-arm motion planning unit 106 will respectively perform inverse kinematics solutions for the two robotic arms to obtain the executable joint trajectories of each robotic arm. These trajectory points will be sent to the robotic arm-gripper control module 200. The robotic arm-gripper control module 200 will send corresponding control instructions to the left robotic arm unit 401 and the right robotic arm unit 402 in the robotic arm module 400, and the left two-finger gripper unit 501 and the right two-finger gripper unit 502 in the end effector module 500 according to the received motion trajectories.

[0157] Step S113: Robot system execution.

[0158] After the robotic arm module 400 and the end effector module 500 receive the control instructions sent by the robotic arm - gripper control module 200 in step S112, the left robotic arm unit 401 and the right robotic arm unit 402 will move according to the instructions. The left two-finger gripper unit 501 and the right two-finger gripper unit 502 will respectively select the corresponding vertices on the fitted polygon corresponding to the selected picking point and placing point in the polygon model, and use the corresponding vertices on the fitted polygon as the corresponding picking point and placing point on the real clothing, so as to complete the folding operation of the current clothing posture. As Figure 6 shown, the S601 posture of the clothing is folded into the S602 posture. After that, the clothing will be in a new posture and enter step S104 again, and this process will be executed cyclically until the 7 operation steps specified by the user are completed, and the clothing is finally folded into the state of S608 as Figure 6 shown.

[0159] The embodiments of the present invention have been described above. However, these embodiments are only for illustrative purposes and not for limiting the scope of the present invention. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present invention.

Claims

1. A method for folding clothes by a two-armed humanoid robot based on clothing pose estimation, characterized in that, The specific steps include: 1) Obtaining a clothing image, segmenting the clothing foreground and desktop background in the clothing image to obtain a segmented image; 2) extracting the contour of the clothing from the segmented image to obtain a contour graphic of the clothing, wherein the contour graphic of the clothing has 500 to 5000 boundary points; 3) performing simplified polygon fitting on the outline of the clothing to obtain a simplified fitting polygon, wherein the fitting polygon has 5 to 30 boundary points; 4) Based on the geometric features of clothing polygons, a Gaussian model is used to model different types of clothing and clothing folding steps to obtain a clothing polygon model database, specifically including: 4.1) For the same type of clothing outline polygons, the initial Gaussian model is trained by learning the three geometric features of the known clothing outline polygons to obtain a trained Gaussian model, thus completing the Gaussian model building; The three geometric features are the distance between two adjacent vertices of the simplified fitting polygon corresponding to the known clothing outline polygon, the ratio of the length of the line segment between vertices to the perimeter of the entire polygon, and the angle corresponding to each vertex; 4.2) Deleting or adding vertices of the trained Gaussian model to form the next polygonal model; 4.3) The trained Gaussian model and the next polygonal model are aggregated to form a clothing polygonal model database; 5) Specifying specific clothing folding operation steps from the clothing polygon model database according to user needs; 6) matching the simplified fitting polygon obtained in step 3) with the specific clothing folding operation steps obtained in step 5) to obtain a polygonal model corresponding to the clothing in the current image, wherein the polygonal model corresponding to the clothing in the current image represents the posture corresponding to the current clothing; 7) Based on the posture corresponding to the current clothing obtained in step 6) and the specific clothing folding operation steps obtained in step 5), the posture corresponding to the next clothing folding operation step is obtained in time sequence; 8) updating the parameters of the posture corresponding to the next clothing folding operation step obtained in step 7) to obtain an updated posture; 9) The dual-arm robot uses an obstacle avoidance motion planning algorithm to complete the obstacle avoidance motion planning from the posture operation corresponding to the current clothing in step 6) to the updated posture in step 8).

2. The method for folding clothes by a two-armed humanoid robot based on clothing pose estimation according to claim 1, characterized in that In step 2), Moore algorithm is used to extract clothing contours from the segmented image.

3. The clothing folding method of a dual-arm humanoid robot based on clothing posture estimation according to claim 1, characterized in that: In step 3), the Ramer-Douglas-Peucker algorithm is used to fit the simplified polygon of the clothing contour.

4. The method for folding clothes by a two-armed humanoid robot based on clothing pose estimation according to claim 1, wherein In step 6), the simplified fitting polygon obtained in step 3) is matched with the specific clothing folding operation steps obtained in step 5), specifically including: Match each vertex of the simplified fitting polygon with each vertex and edge of the polygon model corresponding to the specific clothing folding operation steps in sequence. According to the three defined loss functions, find a mapping f that minimizes the sum of the three loss functions and ensures that each vertex p of the fitting polygon i corresponds to a certain vertex v of the polygon model m or a certain edge s, and use the dynamic programming algorithm to obtain the mapping f to complete the matching.

5. The method for folding clothes by a dual-arm humanoid robot based on clothes posture estimation according to claim 4, characterized in that: In step 6), the mapping f is expressed as follows:

6. The method for folding clothes of a two-armed humanoid robot based on clothing pose estimation according to claim 4, characterized in that In step 6), the three loss functions defined are as follows: The first loss function is: The first loss function represents the degree of conformity of the angle value |∠p i with p h p i p j as the vertex of the fitted polygon to the distribution of the angle values of the corresponding angles of the vertices v m of the polygon model, where p , p h , p i , p j represent any three vertices on the fitted polygon, the mean μ m and variance are obtained through learning from the training data, and λ V represents the weight value of vertex matching; The second loss function is: The second loss function represents the edges on the fitted polygon Value and the perimeter of the fitted polygon The ratio of the value to the edge of the polygon model Distribution corresponding to the proportional value of the polygon model perimeter The degree of compliance, where p i ,p j Represents two adjacent vertices on the fitted polygon, v m ,v m+1 Represents the mean value of the distribution of two adjacent vertices on the polygonal model m and variance is obtained by learning the training data, E Represents the weight value of edge ratio matching; The third loss function is: The third loss function represents the vertex p of the fitted polygon i The penalty term for matching the edge of a polygonal model without matching any vertex of the polygonal model, where |∠p i-1 p i p i+1 | represents the angle value of the three adjacent vertices of the fitted polygon, the mean value of the distribution is ξ=π, and the variance is φ 2 =π 2 / 16,λ S Represents the weight value of edge matching.

7. The method for folding clothes by a two-armed humanoid robot based on clothing pose estimation according to claim 6, characterized in that, In step 8), the parameters of the posture corresponding to the next clothing folding operation step obtained in step 7) are updated to obtain an updated posture, specifically including: Update the polygonal model loss function corresponding to the posture corresponding to the next clothing folding operation step and Parameters of the Gaussian distribution model, loss function and The corresponding distribution μ m Value and υ m The value is updated to the actual observed geometric eigenvalue data corresponding to the current fitted polygon, and the variance of the vertex distribution is newly added. and Updated to twice the model variance in the clothing polygon model database.

8. A two-armed humanoid robot clothing folding system based on clothing pose estimation, characterized in that, Includes the following specific modules: 1) An integrated control module, which is used to arrange the programs and data for implementing the method for folding clothes by a two-armed humanoid robot based on clothing pose estimation described in claim 1; 2) A vision module, which is installed on the head of the two-armed humanoid robot and is located directly above the clothing operation platform module, and is used to collect images of the initial pose of the clothes and the pose of the clothes after each operation is completed, and sends the images to the integrated control module after image acquisition; 3) A control module for the robotic arm and gripper, which is used to receive the motion trajectories of the robotic arm and the two-finger gripper sent by the integrated control module, and convert the trajectories into control instructions and send them to the robotic arm module and the end effector module; 4) A robotic arm module, which is used to receive the control instructions sent by the control module for the robotic arm and gripper and complete the folding operation of the clothes; 5) An end effector module, which is used to receive the control instructions sent by the control module for the robotic arm and gripper. When the gripper reaches the initial picking point coordinates, it performs a clamping operation; when the gripper reaches the target placement point coordinates, it performs a relaxation operation; 6) A clothing operation platform module, which is the platform for the clothes to be operated; 7) A robot torso module, which is used to install the vision module, the robotic arm module and the clothing operation platform module.

Images