Mechanical arm flattening cloth method and system based on hierarchical operation strategy

Abstract

The application discloses a mechanical arm cloth flattening method and system based on a layered operation strategy, relates to the field of automatic control technology, and comprises the following steps: obtaining cloth information, wherein the cloth information comprises corner and contour information; controlling the mechanical arm to perform coarse operation or fine operation on the cloth according to the number of cloth corners in the field of view, wherein the coarse operation is performed when the number of corners is less than 2, and the fine operation is performed when the number of corners is greater than or equal to 2; the coarse operation increases the cloth coverage area and the number of exposed corners by swinging the cloth through the mechanical arm; and the fine operation completely flattens the cloth according to the determined grabbing point and placing track. The application utilizes the gravity to eliminate the wrinkles of the cloth, proposes a layered operation strategy, can reduce the operation times of the mechanical arm for flattening the cloth with high wrinkles, and improves the cloth flattening efficiency.

Description

Technical Field

[0001] This invention relates to the field of automation control technology, and in particular to a method and system for flattening and laying fabric using a robotic arm based on a layered operation strategy. Background Technology

[0002] Flattening fabric is a crucial operation, forming the basis for subsequent operations such as fabric grasping, folding, and sewing. Due to the infinite degrees of freedom and unknown dynamic characteristics of fabric, it's difficult to determine the impact of robotic arm movements on the fabric's state. Therefore, flattening fabric using robotic arms still presents several challenges. Currently, robotic arm fabric flattening primarily employs dual-arm operation and remains in the laboratory stage. However, dual-arm flattening has high requirements for platform deployment and workspace, and the collaborative control of the two arms is quite complex. In comparison, single-arm flattening is more suitable for situations where the robotic arm's range of motion is limited. Existing single-arm robot fabric flattening scenarios mostly rely on machine vision on a workbench, using heuristic methods based on the fabric's feature information to determine the robotic arm's actions. For example:

[0003] CN114723831A discloses a heuristic-based robotic method and system for flattening flexible fabrics. The method involves acquiring and preprocessing image information of the fabric, then filtering out the largest folds from the preprocessed images to obtain their length and center point. Within each flattening cycle, the robot grasps the corner closest to the center point of the largest fold along its perpendicular bisector and drags it for half the length of the fold. The system then determines whether the fabric is flat. However, for highly wrinkled fabrics, the robotic arm requires multiple iterations to flatten them, resulting in low flattening efficiency. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for flattening fabric using a robotic arm based on a layered operation strategy. By utilizing the effect of gravity on eliminating fabric wrinkles, a layered operation strategy is proposed, which can reduce the number of operations required for the robotic arm to flatten highly wrinkled fabric and improve the efficiency of flattening fabric.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide a method for flattening fabric using a robotic arm based on a layered operation strategy, comprising:

[0007] Obtain fabric information, including corner and outline information;

[0008] The robotic arm determines whether to perform coarse or fine operations on the fabric based on the number of fabric corners in the field of view. Coarse operations are performed when the number of corners is less than 2, and fine operations are performed when the number of corners is greater than or equal to 2. Coarse operations increase the coverage area and the number of exposed corners of the fabric by swinging the fabric with the robotic arm. Fine operations completely flatten the fabric according to the determined gripping point and placement trajectory.

[0009] As a further implementation, the method for obtaining fabric information is as follows:

[0010] Mark the edges and corners of the fabric, and use a camera to capture aligned depth and color images of the fabric;

[0011] The color image is processed using HSV, and the edges and corners of the fabric are extracted as labels for the training dataset.

[0012] Fabric depth images and labels constitute the training dataset for the U-Net network;

[0013] The model trained using the U-Net network obtains the pixel positions of the fabric corners from the current fabric depth map;

[0014] The current color image of the fabric undergoes grayscale conversion and image denoising. The Canny detector is then used to obtain the pixel positions of the fabric outline and edges.

[0015] As a further implementation, the coarse operation process is as follows:

[0016] When the number of angles in the field of view is 0, the robotic arm lifts the fabric into the air by grasping the center point of the fabric mass, swings the fabric, and then places it back.

[0017] When the number of corners in the field of view is 1, the robotic arm lifts the fabric into the air by grabbing that corner, swings the fabric, and then places it back.

[0018] As a further implementation, the detailed operation process is as follows:

[0019] For several color images of fabric in different states, the grab points are labeled to form a training dataset, and the dataset is expanded.

[0020] Using the ResNet network structure, the location of the grab point is determined from the current color image of the fabric;

[0021] The placement trajectory is determined based on the position of the grab point, the pixel position of the fabric corner, and the pixel position of the fabric outline and edges.

[0022] As a further implementation, if the two sides of the fabric corner form a triangle with the contour determined by the Canny detector, then the robotic arm unfolds the fabric in a semi-circular trajectory with the contour as the unfolding axis and the vertical distance from the fabric corner to the contour as the radius.

[0023] If there are no upper triangular folds, the placement trajectory is parallel to the working surface.

[0024] As a further implementation, when there are no upper-level triangular folds, the placement vector length is fixed, and the placement vector direction is taken as the average direction of the following two unit vectors:

[0025] The visual quality center of the fabric points towards the gripping point;

[0026] The grab point points to the nearest edge pixel.

[0027] As a further implementation, the dataset is expanded by left-right flipping, up-down flipping, rotation, and affine transformation.

[0028] Secondly, embodiments of the present invention also provide a robotic arm flattening fabric system based on a layered operation strategy, comprising:

[0029] The fabric information sensing module is used to acquire information about the edges and contours of the fabric.

[0030] The robotic arm is used to perform coarse or fine operations on the fabric based on the fabric information obtained by the fabric information sensing module, and to feed back the current fabric information to the fabric information sensing module.

[0031] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the described method for flattening fabric using a robotic arm based on a layered operation strategy.

[0032] Fourthly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the described method for flattening fabric using a robotic arm based on a layered operation strategy.

[0033] The beneficial effects of this invention are as follows:

[0034] (1) This invention combines color images and depth images of the fabric to obtain information on the edges and contours of the fabric, thereby improving the accuracy of the selection of gripping and placement points for the robot to operate the fabric.

[0035] (2) The present invention divides the operation into coarse operation and fine operation. The coarse operation increases the coverage area of ​​the fabric and exposes more corners of the fabric by swinging the fabric in the air with the robotic arm. The fine operation determines the gripping point and placement trajectory based on the coarse operation to completely flatten the fabric. The combination of coarse and fine operation utilizes the effect of gravity on eliminating fabric wrinkles, reducing the number of operations required for the robotic arm to flatten highly wrinkled fabric. This can quickly and effectively improve the flatness of the fabric, thereby improving the efficiency of the flattening process. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a flowchart of the robotic arm flattening fabric according to one or more embodiments of the present invention;

[0038] Figure 2 This is a schematic diagram of the overall flattening process according to one or more embodiments of the present invention;

[0039] Figure 3 This is a schematic diagram of the fabric information sensing module structure according to one or more embodiments of the present invention;

[0040] Figure 4 This is a schematic diagram of the U-Net network structure according to one or more embodiments of the present invention;

[0041] Figure 5 This is a detailed operation flowchart of the present invention according to one or more embodiments;

[0042] Figure 6 This is a schematic diagram of the ResNet network structure according to one or more embodiments of the present invention. Detailed Implementation

[0043] Example 1:

[0044] This embodiment provides a method for flattening fabric using a robotic arm based on a layered operation strategy, including:

[0045] Obtain fabric information, including corner and outline information;

[0046] The robotic arm determines whether to perform coarse or fine operations on the fabric based on the number of fabric corners in the field of view. Coarse operations are performed when the number of corners is less than 2, and fine operations are performed when the number of corners is greater than or equal to 2. Coarse operations increase the coverage area and the number of exposed corners of the fabric by swinging the fabric with the robotic arm. Fine operations completely flatten the fabric according to the determined gripping point and placement trajectory.

[0047] Specifically, such as Figure 1 and Figure 2As shown, the fabric information sensing module (RGB-D camera) acquires the corner and outline information of the fabric. Based on whether the number of fabric corners in the field of view is less than two, the robotic arm is controlled to perform either coarse or fine manipulation of the fabric. Coarse manipulation involves the robotic arm swinging the fabric in the air to increase the coverage area and expose more of the fabric's corners. Fine manipulation, based on the coarse manipulation, determines the gripping point and placement trajectory to completely flatten the fabric. After performing either coarse or fine manipulation, the corner and outline information of the fabric is returned to the fabric information sensing module.

[0048] like Figure 3 As shown, the specific methods for fabric corner recognition and fabric contour and edge detection are as follows:

[0049] Step 1: Mark the edges of the fabric with a colored pen, and use a camera to capture several aligned depth and color images of the fabric.

[0050] Step 2: Perform HSV processing on the color image and extract the edges and corners of the fabric as labels for the training dataset.

[0051] Step 3: The fabric depth image and the corresponding labels from Step 2 constitute the training dataset for the U-Net network. The structure of the U-Net network is as follows: Figure 4 As shown.

[0052] The loss function L1 is defined as the binary cross-entropy loss l for each class k∈K. k Average value:

[0053]

[0054]

[0055] Here, category K has 3 types: edges, corners, and others of the fabric; i is a pixel in the input depth image I; ω k These are the weights of different categories, y i It is the binary label at position i. It is the network prediction value at point i.

[0056] Step 4: Apply the model trained by the U-Net network in Step 3 to obtain the pixel positions S(E1,E2,...E) of the fabric corners from the current fabric depth map. m C1, C2...C n E represents the edge of the fabric, m is the number of edges in the image, C represents the corner of the fabric, and n is the number of corners in the field of view.

[0057] Step 5: After grayscale conversion and image denoising of the current fabric color image, the pixel positions θ(θ1,θ2,...θ) of the fabric outline and edges are obtained using a Canny detector. j), where θ represents a contour or edge, and j is the sum of the number of contours and edges detected by the Canny detector.

[0058] Based on the number of fabric corners detected by the fabric information sensing module, the robotic arm is controlled to perform coarse or fine manipulation of the fabric. After each operation, the fabric information sensing module detects the current fabric state, repeating the process as follows. Figure 1 The judgment process shown:

[0059] When the number of angles in the field of view is less than 2, perform coarse operation:

[0060] (1) When the number of angles in the field of view is 0, the robotic arm lifts the fabric in the air by grasping the center point of the fabric mass, swings the fabric and puts it back in place.

[0061] (2) When the number of angles in the field of view is 1, the robotic arm lifts the fabric into the air by grabbing the angle, swings the fabric and puts it back in place.

[0062] When the number of angles in the field of view is greater than or equal to 2, the robotic arm performs fine operations, such as... Figure 5 As shown, the specific steps are as follows:

[0063] Step 1: For several (e.g., 100) color images of fabric in different states, mark the grab points through the human-computer interaction interface to form a training dataset, and expand the dataset by flipping left and right, flipping up and down, rotating and affine transformation.

[0064] Step 2: The training network in this embodiment uses a ResNet-34 fully convolutional network, and its network structure is as follows: Figure 6 As shown; where 64, 128, 256, and 512 are the number of channels, / 2 indicates a step size of 2, solid arrows represent residual blocks with equal numbers of input and output channels, and dashed arrows represent residual blocks with unequal numbers of input and output channels.

[0065] The final fully connected layer uses the sigmoid activation function, as shown in the following equation:

[0066]

[0067] Where x is the output of the average pooling layer, and the network's loss function L2 uses the binary cross-entropy loss function:

[0068]

[0069] Where N is the number of samples, j is the number of pixels in the fabric color image G, and u i The binary label at position i is p. i It is the probability that point i is predicted to be a grab point.

[0070] Step 3: Apply the model obtained in Step 2 to determine the gripping point position O from the current fabric color image. pick .

[0071] Step 4: The placement point and placement trajectory are determined by two fixed strategies.

[0072] Based on the grab point location O pick The pixel positions of the fabric corners S(E1,E2,...E) m C1, C2...C n The placement trajectory is determined by the pixel positions θ(θ1,θ2,...) of the fabric outline and edges, and the placement trajectory is determined as follows:

[0073] (1) If the two sides (E1,E2) of the fabric angle C1 form a triangle with the contour θ1 determined by the Canny detector, that is, there is an upper triangular fold; then with the contour θ1 as the unfolding axis and the vertical distance from the angle C1 to the contour θ1 as the radius, the robotic arm unfolds the fabric in a semi-circular trajectory.

[0074] (2) If there are no upper triangular folds, the placement trajectory is parallel to the working surface, the length of the placement vector is fixed at l, and the direction of the placement vector is the average direction of the following two unit vectors:

[0075] 1) The visual quality center O of the fabric points to the gripping point O. pick ;

[0076] 2) Grab point O pick Pointing to the nearest outer edge pixel O. N .

[0077] This embodiment utilizes gravity to eliminate fabric wrinkles and proposes a layered operation strategy, which can reduce the number of operations required for the robotic arm to flatten highly wrinkled fabric and improve the efficiency of fabric flattening. By combining color and depth images of the fabric, information on the fabric's edges and contours is obtained, improving the accuracy of the robotic arm's selection of gripping and placement points for fabric manipulation. By combining coarse and fine operations, the flatness of the fabric can be improved quickly and effectively, thereby increasing the efficiency of the flattening process.

[0078] Example 2:

[0079] This embodiment provides a robotic arm flattening fabric system based on a layered operation strategy, and includes the following based on the method described in Embodiment 1:

[0080] The fabric information sensing module is used to acquire information about the edges and contours of the fabric.

[0081] The robotic arm is used to perform coarse or fine operations on the fabric based on the fabric information obtained by the fabric information sensing module, and to feed back the current fabric information to the fabric information sensing module.

[0082] Example 3:

[0083] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the robotic arm flattening fabric method based on a layered operation strategy described in Embodiment 1.

[0084] Example 4:

[0085] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the robotic arm flattening fabric method based on a layered operation strategy described in Embodiment 1.

[0086] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for flattening fabric using a robotic arm based on a layered operation strategy, characterized in that, include: Obtain fabric information, including corner and outline information; The robotic arm determines whether to perform coarse or fine operations on the fabric based on the number of fabric corners in the field of view. Coarse operations are performed when the number of corners is less than 2, and fine operations are performed when the number of corners is greater than or equal to 2. Coarse operations increase the coverage area and the number of exposed corners of the fabric by swinging the fabric with the robotic arm. Fine operations completely flatten the fabric according to the determined gripping point and placement trajectory. The method for obtaining the fabric information is as follows: Mark the edges and corners of the fabric, and use a camera to capture aligned depth and color images of the fabric; The color image is processed using HSV, and the edges and corners of the fabric are extracted as labels for the training dataset. Fabric depth images and labels constitute the training dataset for the U-Net network; loss function. Defined as each category Binary cross-entropy loss Average value: Among them, categories There are three categories: fabric edges, corners, and others. It is the input depth image Pixels in These are the weights of different categories. yes Binary labels at the location, yes Network prediction values ​​at the location; The model trained using the U-Net network obtains the pixel positions of the fabric corners from the current fabric depth map; The current color image of the fabric undergoes grayscale conversion and image denoising. The Canny detector is then used to obtain the pixel positions of the fabric outline and edges. The coarse operation process is as follows: When the number of angles in the field of view is 0, the robotic arm lifts the fabric into the air by grasping the center point of the fabric mass, swings the fabric, and then places it back. When the number of angles in the field of view is 1, the robotic arm grabs that angle to lift the fabric into the air, swings the fabric, and then places it back.

2. The method for flattening fabric using a robotic arm based on a layered operation strategy according to claim 1, characterized in that, The detailed operation process is as follows: For several color images of fabric in different states, the grab points are labeled to form a training dataset, and the dataset is expanded. Using the ResNet network structure, the location of the grab point is determined from the current color image of the fabric; The placement trajectory is determined based on the position of the grab point, the pixel position of the fabric corner, and the pixel position of the fabric outline and edges.

3. The method for flattening fabric using a robotic arm based on a layered operation strategy according to claim 2, characterized in that, If the two sides of the fabric corner form a triangle with the contour determined by the Canny detector, then the robotic arm unfolds the fabric in a semi-circular trajectory with the contour as the unfolding axis and the vertical distance from the fabric corner to the contour as the radius. If there are no upper triangular folds, the placement trajectory is parallel to the working surface.

4. The method for flattening fabric using a robotic arm based on a layered operation strategy according to claim 3, characterized in that, When there are no upper-level triangular folds, the length of the placement vector is fixed, and the direction of the placement vector is taken as the average direction of the following two unit vectors: The visual quality center of the fabric points towards the gripping point; The grab point points to the nearest edge pixel.

5. A method for flattening fabric using a robotic arm based on a layered operation strategy according to claim 2, characterized in that, Expand the dataset by flipping horizontally, flipping vertically, rotating, and affine transformations.

6. A robotic arm flattening fabric system based on a layered operation strategy, characterized in that, include: The fabric information sensing module is used to acquire information about the edges and contours of the fabric. The robotic arm is used to perform coarse or fine operations on the fabric based on the fabric information obtained by the fabric information sensing module, and to feed back the current fabric information to the fabric information sensing module.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the robotic arm flattening cloth method based on the layered operation strategy as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the robotic arm flattening fabric method based on the layered operation strategy as described in any one of claims 1-5.

Images