Real-time collaborative sewing method of multi-dimensional space robot with flexible fabric and complex process

Abstract

The present application relates to a kind of flexible fabric and complex process multidimensional space robot real-time collaborative sewing method, belong to industrial robot field, this method includes: hand-eye calibration is carried out to feeding robot, sewing robot, local camera and global camera;Feeding robot carries flexible material from initial position and spreads to workbench;Global camera identifies sewing starting point, and controls feeding robot to adjust incoming material to suitable position, realizes to incoming material initial positioning and position correction;Feeding robot is guided by vision to feed, sewing robot dynamically tracks and corrects sewing path to adapt to material slight pose change;Enable disturbance observer to correct the speed of sewing robot while informing feeding robot to reduce speed;Feeding robot realizes force control adjustment and compliant feeding during sewing process.The method described in the present application can realize high adaptability and high-precision work, real-time multi-robot collaboration, meet the stringent requirements of complex process on response speed.

Description

Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to a method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots. Background Technology

[0002] Currently, the field of robot-assisted sewing, especially for sewing tasks involving flexible materials and complex spatial structures, faces numerous challenges and technical bottlenecks. The main problems with existing technologies are as follows:

[0003] 1. Unstable material deformation and positioning: Flexible materials are prone to significant deformation during handling and sewing. Traditional methods that rely on fixed fixtures or simple positioning are difficult to balance efficiency and adaptability, resulting in fluctuations in sewing quality.

[0004] 2. Insufficient visual perception and path planning capabilities: Conventional robot vision systems are mostly used for two-dimensional planar operations, lacking the ability to perceive and adjust in real time the multi-curved surfaces and multi-pose sewing points in three-dimensional sewing processes with high precision.

[0005] 3. Lack of real-time control in multi-robot collaborative control: Existing feeding and sewing robots mostly use offline programming or simple trigger linkage, which cannot be flexibly adjusted based on real-time feedback during actual sewing, which can easily lead to material misalignment, sewing errors or even failure.

[0006] 4. Limited force control and compliance control methods: Faced with the uncertainty of flexible materials, existing systems usually lack tension detection and compliance control mechanisms during the feeding process, and cannot adapt to the subtle changes in tension requirements of different materials or different sewing stages. Summary of the Invention

[0007] The purpose of this invention is to at least address one of the aforementioned technical deficiencies.

[0008] Therefore, the purpose of this invention is to propose a real-time collaborative sewing method for flexible fabrics and complex processes using multi-dimensional space robots, which can achieve highly adaptable and high-precision work, real-time multi-robot collaboration, and meet the stringent requirements of complex processes for response speed.

[0009] To achieve the above objectives, embodiments of the present invention provide a method for real-time collaborative sewing of flexible fabrics and complex processes using a multi-dimensional space robot, comprising:

[0010] S1. Perform hand-eye calibration on the feeding robot, sewing robot, local camera and global camera to obtain the pose relationship between the local camera and global camera and the robot coordinate system. Mark the sewing feature points on the flexible material used for feeding. Input the feature points as parameters into the vision servo system so that during the sewing operation, the vision servo system will establish and adjust the robot motion model in real time according to the speed relationship between the feature points in the image and the target feature points.

[0011] S2, the loading robot uses visual feedback data from the global camera located above and moves along the guide rail to transport and spread flexible materials from the initial position to the worktable to achieve loading;

[0012] S3, the global camera begins to identify the sewing start point, uses template matching to locate the material pose, and controls the feeding robot to adjust the material to a suitable position, thereby achieving initial positioning and position correction of the material.

[0013] S4, the feeding robot uses an eye-to-hand and image-based or position-based visual servoing method to guide the feeding and calculate the end-effector speed V. The sewing robot uses an eye-to-hand and position-based visual servoing method to dynamically track and correct the sewing path to adapt to minute changes in the material's pose. When the local camera connected to the sewing robot detects that the displacement error of the feature point continuously exceeds a preset displacement threshold, and the visual servoing control cannot converge within a preset control cycle, the disturbance observer is activated to correct the speed of the sewing robot. The disturbance observation value of the disturbance observer is incorporated into the control equation of the sewing robot, and the feeding robot is notified to reduce its speed to reduce the intensity of the disturbance source.

[0014] S5, during the sewing process, the feeding robot monitors the tension of the flexible material in real time through a force sensor, and adaptively adjusts the feeding posture and feeding speed according to a preset torque threshold to achieve force control adjustment and compliant feeding.

[0015] Furthermore, in S2, the global camera is a binocular structured light 3D color camera, which is installed at an angle of 10° to 30° above the worktable, looking down at the worktable. The three-dimensional vision sensor serves as a global vision module and light source, used to establish a point cloud model and detect the position of the flexible material, and control the robotic arm of the loading robot.

[0016] Furthermore, if a three-dimensional sewing process is adopted, the incoming material is transported to a fixed support, the surface morphology of the material is reconstructed by a vision system, and the ICP algorithm is used to match a preset template.

[0017] Furthermore, in S4, the formula for the end effector speed V of the loading robot is:

[0018]

[0019] Where, λ T To control the gain, For translation of the Jacobian matrix The pseudo-inverse, s T For the current image features, Features of the target image.

[0020] Furthermore, in S4, the moving speed of the sewing feature point is:

[0021]

[0022] in, The velocity of feature points in the image. Obtained by image feature point localization, with units of pixels per second; Let f be the rate of change of the projected area, and l be the camera focal length. x l y The physical size of a pixel. f、l x l y Obtained through camera calibration; The velocity of a feature point in the actual physical world is the end-effector velocity of the sewing robot, measured in meters per second.

[0023] Furthermore, in S4, the disturbance observation value of the disturbance observer... for:

[0024]

[0025] in, The perturbation velocity of local visual feature points, The coordinates of the feature points detected by the local camera in the current frame, Δt is the control period, and L is the coordinates of the feature points detected by the local camera in the current frame. R Let Ω be the rotational Jacobian matrix for local vision. k-1 This is the robot angular velocity command from the previous control cycle.

[0026] Furthermore, the feeding robot is equipped with guide rails to achieve multi-position feeding, and the end of the feeding robot is equipped with a vacuum adsorption flexible gripper and a six-dimensional force sensor.

[0027] Furthermore, the end of the sewing robot is equipped with a sewing head that supports 3D contour sewing and a local camera, which is a 2D color industrial camera with a telecentric lens.

[0028] Furthermore, by using industrial Ethernet and high-speed shared memory mechanisms, real-time data synchronization is achieved between the feeding robot, sewing robot, local cameras, and global cameras, enabling multi-level collaborative real-time control.

[0029] Furthermore, it also includes the following steps: when abnormal material pose, abnormal tension, or loss of feature points is detected, the machine is quickly stopped and an alarm is triggered, supporting manual intervention or automatic adjustment and compensation, thus realizing an anomaly detection and self-recovery mechanism.

[0030] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0031] 1. High adaptability and high precision: Combining a hybrid vision servo system that integrates on-the-hand and off-the-hand perspectives, it balances local trajectory tracking accuracy with global deformation adaptability, making it suitable for sewing tasks on various flexible materials and in complex spaces. It employs a collaborative approach of 3D and 2D vision, combining on-the-hand and off-the-hand perspectives to compensate for the deformation of flexible materials in real time.

[0032] 2. Real-time Multi-Robot Collaboration: Based on force sensor feedback and real-time vision adjustment, the feeding and sewing processes are closely coordinated, effectively preventing material loosening and misalignment, and improving sewing quality and yield. A multi-level real-time collaborative control system is adopted to achieve high-speed data synchronization and decision-making among the feeding robot, sewing robot, vision system, and force control system, meeting the stringent response speed requirements of complex processes.

[0033] 3. Strong compliance control capability: Real-time fabric tension detection and compliance adjustment are achieved through a six-axis force sensor at the end effector, adapting to the different tension requirements of various fabrics (such as cotton, silk, and composite materials). The end effector force sensor compliance control mechanism of the feeding robot detects changes in material stress in real time and dynamically adjusts the feeding path and posture to maintain fabric stability.

[0034] 4. Intelligent Anomaly Handling: Integrating material anomaly detection and self-recovery mechanisms ensures long-term continuous and stable system operation, reduces the frequency of manual intervention, and improves the level of automation. Employing anomaly detection and adaptive compensation mechanisms for flexible materials, the system can quickly identify and take compensatory or protective actions when anomalies occur during feeding or sewing, improving system reliability.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0037] Figure 1 A flowchart of a real-time collaborative sewing method for flexible fabrics and complex processes using a multi-dimensional space robot, according to an embodiment of the present invention.

[0038] Figure 2This is a schematic diagram of a real-time collaborative sewing method for flexible fabrics and complex processes using a multi-dimensional space robot, according to an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of an implementation device for a real-time collaborative sewing method using flexible fabrics and complex processes with multi-dimensional space robots according to an embodiment of the present invention.

[0040] Among them, 1. Sewing robot; 2. Sewing machine head; 3. 2D color camera; 4. 3D color camera; 5. Feeding robot; 6. Vacuum adsorption flexible gripper and six-dimensional force sensor. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0042] This invention proposes a real-time collaborative sewing method for flexible fabrics and complex processes using multi-dimensional spatial robots. This method is a flexible fabric sewing method based on multi-dimensional spatial perception and real-time collaborative control of multiple robots, and uses hybrid visual servoing technology: ① two camera mounting methods: eye on the hand and eye outside the hand; ② image-based visual servoing (IBVS) and position-based visual servoing (PBVS).

[0043] refer to Figure 3 The real-time collaborative sewing method for flexible fabrics and complex processes using multi-dimensional space robots of the present invention employs the following hardware equipment:

[0044] (1) Loading robot

[0045] A seven-axis collaborative robot was selected as the feeding robot. It can be equipped with guide rails to achieve multi-position feeding, and its end effector is equipped with a vacuum suction flexible gripper and a six-dimensional force sensor. The force sensor on the feeding robot is used to detect the fabric tension in real time.

[0046] (2) Sewing robot

[0047] A six-axis industrial robot was selected as the sewing robot, equipped with a special sewing head (supporting 3D contour sewing) and a color industrial camera with a telecentric lens at its end effector. The sewing robot's end effector camera is a high-resolution, low-latency color industrial camera used to accurately identify feature markers on the sewing path.

[0048] (3) Global vision module and its light source

[0049] A three-dimensional vision sensor (binocular structured light 3D color camera) is selected and installed at an angle of 10° to 30° above the worktable to view the worktable. It is used to build a point cloud model and detect the position of flexible materials, thereby controlling the loading robot arm.

[0050] The camera above the worktable is a high-precision structured light 3D camera with a wide field of view and high Z-axis resolution, ensuring accurate measurement of the fabric's global deformation. The tilted installation can expand the effective field of view of the 3D camera (covering the entire worktable and the feeding area), while compensating for pose mapping errors.

[0051] (4) Local vision module and its light source

[0052] A two-dimensional vision sensor (2D color industrial camera) is selected, mounted on the robot flange, and kept parallel to the working mechanism to monitor the movement of the marker points on the flexible material, thereby controlling the movement of the sewing robot arm.

[0053] This invention employs an industrial control unit and a vision processing unit as a core control platform to process visual information and control robot movement using visual servo technology. The entire system requires hand-eye calibration to obtain the pose relationship between the two cameras and the robot's coordinate system.

[0054] like Figure 1 and Figure 2 As shown in the figure, the real-time collaborative sewing method of flexible fabric and complex process multi-dimensional space robot in this invention includes the following steps:

[0055] S1 involves performing hand-eye calibration on the feeding robot 5, sewing robot 1, local cameras, and global cameras to obtain the pose relationships between the local and global cameras and the robot coordinate system. Sewing feature points are then marked on the flexible material used for feeding. These feature points are input as parameters into the visual servo system. During the sewing process, the visual servo system establishes and adjusts the robot's motion model in real time based on the velocity relationship between feature points in the image and target feature points. The flexible material can be, for example, fabric, cloth, or woven material.

[0056] refer to Figure 3 The feeding robot 5 is equipped with guide rails to achieve multi-position feeding. The end of the feeding robot 5 is equipped with a vacuum adsorption flexible gripper and a six-dimensional force sensor 6.

[0057] The sewing robot 1 is equipped with a sewing head 2 that supports 3D contour sewing and a local camera at its end. The local camera is a 2D color industrial camera with a telecentric lens.

[0058] S2, the loading robot 5 uses visual feedback data from the global camera located above, along with the movement of the guide rail, to transport and spread the flexible material from its initial position to the worktable, thus achieving loading.

[0059] The global camera uses a binocular structured light 3D color camera 4, which is mounted at an angle of 10° to 30° above the worktable, looking down at it. The 3D vision sensor serves as the global vision module and light source, used to build a point cloud model and detect the position of the flexible material, controlling the robotic arm of the loading robot 5. That is, based on the 3D vision feedback from above, the loading robot 5 moves along the guide rail to transport and spread the flexible material from its initial position onto the worktable.

[0060] Specifically, the 3D vision sensor reconstructs a 3D point cloud model by acquiring depth images. The vision system filters and segments the point cloud, extracts the edges and feature regions of the flexible material, and calculates its spatial position and orientation in the worktable coordinate system through plane fitting. Combining the hand-eye calibration results, the point cloud pose is transformed into the robot coordinate system, guiding the loading robot 5 to execute the loading trajectory.

[0061] S3, the global camera begins to identify the sewing start point, uses template matching to locate the material pose, and controls the feeding robot 5 to adjust the material to a suitable position, thus achieving initial positioning and position correction of the material.

[0062] This step completes the initial positioning and position correction. The global camera above begins to identify the sewing start point, uses template matching to locate the material's pose, and controls the loading robot to adjust it to the appropriate position.

[0063] If a three-dimensional sewing process is adopted, the incoming materials need to be transported and placed on a fixed support, the surface morphology of the material is reconstructed by the vision system, and the ICP algorithm is used to match the preset template.

[0064] After the 3D point cloud is established in S2, the vision system uses the ICP (Iterative Closest Point) algorithm to iteratively calculate the optimal rigid transformation between the point cloud and the template point cloud, including rotation and translation, to align them in 3D space. ICP accurately obtains the pose change matrix between the actual pose of the incoming material and the template by continuously matching corresponding points and minimizing distance errors. Based on this matrix, the loading robot is controlled to make fine adjustments, achieving high-precision position correction and pose alignment before the 3D sewing operation.

[0065] The following section explains the hybrid vision servo control of the sewing process in conjunction with step S4.

[0066] S4, the feeding robot 5 adopts an eye-to-hand approach and image-based visual servoing (IBVS) or position-based visual servoing (PBVS) to guide the feeding and calculate the end-effector speed V of the feeding robot 5.

[0067] It's worth noting that the feeding robot 5 uses either IBVS or PBVS, depending on whether it's 3D sewing. PBVS is used for 3D sewing; otherwise, IBVS is used. 2D or 3D vision guides the feeding process, ensuring accurate local positioning of the material during sewing.

[0068] The formula for the end velocity V of the feeding robot 5 is:

[0069]

[0070] Where, λ T To control the gain, For translation of the Jacobian matrix The pseudo-inverse, s T For the current image features, Features of the target image.

[0071] The sewing robot 1 employs an eye-to-hand and IBVS (Independent Isostat-Based Visualization) approach to dynamically track and correct the sewing path to adapt to minute changes in the material's pose. This is because the sewing robot 1 only needs to operate within a small area, requiring only minor adjustments to its trajectory; within such a localized area, 3D information is unnecessary. Larger-scale material movement is achieved through a loading robot.

[0072] The moving speed of the sewing feature point is:

[0073]

[0074] in, The velocity of feature points in the image. Obtained by image feature point localization, with units of pixels per second; Let f be the rate of change of the projected area, and l be the camera focal length. x l y The physical size of a pixel. f、l x l y Obtained through camera calibration; The velocity of a feature point in the actual physical world is the end-effector velocity of the sewing robot, measured in meters per second.

[0075] This represents the velocity of a feature point in the real physical world, specifically the end effector velocity of the sewing robot. In the formula... Given the camera intrinsic parameter matrix, multiplying both sides of the equation by the inverse of the intrinsic parameter matrix on the left yields the result.

[0076] Visual servo compensation control:

[0077] The movement or sewing pressure of the feeding robot 5 can cause fabric stretching, which can lead to the sewing point detected by the end-effector camera deviating from the expected position. When the local camera connected to the sewing robot 1 detects that the feature point displacement error continuously exceeds a preset displacement threshold, and the visual servo control cannot converge within a preset control cycle (e.g., 3 cycles), the disturbance observer is activated to correct the speed of the sewing robot 1. The local camera is a 2D color industrial camera 3.

[0078] The disturbance observation values ​​of the disturbance observer The control equations of the sewing robot 1 are incorporated into the control equations, and the feeding robot 5 is instructed to reduce its speed to reduce the intensity of the disturbance source.

[0079] Among them, the disturbance observations of the disturbance observer for:

[0080]

[0081] in, The perturbation velocity of local visual feature points, Let Δt be the coordinates of the feature points detected by the local camera in the current frame.

[0082] To control the cycle, L R Let Ω be the rotational Jacobian matrix for local vision. k-1 This is the robot angular velocity command from the previous control cycle.

[0083] S5, the feeding robot 5 monitors the tension F of the flexible material in real time through the force sensor 6 during the sewing process, and adaptively adjusts the feeding posture and feeding speed according to the preset torque threshold to prevent the fabric from wrinkling and tearing, thus achieving force control adjustment and smooth feeding.

[0084] By using industrial Ethernet and high-speed shared memory, real-time data synchronization is achieved between the feeding robot 5, the sewing robot 1, the local camera, and the global camera. The overall response cycle is less than 50ms, ensuring high dynamic performance of the system and enabling multi-level collaborative real-time control.

[0085] Anomaly detection and self-recovery mechanism: When abnormal material pose, abnormal tension, or loss of feature points is detected, the machine will quickly stop and alarm, supporting manual intervention or automatic adjustment and compensation, thus realizing the anomaly detection and self-recovery mechanism.

[0086] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0087] It will be readily understood by those skilled in the art that this invention includes any combination of the inventive description and specific embodiments outlined in the foregoing specification, as well as the various parts shown in the accompanying drawings. Due to space limitations and for the sake of brevity, not all of these combinations have been described in detail. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0088] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots, characterized in that, Includes the following steps: S1. Perform hand-eye calibration on the feeding robot, sewing robot, local camera and global camera to obtain the pose relationship between the local camera and global camera and the robot coordinate system. Mark the sewing feature points on the flexible material used for feeding. Input the feature points as parameters into the vision servo system so that during the sewing operation, the vision servo system will establish and adjust the robot motion model in real time according to the speed relationship between the feature points in the image and the target feature points. S2, the loading robot uses visual feedback data from the global camera located above and moves along the guide rail to transport and spread flexible materials from the initial position to the worktable to achieve loading; S3, the global camera begins to identify the sewing start point, uses template matching to locate the material pose, and controls the feeding robot to adjust the material to a suitable position, thereby achieving initial positioning and position correction of the material. S4, the feeding robot uses an eye-to-hand and image-based or position-based visual servoing method to guide the feeding and calculate the end-effector speed V. The sewing robot uses an eye-to-hand and position-based visual servoing method to dynamically track and correct the sewing path to adapt to minute changes in the material's pose. When the local camera connected to the sewing robot detects that the displacement error of the feature point continuously exceeds a preset displacement threshold, and the visual servoing control cannot converge within a preset control cycle, the disturbance observer is activated to correct the speed of the sewing robot. The disturbance observation value of the disturbance observer is incorporated into the control equation of the sewing robot, and the feeding robot is notified to reduce its speed to reduce the intensity of the disturbance source. Among them, the disturbance observation value of the disturbance observer for: in, The perturbation velocity of local visual feature points, Let Δt be the coordinates of the feature points detected by the local camera in the current frame. To control the cycle, For local vision, the rotation Jacobian matrix is... The robot's angular velocity command from the previous control cycle; S5, during the sewing process, the feeding robot monitors the tension of the flexible material in real time through a force sensor, and adaptively adjusts the feeding posture and feeding speed according to a preset torque threshold to achieve force control adjustment and compliant feeding. The multi-dimensional space robot real-time collaborative sewing method for flexible fabrics and complex processes achieves real-time data synchronization between the feeding robot, sewing robot, local camera and global camera through industrial Ethernet and high-speed shared memory mechanism, realizing multi-level collaborative real-time control.

2. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, In S2, the global camera is a binocular structured light 3D color camera, which is installed at an angle of 10° to 30° above the worktable, looking down at the worktable. The binocular structured light 3D color camera serves as a global vision module and light source, used to establish a point cloud model and detect the position of the flexible material, and control the robotic arm of the loading robot.

3. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, If a three-dimensional sewing process is adopted, the incoming material is transported to a fixed support, the surface morphology of the material is reconstructed by the vision system, and the ICP algorithm is used to match the preset template.

4. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, In S4, the formula for the end effector speed V of the loading robot is: in, To control the gain, For translation of the Jacobian matrix The false rebellion, For the current image features, Features of the target image.

5. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, In S4, the moving speed of the sewing feature point is: in, , The velocity of feature points in the image. , Obtained by image feature point localization, with units of pixels per second; Let f be the rate of change of the projected area, and f be the camera focal length. , The physical size of a pixel. f , Obtained through camera calibration; , The velocity of a feature point in the actual physical world is the end-effector velocity of the sewing robot, measured in meters per second.

6. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, The feeding robot is equipped with guide rails to enable multi-position feeding, and the end of the feeding robot is equipped with a vacuum adsorption flexible gripper and a six-dimensional force sensor.

7. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, The sewing robot is equipped with a sewing head and a local camera at its end, which are 2D color industrial cameras with telecentric lenses.

8. The method for real-time collaborative sewing of flexible fabrics and complex processes using multi-dimensional space robots according to claim 1, characterized in that, It also includes the following steps: when abnormal material orientation, abnormal tension, or loss of feature points are detected, the machine is quickly stopped and an alarm is triggered, supporting manual intervention or automatic adjustment and compensation, thus realizing an anomaly detection and self-recovery mechanism.

Images