Automatic sewing system and control method thereof

Abstract

An automatic sewing system and its control method are provided. The control method for the sewing system includes: obtaining visual information from one or more captured images of a fabric portion to be sewn; obtaining information about the sewing speed in the sewing direction from a sewing machine motor in the sewing system; generating a seam trajectory of the fabric portion to be sewn based on the information about the visual information and the information about the sewing speed; and manipulating the fabric portion using one or more manipulators in the sewing system based on the generated trajectory.

Description

Technical Field

[0001] This invention relates to an automatic sewing system and its control method. Background Technology

[0002] Garment production remains a labor-intensive industry, heavily reliant on manual labor by human operators. Sewing, accounting for approximately 85% of all textile bonding methods, plays a paramount role in the manufacture and shaping of garments. This technique is a crucial procedural component of garment construction and typically accounts for 35% to 40% of total production costs.

[0003] The quality of sewn items is closely related to the sewing operator's skill in precisely and deftly manipulating the position and stretch of fabric sections. A high level of skill is typically required to ensure a seamless fit where desired, while simultaneously preventing unwanted wrinkles or seam creases, both of which can easily lead to defective products.

[0004] Among various automation technologies, the use of robots offers enormous potential to replace manual sewing operations, and has therefore become an emerging area of ​​development in the automation industry. Several robotic sewing technology companies, such as KMF Maschinenbau, PFAFF Industrial, Softwear Automation, and Sewbo, have already incorporated robots with different configurations into their automated sewing systems to handle the manufacturing of different types of clothing or furniture embellishments. Figure 1 An example of a commercial robotic sewing system is shown. In the example, "SM" stands for sewing machine, "R" stands for robot, and "WIP" represents fabric workpiece.

[0005] However, current automated sewing systems directly use tracking error information to control the sewing process. This control strategy typically involves a pathpoint-by-path method for tracking the expected seam thread trajectory. Furthermore, engineers need to tune controller parameters to ensure good performance and sewing results before implementing existing sewing control schemes for different sewing tasks. Summary of the Invention

[0006] According to one aspect of this disclosure, a method for controlling a sewing system is provided, comprising: obtaining visual information from one or more captured images of a fabric portion to be sewn; obtaining information about a sewing speed in a sewing direction from a sewing machine motor in the sewing system; generating a seam trajectory of the fabric portion to be sewn based on the information about the visual information and the information about the sewing speed; and manipulating the fabric portion using one or more manipulators in the sewing system based on the generated trajectory.

[0007] In one example, time-scaling modeling is used to generate the seam line trajectory of the fabric section.

[0008] In one example, time-scaling modeling is a linear, time-scaling, non-holonomic kinematic model of the sewing process.

[0009] In one example, one or more captured images are captured by a vision sensor and include at least one reference trim of the fabric portion and the position of the sewing needle in the sewing system.

[0010] In one example, manipulating a fabric portion includes obtaining a desired rotational speed of the fabric portion around a needle based on a generated seam line trajectory; and controlling the one or more manipulators to manipulate the fabric portion based on the desired rotational speed of the fabric portion around the needle and the desired speed of the fabric portion.

[0011] In one example, manipulation of the fabric portion is also based on the current state of each of one or more manipulators.

[0012] In one example, the current state of each of the one or more manipulators includes at least one of the following: state information from its force sensor; current state information of its end effector; and desired state information of its end effector.

[0013] In one example, the seam line trajectory for generating the fabric portion to be sewn is also based on the desired margin size relative to the reference cut edge.

[0014] In one example, the generation of the seam path for the fabric portion to be sewn is also based on reference information on the fabric portion. In another example, the expected margin size is either a constant value or a variable defined by a specified function.

[0015] In one example, the reference information on the fabric portion includes at least one of printed lines, stitch lines, printed patterns, and stitch patterns.

[0016] In one example, the method also obtains an additional trajectory calculated from the coordinated motion control section based on inputs including actual force and torque signals from force sensors, desired external force and torque of the end effector, incremental information regarding the angle and linear position of the end effector, and velocity. In another example, the incremental information is obtained based on the difference between (i) the real-time angle and linear position and velocity of the end effector and (ii) the planned angle and linear position and velocity of the end effector calculated based on the generated (or planned) seam line trajectory; and one or more manipulators are controlled based on the generated trajectory and the additional trajectory.

[0017] In one example, one or more captured images include at least one reference trim of the fabric and the location of the sewing needle.

[0018] According to another aspect of this disclosure, a sewing system is provided, comprising: one or more manipulators, each manipulator including a force sensor and an end effector; one or more vision sensors; a sewing machine including a motor and a needle; a system controller including a vision module and a trajectory control module, the vision module being configured to obtain visual information from one or more images of a portion of fabric to be sewn captured by the vision sensors, and the trajectory control module being configured to perform: obtaining the visual information from the vision module; obtaining information about the sewing speed in the sewing direction from the sewing machine motor; generating a sewing trajectory for the portion of fabric to be sewn based on the information about the visual information and the information about the sewing speed; and manipulating the fabric portion using the one or more manipulators based on the generated trajectory.

[0019] According to another aspect of this disclosure, one or more non-transitory computer-readable media are provided that store instructions which, when executed by a processor of an electronic device, cause the electronic device to perform any of the operations described above. Attached Figure Description

[0020] Aspects, features, and advantages of this disclosure will become apparent from the following description of embodiments in conjunction with the accompanying drawings, in which: Figure 1 An example of a commercial robotic sewing system is shown; Figure 2 A diagram illustrating an example configuration of an automatic sewing system according to an embodiment of the present disclosure is shown; Figure 3 A block diagram of the main components of a system controller according to an embodiment of the present disclosure is shown; Figure 4 A block diagram illustrating a multi-level control architecture for automated sewing according to embodiments of the present disclosure is shown; and Figure 5 The sewing results with three different initial states are shown, with overall and magnified views of the stitches on the front (top) and back (bottom) sides of the fabric. Figure 6 The periodicity of the stitches is shown, derived from a digital image of the sewing result; Figure 7 A flowchart illustrating an operation method of a sewing system according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] To make the objectives, solutions, and advantages of the embodiments of this disclosure clearer, the solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this disclosure, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments described in this disclosure without creative effort are within the scope of protection of this invention.

[0022] In the field of automated sewing of textiles, precise control of the sewing path to align it with the intended seam is crucial for automated systems. Figure 1 An example of a commercial robotic sewing system is shown. Many commercial automated sewing systems require the use of fixing devices or clamps (such as...) Figure 1 The fixture or clamp shown in the upper left figure provides built-in channels / grooves to define the sewing path, allowing the sewing process to be operated through a simple procedure without considering feedback from the environment. However, this type of system requires changing the fixture for each different size or style of fabric section, resulting in tedious reconfiguration work between production runs, especially when dealing with small-batch, highly mixed production scenarios.

[0023] Using robots to directly manipulate fabric portions during sewing without the use of fixed devices is considered a flexible automation approach, better suited for custom garment production. However, controlling the fabric position relative to the sewing needle using robotic manipulators is crucial and often involves complex control schemes. Several control methods exist suitable for manipulating the position of an object, including adaptive control using different types of feedback during the manipulation process, visual servoing, impedance control, and learning-based control. The choice of control model depends on various factors, such as the manipulator's capabilities, the complexity of the task, and the desired level of control accuracy.

[0024] Among various control schemes, visual servoing is a popular technique where the controller uses real-time visual information from a camera to guide the movement of the manipulator. When it comes to trajectory control of an object, the camera typically images the actual position of the manipulated object based on extracted image features, and the system compares this position to a reference path point to determine the position error. The manipulator then applies corrective movements to the object to minimize any such errors. This process is repeated in continuous loops, causing the object to follow the desired trajectory. Visual feedback plays a crucial role in visual servoing by providing the system with real-time information for position control. To ensure effective and stable control performance, different types of visual feedback gains and weighting parameters need to be tuned or adjusted before operation. These include, for example, velocity gain, which determines how fast the manipulator should move to correct for errors when the operating speed changes. Higher velocity gain results in faster movements, but excessive gain can lead to instability or oscillations. Position error gain and image Jacobian gain, among others, also play a vital role in determining the behavior of visual servoing control, and all of these require meticulous tuning to ensure optimal performance.

[0025] Regarding automated sewing using a manipulator for fabric position control, we propose a control method involving “time-scaling” modeling of the sewing process. This model is operable to generate a desired sewing trajectory based on the desired seam line (which can be used interchangeably with the word “planning” in this context), and then convert it into a continuous trajectory for the end effector. Thus, sewing is performed along the desired seam line, without the need for progressively comparing actual positions and path points during operation, as is required in traditional visual servoing. A key benefit of the proposed method is the ability to achieve consistent sewing performance across a wide range of sewing speeds (provided the motion capabilities of the manipulator and sewing machine allow), where sewing performance refers to system rise time, convergence speed, settling time, etc., from a control perspective. Furthermore, the proposed method simplifies operation by eliminating the need to adjust various control parameters or weighting factors before initiating different sewing operations (e.g., sewing at different speeds), which are essential for traditional visual servoing.

[0026] According to one embodiment of the present disclosure, an automated sewing system is provided, which includes an industrial sewing machine, various sensing sensors (e.g., including vision sensors and force sensors) and at least one robot manipulator, each robot manipulator having an end effector for manipulating parts of fabric to facilitate sewing operations.

[0027] According to another embodiment of this disclosure, a control method is provided that involves modeling a sewing process as a nonholonomic constrained process using time scaling. This method models the kinematics of the sewing process in the desired seam direction domain rather than the time domain. This allows for independent control of the sewing speed and rotational speed of the fabric.

[0028] Based on the disclosed hardware and the proposed kinematic model of the sewing process, a feedback control scheme is designed to ensure essentially the same sewing performance and sewing thread results at different sewing speeds, provided that the feedback fabric rotation speed and sewing speed are limited by the hardware.

[0029] Specifically, based on time-scaling modeling of the sewing process and state feedback control, the sewing speed control and sewing direction control are separated in the design. Furthermore, if the feedback rotation speed is within limits, the designed system ensures that the same sewing results can be obtained at different sewing speeds without changing the control scheme and coefficients.

[0030] This advantage significantly improves the efficiency of sewing controllers designed for automated sewing processes. Specifically, it saves workload before different sewing tasks, where engineers typically need to adjust controller parameters of existing control schemes to ensure the obtained seam thread matches the desired seam thread. Instead, they only need to consider the maximum sewing speed by using the proposed method, ensuring the feedback rotation speed is within its maximum limit. As long as the feedback rotation speed is within the limit, the same sewing performance can be obtained across different sewing tasks.

[0031] By modeling the proposed sewing process and detecting the edge of the fabric or the desired sewing seam path through a vision module, the sewing machine controls the sewing speed, and the end effector controls the sewing direction by manipulating the fabric on the worktable to complete the sewing task and obtain the designed sewing thread.

[0032] This disclosure relates to an automated sewing system comprising an industrial sewing machine (or simply a sewing machine in the context), the industrial sewing machine including a needle for sewing one or more portions of fabric, a working surface, a manipulator (e.g., a robotic manipulator), an end effector for manipulating one or more portions of fabric, a force sensor, a vision sensor (such as a camera), and a system controller.

[0033] exist Figure 2 An exemplary physical structure of an automated sewing system is shown in the figure. Figure 2 In this context, an automatic sewing system (also simply called a sewing system) includes one or more manipulators, each manipulator including a force sensor and an end effector; and one or more vision sensors (in... Figure 2 The image shown is of a camera; a sewing machine, including a motor ( Figure 2 (Not shown) and a needle; and a system controller. The needle coordinate is defined as x, which is always parallel to the edge of the fabric. The end effector for moving the fabric material can have different designs and mechanisms depending on the type of fabric material being processed. For example, the end effector may include one or more vacuum cups that provide sufficient traction on the fabric, or a soft pad made of an elastic material such as rubber or foam. In this embodiment, we use a simple foam pad without an internal actuator as an exemplary end effector for manipulating the fabric.

[0034] A force sensor is positioned between the distal connector of the manipulator and the end effector to detect (i) the downward force applied to the fabric by the end effector, (ii) the lateral force associated with the tension in the fabric, so that these forces can be assessed and controlled during fabric manipulation, and (iii) the torque applied to the contact surface between the end effector and the working surface to adjust the posture of the end effector to obtain uniform force.

[0035] When operation begins, a portion of the fabric on the work surface is engaged by the end effector and subsequently brought to the needle of the sewing machine by the controlled movement of the manipulator. (Note that the fabric portion here can refer to a single piece of fabric or a stack of multiple pieces of fabric). An image of the fabric to be sewn (including at least one reference trim edge of the fabric and the position of the sewing needle) is always present at the vision sensor (e.g., as shown in the image). Figure 2 Under the monitoring of the camera shown, the vision sensor provides relevant visual data to the controller.

[0036] Figure 3 This is a block diagram showing the main components of the system controller. The system controller includes a processor (e.g., an Intel Central Processing Unit) and one or more memories storing control programs and data for calculations. The memories include modules such as a trajectory control module, a manipulator control module, a sewing motor control module, and a vision module. The trajectory control module, stored in the memories, controls the sewing path over time, the direction of which is manipulated by the movement of the manipulator relative to the sewing needle. The trajectory control module further communicates with the manipulator control module, the sewing motor control module, and the vision module. The manipulator control module implements control routines in conjunction with a servo controller, which is further connected to a force sensor (e.g., an ATI multi-axis F / T sensor) and a manipulator (e.g., a Denso 6-DoF industrial robot). The sewing motor control module controls the speed and stitch size of the sewing machine via another servo controller. On the other hand, the vision module processes image data from a digital camera (e.g., an optoelectronic monochrome flow camera) configured to detect the trim of fabric sections. The processed image data is then transmitted as an input signal to the trajectory control module.

[0037] Time scaling modeling of the sewing process

[0038] The key feature of this disclosure is the incorporation of a time-scaling modeling scheme into the control of the sewing process to achieve stable and precise stitches. Time-scaling modeling for kinematic systems involves manipulating time-scale variables in the kinematic equations describing the motions associated with the sewing process. The application of this modeling allows for (i) generating desired trajectories and (ii) precise control of the motion of fabric portions, including their position, velocity, and acceleration, over specified time periods.

[0039] To apply the aforementioned modeling, the kinematics of the sewing process is modeled as a nonholonomic constraint process. A nonholonomic constraint system is a system in which the number of control inputs (e.g., velocity) is less than the number of independent degrees of freedom or directions in which the system can move. Consider a sewing process, for example, where the fabric engaged by the feed teeth and presser foot of a sewing machine can move forward or backward or rotate around the sewing needle, but cannot move laterally, resulting in a nonholonomic constraint that restricts the possible movements of the fabric.

[0040] In the embodiment, the nonholonomic constrained kinematics of the sewing process can be modeled as follows:

[0041] in, s ω is the sewing speed in the sewing direction (i.e., the feeding direction of the feed teeth). s The rotational speed of the fabric around the needle. x The distance traveled by the needle tip (i.e., the position of the needle on the fabric during sewing, or the projection of the needle on the fabric when it is pointing upwards and not in contact with the fabric) along the desired sewing direction. y Let θ be the distance from the needle tip to the desired seam line, where the desired sewing direction is the tangent to the desired seam line at the point closest to the needle tip; θ represents the desired sewing direction relative to the coordinate system of the needle. n x n Angle between axes (reference) Figure 2 ), coordinate system n x in n axis, y n axis, z n The axes are perpendicular to each other. Coordinate system. n Defined as x n The axis points in the opposite sewing direction. Subsequently, a time scaling method is introduced into the above model to represent the sewing process more accurately and linearly. Taking a general nonlinear system with state transitions f(t, ξ(t), u(t)) as an example, the time scaling model is defined in this embodiment as:

[0042] Where ξ and u represent the system state and input, respectively, and the variables are... It is defined as a time-scaled variable, and the function η(ξ) is defined as a continuous-time scaling function. If If it is monotonically increasing (or decreasing) and ∞>η(ξ)>0 (or -∞<η(ξ)<0), then it satisfies the time scaling model.

[0043] To derive the time-scaled kinematic model of the proposed nonholonomic constrained sewing process, coordinate transformations z1=x, z2=tan θ, and z3=y are applied to transform the time-scaled kinematics. The corresponding transformation input is v1=-v s cos θ and v2=ω s / cos 2 θ. Therefore, the chain-like representation of the kinematics of the sewing process is given by the following equation:

[0044] Then, a time-scale variable satisfying the relation dt / dz1=1 / v1 can be introduced. =z1 and the time scaling function η(ξ)=v1(z1) are used to obtain the time-scaled kinematics. By substitution, the kinematic model is transformed from the time domain to the z1 domain, and the state-space model of the sewing process kinematics can be represented as follows:

[0045] μ1=v1 and μ2=v2 / v1 are defined as new inputs for model linearization.

[0046] Real-time sewing control architecture

[0047] In this embodiment, a time-scaling nonholonomic constrained kinematic model of the sewing process is used to control automatic sewing along a desired seam line that maintains a margin "m" relative to the aligned edges of two or more fabric layers without causing wrinkles. The margin "m" can be a constant value or a variable defined by a specified function.

[0048] use Figure 2 The exemplary configuration shown comprises three sub-tasks: (i) control of sewing speed, (ii) control of sewing direction, and (iii) control of fabric tension. The first two sub-tasks are primarily related to controlling the trajectory of the seam thread, while the third sub-task is crucial for ensuring the quality of the stitches. The basic workflow of the sewing process is outlined below.

[0049] During the sewing operation, one side of the fabric section, located between the presser foot and the feed teeth and engaged by them, is stitched at a specific feed speed controlled by the sewing machine. Simultaneously, the manipulator applies pressure against the sewing table on the other side of the fabric section via the end effector. This pressure generates static friction, which stabilizes the position of the fabric layers. This ensures that the end effector maintains non-slip static contact with the upper (topmost) sliding fabric, and that the individual fabric layers within the fabric stack remain stationary relative to each other. Therefore, the fabric position can be controlled by the movement of the end effector on the sewing table, which has a flat and smooth surface. The movement of the end effector is strategically used to control the feed angle of the fabric, i.e., the rotation around the sewing needle, thereby achieving the desired direction of the sewing seam. To prevent undesirable deformations such as wrinkles, bunching, and seam shrinkage, the end effector applies a controlled lateral force to the fabric in the opposite direction to the sewing during the feed motion. The resulting tension is crucial for avoiding wrinkles and seam shrinkage, and thus maintains the quality of the resulting stitches.

[0050] Figure 4 This is a block diagram illustrating a multi-level real-time control strategy for automated sewing operations. The architecture includes higher-level controllers (hereinafter referred to as the "upper control layer") and lower-level controllers (also called the "lower control layer"), each for a specific purpose. The upper control layer can be in a Windows OS node and can involve a feedback controller based on a time-scaling model (which corresponds to...). Figure 3 The system includes a trajectory control module and a vision sensor. The upper control layer of this control system is responsible for planning the sewing task. It receives real-time data about the ongoing sewing operation from the vision sensor and the sewing machine. Subsequently, it formulates the seam trajectory based on a linear time scaling model pre-described in the z1 domain, using a feedback controller based on the time scaling model. In this specific layer, the upper layer is controlled within the z1 domain. Specifically, the image I captured by the vision sensor at z1... k The input is used as the visual sensor part, which estimates z2 and z3. The sewing speed is read from the industrial sewing machine. z2 and z3 are provided to a feedback controller based on a time-scaling model to generate or plan trajectories based on the model. Using the inputs, v2 can be determined by the following formula:

[0051] Where k2 and k3 are the control coefficient and scaling factor, respectively.

[0052] Once the trajectory is generated, model-based control is based on the following relationship: the desired rotational speed ω is transmitted via shared memory. s Output to the lower control layer:

[0053] The lower control layer is responsible for synchronizing the rotation and feed motions, which yields the desired trajectory of the end effector, thus creating the seam line trajectory on the fabric. This lower control layer can reside within an Intime OS node and may involve a trajectory generator, manipulator, manipulator controller, one or more servo controllers, the industrial sewing machine, force sensors, and a coordinated motion control unit. The desired sewing speed v is received as an input signal. s and desired rotational speed ω s The lower control layer controls the industrial sewing machine, actuators, and force sensors, and coordinates motion control within this layer. First, using these inputs, the geometry-based trajectory generation component of this control layer achieves synchronized motion control and outputs the desired angular and linear velocities of the end effector, which can be represented by a 6 × 1 velocity twist vector relative to the world coordinate system. express.

[0054] To ensure precise coordination between feed motion and fabric tension during operation, the lower control layer also manages coordinated motion control. The coordinated motion control section is configured to receive multiple inputs, including (i) from force / torque sensors (in... Figure 4 The actual force and torque signals (displayed as F / T sensors) can be obtained from the 6x1 body wrench vector. (ii) The desired external force and torque of the end effector relative to the end effector coordinate system can be expressed by force spinor. (i) and (ii) incremental information about the angle and linear position of the end effector and its velocity relative to the end effector coordinate system, respectively, can be obtained from... In another example, based on the difference between (i) the real-time angle and linear position and velocity of the end effector of the manipulator and (ii) the planned angle and linear position and velocity of the end effector, incremental information about the angle and linear position and velocity of the end effector (respectively from...) The planned angles and linear positions, as well as velocities, of the end effector are obtained. In another example, the planned angles and linear positions, as well as velocities, of the end effector are calculated by a geometry-based trajectory generation component. The trajectory calculated from the coordinated motion control component can be defined as an "offset," "incremental," or "compensated" trajectory, compared to the master trajectory obtained from the geometry-based trajectory generation component. The coordinated motion control component then determines the incremental attitude and velocity displacements of the end effector relative to the end effector coordinate system (which can be obtained from...). This indicates that appropriate fabric tension is necessary to maintain the shape and flatness of the fabric layer during feeding and rotation. Before feedback is sent to the actuator controller section of the control loop, Further experience relative to the world coordinate system transformation and with Add them together.

[0055] Other key parameters used in the lower layer include the desired sewing speed, which serves as input to the trajectory generator section. The trajectory generator part will then determine the desired motor speed. A servo controller outputs to the industrial sewing machine motor. In another embodiment, the desired motor speed is... The input is fed to a servo controller, which generates electrical pulses, such as pulse width modulation signals, which are then sent to a servo mechanism. The servo mechanism, in turn, changes the speed of the motor in the industrial sewing machine. The trajectory obtained from the geometry-based trajectory generation section and the trajectory obtained from the coordinated motion control section (after transformation to world coordinates) are combined, and the combined result is used as input to the manipulator controller section, which sets the desired speed. The output is sent to the servo controller of the manipulator. In another embodiment, the desired speed... The output from the inverse Jacobian matrix is ​​further used as the input to the servo controller, where electrical pulses are generated to control the motors of the manipulator (which may also be called the robotic arm).

[0056] By implementing the above control strategy, this automatic sewing system is operable to plan the seam trajectory in real time using a time-scaling model, maintaining the desired stitch size regardless of changes in feed and rotational speeds. This model offers significant advantages for industrial applications because it eliminates the need for tedious adjustments to various parameters or weighting factors, which are required in traditional vision servo control schemes. Furthermore, the algorithm for this synchronous motion control is operable to prevent excessive force from being applied to the target fabric. To this end, when the sewing needle rises above the fabric portion, the end effector actuates the fabric portion only at specific intervals of the sewing cycle, and stops actuating when the needle is inserted into the fabric, thereby mitigating the risk of excessive localized damage to the fabric portion at the needle tip. It is evident that the system and method disclosed herein can be used for sewing operations on any two-dimensional fabric.

[0057] Experimental verification

[0058] Figure 5 Three different scenarios are illustrated, in which this sewing system performs automatic sewing of two aligned rectangular pieces of fabric along a predetermined seam line, wherein the initial positions of the fabric relative to the sewing needle are different. Sewing speed. The speed was set to 12 mm / s, and a constant edge distance of 20 mm from the longer edge of the fabric was predefined for the desired seam line. Experimental results showed that the seam line converged to a constant edge distance relative to the reference cut edge under different initial conditions. The initial edge distances for the three scenarios were approximately 15 mm, 20 mm, and 23 mm, respectively. In scenarios 1 and 3, the real-time control of the sewing system effectively achieved the desired edge distance with a minimum static error of 0.1 mm and no overshoot. In scenario 2, where the initial position error was 0.1 mm, the static error was measured at convergence as 0.1 mm.

[0059] Figure 6 The periodicity of the stitches, derived from digital images of the sewing results, is shown. Before and after convergence to a steady state, the stitch size for all the above scenarios was precisely controlled to approximately 2.4 mm. These results validate the accuracy of this sewing system and its ability to reach and stably maintain the desired state over time.

[0060] Figure 7 A flowchart illustrating an operation method of a sewing system according to an embodiment of the present disclosure is shown. Figure 7 In step 101, visual information is obtained from one or more captured images of the fabric portion to be sewn; in step 201, information about the sewing speed in the sewing direction is obtained from the sewing machine motor; in step 301, a sewing trajectory for the fabric portion is generated based on the information about the visual information and the information about the sewing speed. The information about the visual information can be I k This refers to the image captured by the vision sensor as described above. In another embodiment, the information about the vision can provide unique features on the fabric portion (e.g., lines or patterns, printed lines or patterns, or stitched lines or patterns) that can be intentionally used as a reference for the sewing trajectory to be sewn. In another embodiment, the sewing trajectory of the fabric portion to be sewn can be further based on a desired margin size relative to a reference cut edge, or alternatively, based on some other reference information that provides the trajectory clue (e.g., lines or patterns, printed lines or patterns, or stitched lines or patterns). In embodiments, the desired margin size can be predetermined in the sewing system or set as needed. And in step 401, the fabric portion is manipulated using one or more manipulators based on the generated trajectory.

[0061] Those skilled in the art will understand that the illustrative embodiments described above are not intended to be limiting. It should be understood that any two or more embodiments disclosed herein can be combined in any combination. Furthermore, other embodiments can be utilized and other changes can be made without departing from the spirit and scope of the subject matter set forth herein. It is readily understood that, as generally described herein and illustrated in the accompanying drawings, various aspects of the disclosed invention can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0062] Those skilled in the art will understand that the various illustrative logic blocks, modules, circuits, and steps described herein can be implemented in hardware, software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described around their functional sets. Whether such a set of features is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functional sets in different ways for each specific application, but such design decisions should not be construed as departing from the scope of this application.

[0063] The various illustrative logic blocks, modules, and circuits described in this application may be implemented or executed in a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0064] The steps of the methods or algorithms described herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, allowing the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in an electronic device. Alternatively, the processor and storage medium may reside as discrete components in the electronic device.

[0065] In one or more exemplary designs, functionality may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or code in a computer-readable medium. Computer-readable media include both computer storage media and communication media, with communication media including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible by a general-purpose or special-purpose computer.

[0066] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for controlling a sewing system, comprising: Visual information is obtained from one or more captured images of the fabric portion to be sewn; Information about the sewing speed in the sewing direction is obtained from the sewing machine motor in the sewing system; Based on information about the visual information and information about the sewing speed, a seam trajectory for the fabric portion to be sewn is generated; as well as Based on the generated trajectory, the fabric portion is manipulated using one or more manipulators in the sewing system.

2. The method according to claim 1, wherein, The seam line trajectory of the fabric section is generated based on time-scaling modeling.

3. The method according to claim 2, wherein, The time-scaling modeling is a linear, time-scaling, nonholonomic constrained kinematic modeling of the sewing process.

4. The method according to any one of claims 1-3, wherein the one or more captured images are captured by a vision sensor in the sewing system and include at least one reference cut edge of the fabric portion and the position of the sewing needle in the sewing system.

5. The method according to any one of claims 1-4, wherein manipulating the fabric portion comprises obtaining a desired rotational speed of the fabric portion around the needle based on a generated seam line trajectory; and controlling the one or more manipulators to manipulate the fabric portion based on the desired rotational speed of the fabric portion around the needle and the desired speed of the fabric portion.

6. The method according to any one of claims 1-5, wherein, Manipulation of the fabric portion is also based on the current state of each of the one or more manipulators.

7. The method of claim 6, wherein the current state of each of the one or more manipulators includes at least one of the following: state information from its force sensor; Its end effector's current status information; And the expected state information of its end effector.

8. The method according to any one of claims 1-6, wherein, The seam path for generating the fabric portion to be sewn is also based on the desired margin size relative to the reference cut edge.

9. The method according to claim 8, wherein, The desired margin size is either a constant value or a variable defined by a specified function.

10. The method according to any one of claims 1-9, wherein, The generation of the seam line trajectory for the fabric portion to be sewn is also based on reference information on the fabric portion.

11. The method according to claim 10, wherein, The reference information on the fabric portion includes at least one of printed lines, stitch lines, printed patterns, and stitch patterns.

12. The method according to any one of claims 1-11, wherein the one or more captured images include at least one reference cut edge of the fabric and the position of the sewing needle.

13. The method of claim 6 or 7, further comprising obtaining an additional trajectory calculated from a coordinated motion control unit based on inputs including actual force and torque signals from the force sensor, desired external force and torque of the end effector, real-time angle and linear position and velocity of the end effector and planned angle and linear position and velocity of the end effector based on the generated seam line trajectory; and controlling one or more manipulators based on the generated trajectory and the additional trajectory.

14. A sewing system, comprising: One or more manipulators, each manipulator including a force sensor and an end effector; One or more vision sensors; A sewing machine, including a motor and a sewing needle; The system controller includes a vision module and a trajectory control module. The vision module is configured to obtain visual information from one or more images of the fabric portion to be sewn captured by one or more vision sensors, and the trajectory control module is configured to execute: Visual information is obtained from the visual module; Information about the sewing speed in the sewing direction is obtained from the motor; Based on information about the visual information and information about the sewing speed, a sewing trajectory for the fabric portion to be sewn is generated; and The fabric portion is manipulated using one or more manipulators based on the generated trajectory.

15. The sewing system according to claim 14, wherein, The sewing system also includes a motion control component.

16. The sewing system according to claim 14 or 15, wherein, The sewing system is configured to perform the method according to any one of claims 2 to 13.

17. A non-transitory computer-readable medium storing one or more instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the operation of the method according to any one of claims 1 to 13.

Images