Control method and control system of grabbing equipment and grabbing equipment

Abstract

The invention provides a control method and system of grabbing equipment and the grabbing equipment. The control method of the grabbing equipment comprises the steps that an original image of a cloth bearing device used for bearing cloth to be grabbed is obtained; correcting a target monitoring area in the original image into a rectangular shape through perspective transformation processing to obtain a to-be-processed image; identifying a target grabbing position of to-be-grabbed cloth in the to-be-processed image, wherein the to-be-grabbed cloth is located in the target monitoring area; and controlling the grabbing equipment to grab the target grabbing position of the to-be-grabbed cloth. Distortion of an original image is corrected through perspective transformation, a target monitoring area is corrected to be in a regular rectangular shape, the processed image can accurately reflect the actual position and shape of cloth, and interference of perspective distortion on cloth position measurement is effectively eliminated. And the target grabbing position of the to-be-grabbed cloth is recognized based on the corrected to-be-processed image, and the grabbing device is controlled to grab the target grabbing position, so that the accuracy of cloth grabbing control is improved.

Description

Technical Field

[0001] This invention relates to the field of textile technology, and in particular to a control method, control system and gripping device for a gripping device. Background Technology

[0002] In the textile and apparel manufacturing industry, the fabric gripping and stacking process has long relied on manual operation, which is not only labor-intensive but also inefficient and costly. To automate this process, related technologies use cameras to capture images of the fabric area and control the gripping equipment by identifying the fabric's position within the image. However, in practical applications, the lack of standardized camera installation locations leads to image distortion, causing the images to fail to accurately reflect the actual position and shape of the fabric. This results in positioning errors in the gripping equipment, severely impacting the accuracy of fabric gripping. Summary of the Invention

[0003] This application provides a control method, control system, and gripping device for a gripping device to improve the accuracy of fabric gripping, thereby meeting the precision requirements of automated production.

[0004] This application provides a control method for a gripping device, comprising: obtaining an original image of a fabric carrying device for carrying fabric to be gripped; correcting the target monitoring area in the original image into a rectangular shape through perspective transformation processing to obtain an image to be processed; identifying the target gripping position of the fabric to be gripped in the image to be processed, wherein the fabric to be gripped is located in the target monitoring area; and controlling the gripping device to grip the target gripping position of the fabric to be gripped.

[0005] Optionally, the target monitoring area has multiple marked corner points, which can be connected in the fabric carrying device to form a rectangle, and the multiple marked corner points form an irregular quadrilateral in the original image; through perspective transformation processing, the target monitoring area in the original image is corrected into a rectangular shape to obtain the image to be processed, including: determining the coordinates of multiple marked corner points in the original image in a first image coordinate system; defining multiple corner points in a second image coordinate system that correspond one-to-one with each marked corner point, and determining the coordinates of the multiple corner points in the second image coordinate system; the multiple corner points in the second image coordinate system can be connected to form a rectangle; determining the mapping relationship between the first image coordinate system and the second image coordinate system based on the coordinates of each marked corner point in the first image coordinate system and the coordinates of each corner point corresponding to each marked corner point in the second image coordinate system; for multiple pixels in the original image, mapping them to the second image coordinate system according to the mapping relationship to obtain the image to be processed; wherein, the multiple pixels in the original image include at least the multiple pixels of the target monitoring area.

[0006] Optionally, identifying the target gripping location of the fabric to be gripped in the image to be processed includes: identifying the area where the fabric to be gripped is located in the image to be processed; and determining the target gripping location of the fabric to be gripped within the area where the fabric to be gripped is located.

[0007] Optionally, in the area where the fabric to be grabbed is located, the target grab position of the fabric to be grabbed is determined, including: determining the centroid of the fabric to be grabbed based on the outer contour of the area where the fabric to be grabbed is located, and using the centroid as the target grab position.

[0008] Optionally, after correcting the target monitoring area in the original image into a rectangular shape through perspective transformation processing to obtain the image to be processed, the control method further includes: identifying the tilt angle of the fabric to be grasped in the image to be processed; controlling the target grasping position of the grasping device to grasp the fabric to be grasped, including: controlling the target grasping position of the grasping device to grasp the fabric to be grasped according to the tilt angle.

[0009] Optionally, identifying the tilt angle of the fabric to be grasped in the image to be processed includes: identifying the area where the fabric to be grasped is located in the image to be processed; and determining the tilt angle of the smallest bounding rectangle of the area where the fabric to be grasped is located, as the tilt angle of the fabric to be grasped.

[0010] Optionally, identifying the region of the fabric to be grasped in the image to be processed includes: performing inter-frame difference operation on the image to be processed to obtain a difference image representing pixel differences; identifying the foreground region and background region based on the difference image; and determining the region of the fabric to be grasped based on the foreground region.

[0011] Optionally, identifying foreground and background regions based on the difference image includes converting the difference image into a binary mask to identify the foreground and background regions.

[0012] Optionally, converting the difference image into a binary mask to identify foreground and background regions includes: converting the difference image into an initial binary mask; preprocessing the initial binary mask to obtain a preprocessed binary mask; and identifying foreground and background regions based on the preprocessed binary mask; wherein the preprocessing includes erosion and dilation operations, and the dilation operation is performed after the erosion operation is completed.

[0013] Optionally, the area where the fabric to be grabbed is located can be determined based on the foreground area, including: filtering out independent areas in the foreground area whose area is greater than an area threshold, as the area where the fabric to be grabbed is located.

[0014] Optionally, the target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction, causing the fabric to be grasped to move within the target monitoring area; after correcting the target monitoring area in the original image into a rectangular shape through perspective transformation processing to obtain the image to be processed, the control method further includes: determining the target time for the fabric to be grasped to reach the target grasping position based on the image to be processed; controlling the grasping device to grasp the target grasping position of the fabric to be grasped, including: controlling the grasping device to grasp the target grasping position of the fabric to be grasped at the target time.

[0015] Optionally, the coordinates of the image to be processed are based on a second image coordinate system, and the preset direction is along the horizontal axis of the second image coordinate system; the gripping device performs gripping within a set gripping area; based on the image to be processed, the target time for the fabric to be gripped to reach the target gripping position is determined, including: obtaining the lateral distance between the target gripping position and the gripping area along the horizontal axis of the second image coordinate system; obtaining the movement speed of the fabric to be gripped along the horizontal axis of the second image coordinate system; and determining the target time for the fabric to be gripped to reach the target gripping position based on the lateral distance and the movement speed.

[0016] Optionally, the target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction to move the fabric to be grasped within the target monitoring area; and the grasping device performs grasping within a set graspable area; controlling the grasping device to grasp the target grasping position of the fabric to be grasped includes: determining the grasping operation point when the target grasping position moves to the graspable area based on the target grasping position of the fabric to be grasped; and controlling the grasping device to grasp the fabric to be grasped at the grasping operation point.

[0017] Optionally, the coordinates of the image to be processed are based on a second image coordinate system, and the control of the gripping device is based on a mechanical coordinate system; the preset direction is along the horizontal axis of the second image coordinate system, and the horizontal coordinate of the grippable area is fixed; based on the target gripping position of the fabric to be gripped, the gripping operation point when the target gripping position moves to the grippable area is determined, including: based on the target gripping position of the fabric to be gripped, the coordinates of the gripping operation point in the second image coordinate system are determined; through coordinate system transformation, the coordinates of the gripping operation point in the second image coordinate system are converted to the coordinates in the mechanical coordinate system; wherein, the horizontal coordinate of the gripping operation point in the second image coordinate system is the horizontal coordinate of the grippable area in the second image coordinate system, and the vertical coordinate of the gripping operation point in the second image coordinate system is the vertical coordinate of the target gripping position in the second image coordinate system.

[0018] This application provides a control system for a gripping device, including one or more processors for implementing the aforementioned control method for the gripping device.

[0019] This application provides a gripping device, including: a gripping device body, and a control system of the aforementioned gripping device, disposed on the gripping device body.

[0020] The control method, control system, and gripping device provided in this application, after obtaining the original image of the fabric-carrying device used to carry the fabric to be gripped, correct the distortion of the original image through perspective transformation, correcting the target monitoring area into a regular rectangular shape. This ensures that the processed image accurately reflects the actual position and shape of the fabric, effectively eliminating the interference of perspective distortion on the fabric position measurement and providing a good foundation for accurate identification of the target gripping position. Based on the corrected image to be processed, the target gripping position of the fabric to be gripped is identified, and the gripping device is controlled to grip the target gripping position to achieve the gripping of the fabric to be gripped. This helps to improve the accuracy of fabric gripping control, thereby meeting the precision requirements of automated production. Attached Figure Description

[0021] Figure 1 This is a schematic flowchart illustrating a control method for a gripping device according to an embodiment of this application; Figure 2 This is a flowchart illustrating a control method for a gripping device according to another embodiment of this application; Figure 3 This is a schematic diagram of the original image provided in one embodiment of this application; Figure 4 This is a schematic diagram of an image to be processed provided in one embodiment of this application; Figure 5 This is a flowchart illustrating a control method for a gripping device according to another embodiment of this application; Figure 6 This is a schematic diagram of the image to be processed provided in another embodiment of this application; Figure 7 This is a schematic diagram of the area where the fabric to be grasped is located, provided in one embodiment of this application; Figure 8 This is a flowchart illustrating a control method for a gripping device according to another embodiment of this application; Figure 9 This is a flowchart illustrating a control method for a gripping device according to another embodiment of this application; Figure 10 This is a schematic diagram of the image to be processed provided in another embodiment of this application; Figure 11 This is a flowchart illustrating a control method for a gripping device provided in another embodiment of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings.

[0023] This application provides an application scenario for a control method for a gripping device. This application scenario includes a gripping device, an image acquisition device, and a fabric support device.

[0024] The fabric to be grasped is placed on the support surface of the fabric carrying device. An image acquisition device is used to acquire an original image of the fabric carrying device, including at least a portion of the support surface, to obtain an image of the fabric to be grasped placed on the support surface. The image acquisition device is, for example, a camera. The grasping device includes, for example, a robotic arm and grippers. The position of the grippers can be adjusted by controlling the posture of the robotic arm, and the grippers ultimately grasp the fabric to be grasped.

[0025] In some embodiments, at least a portion of the fabric carrying device slides along a preset direction to move the fabric to be grasped within the target monitoring area. Specifically, in the fabric carrying device, at least the carrying surface can slide along the preset direction, and the fabric to be grasped placed on the carrying surface moves as the carrying surface moves.

[0026] This application provides a control method for a gripping device, which can be executed in the aforementioned application scenarios. Combined with... Figure 1 As shown, the control method includes steps S10 to S40.

[0027] Step S10: Obtain the original image of the fabric carrier device used to carry the fabric to be grasped.

[0028] The original image is acquired using an image acquisition device.

[0029] Step S20: Through perspective transformation, the target monitoring area in the original image is corrected into a rectangular shape to obtain the image to be processed.

[0030] Step S30: Identify the target grasping position of the fabric to be grasped in the image to be processed. The fabric to be grasped is located in the target monitoring area.

[0031] The target grab location here is a portion of the fabric to be grabbed.

[0032] Step S40: Control the gripping device to grip the target gripping position of the fabric to be gripped.

[0033] The control method for the gripping device provided in this application obtains the original image of the fabric-carrying device used to hold the fabric to be gripped. Then, perspective transformation is used to correct the distortion of the original image, correcting the target monitoring area into a regular rectangular shape. This ensures that the processed image accurately reflects the actual position and shape of the fabric, effectively eliminating the interference of perspective distortion on fabric position measurement and providing a good foundation for accurate identification of the target gripping position. Based on the corrected image, the target gripping position of the fabric to be gripped is identified, and the gripping device is controlled to grip the target gripping position to achieve the gripping of the fabric. This improves the accuracy of fabric gripping control, thereby meeting the precision requirements of automated production.

[0034] In some embodiments, the target monitoring area has multiple marked corner points. These marked corner points can be connected in the fabric carrying device to form a rectangle, and in the original image, they also form a non-rectangular quadrilateral. In some embodiments, four or more marked corner points are provided, with four of these points being the four corners of the rectangle formed by connecting the marked corner points. This allows for precise positioning of the boundary of the target monitoring area, providing clear reference points for subsequent perspective transformation, thereby ensuring that the distorted target monitoring area is accurately corrected into a regular rectangular shape. When the number of marked corner points is greater than four, additional marked corner points can be added for redundancy verification, thereby improving the stability of perspective transformation.

[0035] Combination Figure 2 As shown, the aforementioned step S20 corrects the target monitoring area in the original image into a rectangular shape through perspective transformation processing to obtain the image to be processed, including steps S201 to S204.

[0036] Step S201: Determine the coordinates of multiple marked corner points in the original image in the first image coordinate system.

[0037] Step S202: Define multiple corner points in the second image coordinate system that correspond one-to-one with each marked corner point, and determine the coordinates of the multiple corner points in the second image coordinate system; the multiple corner points in the second image coordinate system can be connected to form a rectangle. Step S203: Determine the mapping relationship between the first image coordinate system and the second image coordinate system based on the coordinates of each marked corner point in the first image coordinate system and the coordinates of each corner point corresponding to each marked corner point in the second image coordinate system.

[0038] Step S204: For multiple pixels in the original image, map them to the second image coordinate system according to the mapping relationship to obtain the image to be processed.

[0039] The multiple pixels in the original image include multiple pixels within the target monitoring area. Furthermore, the multiple pixels in the original image include all pixels within the area defined by multiple marked corner points.

[0040] The original image is represented in a first image coordinate system, and the image to be processed after perspective transformation is represented in a second image coordinate system. In the first image coordinate system, the coordinates of multiple marked corner points in the original image are determined. In the second image coordinate system, multiple corresponding corner points are defined and their coordinates are specified. The multiple marked corner points in the first image coordinate system correspond one-to-one with the multiple defined corner points in the second coordinate system, providing a precise and unique coordinate reference for perspective transformation, thereby ensuring the accurate establishment of the mapping relationship between the first and second image coordinate systems. Based on this mapping relationship, the coordinate points in the region enclosed by multiple marks in the original image can be determined in the second image coordinate system. By mapping all pixels in at least the target monitoring area in the original image to the second image coordinate system according to the mapping relationship, the target monitoring area with perspective distortion can be completely corrected into a regular rectangular shape, resulting in the image to be processed.

[0041] By employing precise coordinate system transformation, distortion correction can be achieved while preserving crucial information about the target monitoring area in the image, namely, the position and shape information of the fabric to be grasped. This helps ensure the accuracy of subsequent grasping control.

[0042] like Figure 3 and Figure 4 As shown, Figure 3 A schematic diagram of the original image of the fabric carrier used to hold the fabric to be grasped. Figure 3 The original image shown contains the fabric carrier 10 and the target detection area P within the fabric carrier 10. A Four corner markers, A1, A2, A3, and A4, are set.

[0043] For example Figure 4 To Figure 3 The image to be processed is obtained by performing perspective transformation on the original image. The target monitoring area in the image to be processed is P. B . Figure 4 The definition in the middle has the same Figure 2Each marked corner point corresponds one-to-one with multiple corner points B1, B2, B3, and B4. Specifically, A1 corresponds to B1, A2 to B2, A3 to B3, and A4 to B4. Based on the coordinates of A1, A2, A3, and A4 in the first image coordinate system and the coordinates of B1, B2, B3, and B4 in the defined second image coordinate system, the mapping relationship between the first and second image coordinate systems can be determined. Based on this mapping relationship, after processing the original image, the desired result can be obtained. Figure 4 The image to be processed is shown.

[0044] This section provides a more detailed explanation of the perspective transformation process.

[0045] (1) Let the first image coordinate system be... The second image coordinate system is Four known ArUco (Augmented Reality University of Cordoba) marker corners were selected from the original image as source points. These points form a non-rectangular, irregular quadrilateral in the original image, which is used as the target detection area. The coordinates of the marker corners are represented as follows: Where each coordinate It corresponds to a specific ArUco marker corner.

[0046] (2) In the second image coordinate system, define a regular rectangle with the coordinates of its four corner points as follows: in, and These represent the width and height in pixels of the target detection region in the second coordinate system, respectively. ) represents the width of the target detection region in the second coordinate system, ( ) represents the height of the target detection area in the second coordinate system.

[0047] (3) Solve for the perspective transformation matrix using four pairs of corresponding points. The mathematical form of the perspective transformation is: in, These are homogeneous coordinates; the actual coordinates are: Substituting the four pairs of corresponding points into the above equation, we get the following equation: Rearranging it into a system of linear equations, it is as follows: By solving this system of linear equations, the perspective transformation matrix is ​​obtained. The 8 parameters.

[0048] (4) For each pixel in the original image Apply the transformation matrix: .

[0049] After normalization, we obtain: .

[0050] By using methods such as bilinear interpolation, the transformed coordinates are mapped onto a regular rectangular region.

[0051] The target detection region has a known ratio of physical size to pixel size. Therefore, a transformation relationship from pixel to physical size can be established to obtain physical coordinates: In this way, the perspective transformation method can effectively correct the tilted and perspective-distorted field of view into a regular detection area, providing a standardized coordinate framework for subsequent object detection and tracking.

[0052] Combination Figure 5 As shown, in some embodiments, the aforementioned step S30, identifying the target gripping position of the fabric to be gripped in the image to be processed, includes steps S301 to S302.

[0053] Step S301: Identify the area of ​​the fabric to be grabbed in the image to be processed. Step S302: Determine the target gripping position of the fabric to be gripped in the area where the fabric to be gripped is located.

[0054] The image to be processed includes the fabric to be grasped and its background area. Identifying the area containing the fabric to be grasped within this image accurately separates it from the background, clearly defining its position and shape. This provides a clear analytical object for subsequent target grasping location determination. After locating the area containing the fabric, the target grasping position within that area is then determined, improving the accuracy of grasping and positioning. This ensures the accuracy and stability of target grasping location identification, providing a reliable positional basis for the grasping device to perform the grasping action, further guaranteeing the accuracy of fabric grasping and better meeting the precision requirements of automated production.

[0055] In some embodiments, identifying the region of the fabric to be grasped in the image to be processed includes: performing inter-frame difference operation on the image to be processed to obtain a difference image representing pixel differences; identifying a foreground region and a background region based on the difference image; and determining the region of the fabric to be grasped based on the foreground region.

[0056] Performing inter-frame differencing on the image to be processed accurately captures pixel-level differences, forming a difference image representing the pixel differences between different locations. Based on these differences, foreground and background regions can be distinguished, thus initially separating the foreground region that may contain the fabric to be grasped. Further filtering within this foreground region determines the area where the fabric to be grasped is located. In this way, through multi-layered filtering, precise positioning of the area containing the fabric to be grasped can be achieved, providing a reliable analytical range for subsequent target grasping location determination, thereby improving the overall accuracy of fabric grasping and meeting the precision requirements of automated production.

[0057] Here, the specific steps of the inter-frame difference operation are as follows: After the program starts running, set frame 30 as the background frame. , will the current frame and Perform absolute difference operations.

[0058] For each pixel in the image and each color channel : .

[0059] The combined value of the three channel differences is then calculated and converted into a grayscale image. , For a single-channel image, the larger the value, the greater the difference between the pixel and the background.

[0060] In some embodiments, performing inter-frame difference operations on the image to be processed to obtain a difference image representing pixel differences includes: performing inter-frame difference operations on the image to be processed to obtain a difference grayscale image, and performing noise reduction and edge enhancement operations on the difference grayscale image.

[0061] Specifically, noise can be suppressed and edges enhanced for differential grayscale images. Apply Gaussian blur to suppress high-frequency noise: in This represents the convolution operation. It is the standard deviation of the Gaussian kernel. This step can make the subsequent binarization results smoother and reduce isolated noise.

[0062] To further enhance the edges of the target, a sharpening convolution kernel is used to convolve the Gaussian-filtered image: This makes the boundary of the moving target, i.e. the fabric to be grasped, clearer, which helps to form a more complete outline in subsequent steps.

[0063] In some embodiments, identifying foreground and background regions based on the difference image includes converting the difference image into a binary mask to identify the foreground and background regions.

[0064] Converting the difference image into a binary mask clearly distinguishes the foreground and background regions in a distinct black-and-white binary format, enhancing the contrast between the fabric to be grasped and the background, making the outline of the fabric easier to identify. It also simplifies image information. This avoids the recognition ambiguity caused by grayscale gradations in the original difference image, thus improving the accuracy of foreground region recognition. It provides a high-contrast, low-interference image foundation for subsequent precise positioning of the fabric to be grasped, improving the accuracy of fabric location identification, and consequently, enhancing the accuracy of target grasping position identification, ultimately ensuring the grasping accuracy of the device.

[0065] Specifically, in some embodiments, converting the difference image into a binary mask to identify foreground and background regions includes: converting the difference image into an initial binary mask; preprocessing the initial binary mask to obtain a preprocessed binary mask; and identifying the foreground and background regions based on the preprocessed binary mask. The preprocessing includes erosion and dilation operations, with the dilation operation performed after the erosion operation.

[0066] After converting the difference image into an initial binary mask, an erosion operation is used to effectively eliminate small noise points in the initial mask, preventing noise from interfering with foreground region recognition. A dilation operation is then performed to recover from the shrinkage of the fabric region boundaries caused by the erosion operation, fill any voids within the fabric region, and connect fabric edges that are broken due to occlusion or similar colors, resulting in a more complete and smooth fabric outline. This preprocessing workflow optimizes the quality of the binary mask, improving the accuracy of subsequent foreground and background region differentiation. It ensures that the subsequently recognized fabric region more closely matches the actual fabric shape, providing high-quality image data for accurately determining the target grasping position, thereby enhancing the grasping accuracy of the grasping device.

[0067] Specifically, the dilation operation includes primary dilation and secondary dilation. Primary dilation is used to counteract the shrinkage caused by the erosion operation, while secondary dilation is used to fill voids and connect broken areas, thereby achieving further image optimization.

[0068] In some embodiments, the preprocessed grayscale image is converted into a binary mask to clearly distinguish the foreground and background. A threshold is then used to sharpen the image. Filter: .

[0069] in, This is the difference threshold; pixels exceeding this threshold are considered foreground pixels, otherwise they are considered background pixels.

[0070] Furthermore, to make the contour more complete and smoother, it is necessary to remove small noise points from the binary mask and fill any holes that may exist inside the target. Specific operations include: (1) To First perform the erosion operation, then the dilation operation to remove small foreground noise points: in, For a size of The core, corrosion operation It will indiscriminately erode all foreground areas. Those isolated areas, smaller than the structural element, will also be affected. Noise points will be completely eliminated. At the same time, larger foreground objects will shrink at their boundaries, so a dilation operation is performed on the eroded image. This restores it to roughly its original size with smoother boundaries. The expansion operation here is a single expansion.

[0071] (2) Connect adjacent areas that are broken due to occlusion or similar color and fill the voids inside the target, directly to The result is then subjected to a second expansion: In this way, the white foreground area expands outward, and the expanded foreground area will submerge the small background area inside. Simultaneously, if two areas that should belong to the same object are very close together, the secondary operation merges them into a connected region. Here, "two areas that should belong to the same object are very close together" specifically refers to these two parts being within the structural element... Within the scope of influence.

[0072] In some embodiments, determining the area where the fabric to be grasped is located based on the foreground area includes: filtering out independent areas in the foreground area whose area is greater than an area threshold as the area where the fabric to be grasped is located.

[0073] In this way, area filtering is used to identify independent regions from the binary mask, filtering out areas that are too small or may be noise, and retaining only contours with an area greater than a set threshold, which are then identified as valid moving targets. This effectively removes residual small particle noise, ensuring that only valid regions that conform to the fabric size characteristics are retained. This further narrows the analysis scope, avoids interference from interfering regions in determining the gripping position, improves the accuracy of identifying the area of ​​the fabric to be gripped, and facilitates the accuracy of subsequent target gripping position calculations, thereby ensuring the accuracy of fabric gripping and meeting the precision requirements of automated production.

[0074] Figure 6 This is a schematic diagram of an image to be processed provided in an embodiment of this application, which includes fabric 20 to be processed. Figure 7 To Figure 6 The image obtained after performing inter-frame difference operations and converting to a binary mask includes the area 201 where the fabric to be captured is located.

[0075] In some embodiments, determining the target gripping position of the fabric to be gripped within the area containing the fabric to be gripped includes: determining the centroid of the fabric to be gripped based on the outer contour of the area containing the fabric to be gripped, and using the centroid as the target gripping position. For example Figure 7 As shown, the location of the centroid 200 is determined based on the outer contour of the area 201 where the fabric to be grabbed is located, and is used as the target grab position.

[0076] This provides a clearer benchmark for locating the target gripping position. Using the centroid of the outer contour of the area containing the fabric to be gripped as the target gripping position allows the gripping operation to lock onto the core area of ​​the fabric. This avoids gripping point offsets caused by irregular edges, local wrinkles, or obstructions of the fabric, and balances the forces on different parts of the fabric, reducing the risk of slippage or posture deviation during gripping. This ensures greater stability of the gripped fabric during the gripping operation. This guarantees the accuracy and stability of the fabric gripping process, ensures the neatness of subsequent stacking operations, and better adapts to the requirements of automated production for gripping precision and stability.

[0077] In some embodiments, determining the centroid of the fabric to be grasped based on the outer contour of the area where the fabric to be grasped is located specifically includes: from the expanded mask image Extract the contours of all connected regions. Find the set of boundary points for all white regions in the image. Each of them It is a set of contour points. For each extracted contour... The centroid is calculated using image moments. Image moments are defined as: in, It is a point Pixel value at that location, and It is the order of the moment. The centroid coordinates are: in It is the area of ​​the outline. and It is a first-order moment.

[0078] Combination Figure 8 As shown, this application embodiment provides a control method for a gripping device, including steps S10 to S41.

[0079] Step S10: Obtain the original image of the fabric carrier device used to carry the fabric to be grasped.

[0080] Step S20: Through perspective transformation, the target monitoring area in the original image is corrected into a rectangular shape to obtain the image to be processed.

[0081] Step S30: Identify the target grasping position of the fabric to be grasped in the image to be processed. The fabric to be grasped is located in the target monitoring area.

[0082] Step S31: Identify the tilt angle of the fabric to be grasped in the image to be processed.

[0083] Step S401: Based on the tilt angle, control the gripping device to grip the target gripping position of the fabric to be gripped.

[0084] In this way, in addition to identifying the target gripping position in the fabric to be gripped, the tilt angle of the fabric in the image to be processed is further identified to accurately obtain the actual posture information of the fabric. Based on the target gripping position and the tilt angle, the gripping device adjusts its end effector posture before performing the gripping action, making the gripping posture of the gripping device compatible with the tilt state of the fabric. This effectively avoids problems such as unstable gripping, slippage, or uneven stacking caused by fabric tilt, further improving the adaptability and stability of the gripping action, ensuring that the gripping device accurately grips the target gripping position, thereby improving the overall accuracy of fabric gripping and better meeting the precision and stability requirements of automated production.

[0085] like Figure 7As shown, in some embodiments, identifying the tilt angle of the fabric to be grasped in the image to be processed includes: identifying the region where the fabric to be grasped is located in the image to be processed; determining the tilt angle of the smallest bounding rectangle of the region where the fabric to be grasped is located, as the tilt angle of the fabric to be grasped. First, the region where the fabric to be grasped is located in the image to be processed is identified, defining a precise range for the detection of the tilt angle and avoiding interference from irrelevant regions with the measurement results; then, by fitting the smallest bounding rectangle of the region, the tilt angle of the rectangle is taken as the tilt angle of the fabric. This accurately determines the tilt angle of the fabric, whose outer contour may be irregular, in a clear and quantifiable manner, ensuring the accuracy of the tilt angle, thereby improving the accuracy of the grasping operation performed accordingly.

[0086] Based on the contour C extracted in the aforementioned embodiments, the process of identifying the tilt angle will be further explained in detail here.

[0087] Specifically, for each contour Fit a minimum bounding rectangle. This rectangle is the minimum bounding rectangle, meaning it completely encloses the outline and has the smallest area; its orientation can be rotated arbitrarily. Let the center of the rectangle be... Width is The height is The rotation angle is The angle of the smallest bounding rectangle. It reflects the main direction of the object's tilt. It defines the tilt angle of the fabric. Let the angle between the long side of the rectangle and the horizontal axis of the image be the counterclockwise angle in degrees. The calculated tilt angle of the fabric to be grasped is: .

[0088] The target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction, causing the fabric to be grasped to move within the target monitoring area.

[0089] Combination Figure 9 As shown, this application embodiment provides a control method for a gripping device, including steps S10 to S401.

[0090] Step S10: Obtain the original image containing the target monitoring area.

[0091] Step S20: Through perspective transformation, the target monitoring area in the original image is corrected into a rectangular shape to obtain the image to be processed.

[0092] Step S30: Identify the target grasping position of the fabric to be grasped in the image to be processed. The fabric to be grasped is located in the target monitoring area.

[0093] Step S32: Determine the target time for the fabric to be grasped to reach the target grasping position based on the image to be processed.

[0094] Step S402: Control the gripping device to grip the target gripping position of the fabric to be gripped at the target time.

[0095] In an assembly line workflow, the fabric to be grasped is in motion. Here, based on locking the target grasping position, the timing of the grasping operation, i.e., the target time, is further determined. By combining the target time and the target grasping position to control the grasping device to perform the grasping operation, the accuracy of the fabric grasping process can be ensured.

[0096] In some embodiments, the coordinates of the image to be processed are based on a second image coordinate system, and a preset direction is along the horizontal axis of the second image coordinate system. The gripping device performs gripping within a set gripping area. Based on the image to be processed, determining the target time for the fabric to be gripped to reach the target gripping position includes: obtaining the lateral distance between the target gripping position and the gripping area along the horizontal axis of the second image coordinate system; obtaining the movement speed of the fabric to be gripped along the horizontal axis of the second image coordinate system; and determining the target time for the fabric to be gripped to reach the target gripping position based on the lateral distance and the movement speed.

[0097] The fabric to be grasped moves along the horizontal axis of the second image coordinate system, following the fabric carrying device. Based on the lateral distance between the target grasping position and the graspable area along the horizontal axis, the distance the fabric needs to travel can be determined. Combining this lateral distance with the speed of the fabric, the target time for the fabric to reach the target grasping position can be accurately determined. This ensures that the grasping device plans its trajectory in advance and executes the grasping action on time, effectively solving the technical problem of difficulty in judging the timing of dynamic grasping caused by continuous movement on the production line. This further improves the accuracy and reliability of fabric grasping operations, better meeting the precision requirements of automated production.

[0098] In some embodiments, the movement speed of the assembly line is a set value, assuming that the fabric to be grasped moves synchronously with the assembly line; therefore, the movement speed of the fabric to be grasped is this set value. Thus, the movement speed of the fabric to be grasped in the first image coordinate system is known. Based on the proportional relationship between the first and second image coordinate systems, the movement speed of the fabric to be grasped in the second image coordinate system can be determined.

[0099] In some embodiments, the moving speed of the fabric to be grasped in the second image coordinate system can be determined using the moving distance and time difference of the fabric in the second image coordinate system. Specific steps include, for example: (1) such as Figure 10 As shown, two virtual vertical lines are defined within the target detection region in the second image coordinate system as a reference for velocity measurement. For example... Figure 11The two virtual vertical lines L1 and L2 are located at the width of the image, respectively. Location and image width Place.

[0100] These two vertical lines are represented as ; .

[0101] in It is the image width.

[0102] These two lines divide the target detection area in the second image coordinate system into three parts at fixed distances, which are used to accurately measure the time required for an object to travel a fixed distance.

[0103] (2) For each tracked fabric object, continuously monitor its centroid coordinates and record the corresponding timestamp. When the centroid Record the centroid coordinates as the coordinates successively cross the two baselines. and and time and Since the assembly line moves along the horizontal axis, the displacement of the fabric to be gripped between the two baselines is: (3) The actual physical size of the target detection area is known to be The image resolution is Therefore, the actual horizontal displacement is Calculate the average pixel velocity using displacement and time difference. and actual speed : .

[0104] (4) Calculate the time it takes for the fabric to reach the graspable area. .

[0105] in, The x-axis coordinates represent the area that can be grabbed.

[0106] In some embodiments, the target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction, causing the fabric to be grasped to move within the target monitoring area; and the grasping device performs grasping within a set graspable area. Controlling the grasping device to grasp the target grasping position of the fabric to be grasped includes: determining a grasping operation point when the target grasping position moves to the graspable area based on the target grasping position of the fabric to be grasped; and controlling the grasping device to grasp the fabric to be grasped at the grasping operation point.

[0107] In an assembly line workflow, the fabric to be gripped is in motion. By locking the target gripping position and further determining the actual gripping point where the gripping operation is performed while the fabric is in motion, the gripping equipment can be controlled. This avoids positioning deviations caused by the fabric's movement, improves the matching degree between the gripping action and the fabric position, further ensures the accuracy of fabric gripping, and guarantees that the gripping equipment can stably and accurately complete the gripping action in dynamic scenarios, thus better meeting the precision requirements of automated production.

[0108] In some embodiments, the coordinates of the image to be processed are based on a second image coordinate system, and the control of the gripping device is based on a mechanical coordinate system. A preset direction is along the horizontal axis of the second image coordinate system, and the horizontal coordinate of the grippable area is fixed. The aforementioned determination of the gripping operation point when the target gripping position moves to the grippable area based on the target gripping position of the fabric to be gripped includes: determining the coordinates of the gripping operation point in the second image coordinate system based on the target gripping position of the fabric to be gripped; and converting the coordinates of the gripping operation point in the second image coordinate system to coordinates in the mechanical coordinate system through coordinate system transformation, using these coordinates as the gripping operation point. Wherein, the horizontal coordinate of the gripping operation point in the second image coordinate system is the horizontal coordinate of the grippable area in the second image coordinate system, and the vertical coordinate of the gripping operation point in the second image coordinate system is the vertical coordinate of the target gripping position in the second image coordinate system.

[0109] The image to be processed is based on a second image coordinate system, while the gripping device is controlled based on a mechanical coordinate system. The preset direction of the fabric to be gripped is along the horizontal axis of the second image coordinate system, and the vertical coordinate of the target gripping position within the fabric remains unchanged during the movement. By using the horizontal coordinate of the grippable area as the horizontal coordinate of the gripping operation point in the second image coordinate system, and the vertical coordinate of the target gripping position as the vertical coordinate of the gripping operation point, the coordinates of the fabric when it moves into the grippable area can be accurately located. After determining the coordinates of the gripping operation point in the second image coordinate system, a coordinate system transformation is performed to convert these coordinates to the coordinates in the mechanical coordinate system. The gripping device then performs the gripping operation accordingly, achieving precise coordinate mapping and effectively solving the problem of inconsistency between the second image coordinate system and the mechanical coordinate system. This ensures the accuracy of gripping positioning and guarantees that the gripping device accurately executes gripping actions in dynamic scenes, better meeting the precision requirements of automated production.

[0110] Specifically, the x-coordinate of the grabbable region in the second image coordinate system is... Since the fabric to be grabbed moves along the horizontal axis, the coordinates of the fabric reaching the grabbable area can be calculated as follows: The coordinates The coordinates in the second image coordinate system need to be further converted to coordinates in the machine coordinate system. The specific steps of the conversion process are as follows: (1) Define the origin of the second image coordinate system as the top left corner of the image, with units of pixels, and the point as... The origin of the mechanical coordinate system is the center point of the base, and the unit is millimeters. Points are represented as... .

[0111] (2) Measure and obtain four sets of corresponding point pairs: in, These are the x and y coordinates of the corresponding points in the machine coordinate system obtained from actual measurements.

[0112] (3) Use a two-dimensional affine transformation model to describe the mapping relationship between the two coordinate systems: in These are the affine transformation parameters to be determined. Substituting the four corresponding points into the system of equations, we construct an overdetermined system of equations: Abbreviated as: (4) Solve for the optimal transformation parameters using the least squares method: The affine transformation matrix is ​​obtained as follows: For grab points in any image coordinate system Applying affine transformation: (5) Since the horizontal relationship between the production line and the gripping equipment remains unchanged, For fixed values, The result obtained in step three The deviation is fixed as The complete grasping pose of the grasping device can be obtained as follows: .

[0113] Combination Figure 11 As shown in the figure, this application embodiment also provides a control method for a gripping device, including steps S11 to S19.

[0114] Step S11: Read the data from the image acquisition device and use perspective transformation to convert the field of view into a rectangular detection area.

[0115] The original image of the fabric carrier used to hold the fabric to be grasped is obtained by the image acquisition device. The target monitoring area in the original image is corrected into a rectangular shape by perspective transformation to obtain the image to be processed.

[0116] Step S12: Use a segmentation detection algorithm to extract the outline of the fabric to be captured.

[0117] The segmentation detection algorithm includes the aforementioned inter-frame difference operation and the conversion of the difference image into a binary mask. In some embodiments, it also includes noise reduction and edge enhancement operations after inter-frame difference thresholding, as well as preprocessing procedures involved in converting the difference image into a binary mask.

[0118] Step S13: Calculate the centroid and tilt angle of the fabric to be grasped based on the outline of the fabric to be grasped.

[0119] The centroid of the fabric to be grasped, calculated based on its outline, is the target grasping position.

[0120] Step S14: Calculate the assembly line speed based on the position change of the centroid of the fabric to be grasped in the rectangular detection area.

[0121] That is, the aforementioned method uses the moving distance and time difference of the fabric to be grasped in the second image coordinate system to determine the moving speed of the fabric to be grasped in the second image coordinate system.

[0122] Step S15: Predict the time it takes for the fabric to reach the gripping position based on the gripping position coordinate x value.

[0123] The grab position here corresponds to the grabbable area mentioned above.

[0124] Step S16: Calculate the gripping coordinates of the fabric based on the fabric centroid and the gripping position coordinate x value.

[0125] That is, based on the target gripping position of the fabric to be gripped, the coordinates of the gripping operation point in the second image coordinate system are determined. The x-coordinate of the gripping operation point in the second image coordinate system is the x-coordinate of the grippable area in the second image coordinate system, and the y-coordinate of the gripping operation point in the second image coordinate system is the y-coordinate of the target gripping position in the second image coordinate system.

[0126] Step S17: Convert the fabric gripping coordinates into robotic arm coordinates.

[0127] That is, by transforming the coordinate system, the coordinates of the gripping operation point in the second image coordinate system are converted into the coordinates in the machine coordinate system, thereby controlling the gripping device to perform the gripping action.

[0128] Step S18: The gripping device arrives at the gripping waiting position in advance and adjusts its posture according to the tilt angle of the fabric.

[0129] Step S19: The gripping device contacts the fabric to be gripped, performs the gripping action, and places the fabric to be gripped at the target position.

[0130] Specifically, the robotic arm contacts the fabric and controls the gripper to grasp it, and then places the fabric to be grasped at the target location.

[0131] This application provides a control system for a gripping device, including one or more processors for implementing the aforementioned control method for the gripping device. In some embodiments, the control system is electrically connected to the gripping device. In some embodiments, the gripping device includes a gripping device body and a control system, the control system being disposed on the gripping device body.

[0132] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are quite specific and detailed. However, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of this application, and these modifications and improvements all fall within the protection scope of this application.

[0133] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A control method for a gripping device, characterized in that, include: Obtain the original image of the fabric carrier used to hold the fabric to be grasped; By performing perspective transformation, the target monitoring area in the original image is corrected into a rectangular shape to obtain the image to be processed; Identify the target grasping position of the fabric to be grasped in the image to be processed, wherein the fabric to be grasped is located in the target monitoring area; Control the gripping device to grip the target gripping position of the fabric to be gripped.

2. The control method according to claim 1, characterized in that, The target monitoring area has multiple marked corner points, which can be connected in the fabric carrying device to form a rectangle, and the multiple marked corner points form an irregular quadrilateral in the original image; The step of correcting the target monitoring area in the original image into a rectangular shape through perspective transformation to obtain the image to be processed includes: Determine the coordinates of the plurality of marked corner points in the original image in the first image coordinate system; In the second image coordinate system, define multiple corner points that correspond one-to-one with each marked corner point, and determine the coordinates of the multiple corner points in the second image coordinate system; the multiple corner points in the second image coordinate system can be connected to form a rectangle; Based on the coordinates of each marked corner point in the first image coordinate system and the coordinates of each corner point corresponding to each marked corner point in the second image coordinate system, the mapping relationship between the first image coordinate system and the second image coordinate system is determined. For multiple pixels in the original image, they are mapped to the second image coordinate system according to the mapping relationship to obtain the image to be processed; The multiple pixels in the original image include the multiple pixels in the target monitoring area.

3. The control method according to claim 1, characterized in that, The process of identifying the target grasping position of the fabric to be grasped in the image to be processed includes: Identify the area of ​​the fabric to be grasped in the image to be processed; In the area where the fabric to be grabbed is located, the target grab position of the fabric to be grabbed is determined.

4. The control method according to claim 3, characterized in that, Determining the target gripping position of the fabric to be gripped within the area where the fabric to be gripped is located includes: Based on the outer contour of the area where the fabric to be grasped is located, the centroid of the fabric to be grasped is determined, and the centroid is used as the target grasping position.

5. The control method according to claim 1, characterized in that, After correcting the target monitoring area in the original image to a rectangular shape through perspective transformation to obtain the image to be processed, the control method further includes: Identify the tilt angle of the fabric to be grasped in the image to be processed; The control of the gripping device to grip the target gripping position of the fabric to be gripped includes: The target gripping position for the gripping device to grip the fabric to be gripped is controlled according to the tilt angle.

6. The control method according to claim 5, characterized in that, The process of identifying the tilt angle of the fabric to be grasped in the image to be processed includes: Identify the area of ​​the fabric to be grasped in the image to be processed; The tilt angle of the smallest bounding rectangle of the area where the fabric to be grasped is located is determined, and this is taken as the tilt angle of the fabric to be grasped.

7. The control method according to any one of claims 3 to 6, characterized in that, The process of identifying the area of ​​the fabric to be grasped in the image to be processed includes: Perform inter-frame difference operation on the image to be processed to obtain a difference image representing pixel differences; The foreground and background regions are identified based on the differential image; Based on the foreground area, determine the area where the fabric to be grabbed is located.

8. The control method according to claim 7, characterized in that, The step of identifying the foreground and background regions based on the difference image includes: The difference image is converted into a binary mask to identify the foreground and background regions.

9. The control method according to claim 8, characterized in that, The step of converting the difference image into a binary mask to identify foreground and background regions includes: The difference image is converted into an initial binary mask; The initial binary mask is preprocessed to obtain a preprocessed binary mask; The foreground and background regions are identified based on the pre-processed binary mask. The pretreatment includes an etching operation and an expansion operation, and the expansion operation is performed after the etching operation is completed.

10. The control method according to claim 7, characterized in that, The step of determining the area where the fabric to be grasped is located based on the foreground area includes: Select independent regions in the foreground area whose area is greater than the area threshold as the region where the fabric to be grabbed is located.

11. The control method according to claim 1, characterized in that, The target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction, causing the fabric to be grasped to move within the target monitoring area; After correcting the target monitoring area in the original image to a rectangular shape through perspective transformation to obtain the image to be processed, the control method further includes: Based on the image to be processed, determine the target time for the fabric to be grasped to reach the target grasping position; The control of the gripping device to grip the target gripping position of the fabric to be gripped includes: The gripping device is controlled to grip the target gripping position of the fabric to be gripped at the target time.

12. The control method according to claim 11, characterized in that, The coordinates of the image to be processed are based on a second image coordinate system, and the preset direction is along the horizontal axis of the second image coordinate system; The grasping device performs grasping within the designated graspable area; The step of determining the target time for the fabric to be grasped to reach the target grasping position based on the image to be processed includes: Obtain the lateral distance between the target grabbing position and the grabbable area along the horizontal axis of the second image coordinate system; The speed at which the fabric to be grasped moves along the horizontal axis of the second image coordinate system is obtained; Based on the lateral distance and movement speed, the target time for the fabric to be grasped to reach the target grasping position is determined.

13. The control method according to claim 1, characterized in that, The target monitoring area includes a fabric carrying device, at least a portion of which slides along a preset direction, causing the fabric to be grasped to move within the target monitoring area; and the grasping device performs grasping within the set graspable area; The control of the gripping device to grip the target gripping position of the fabric to be gripped includes: Based on the target gripping position of the fabric to be gripped, determine the gripping operation point when the target gripping position moves to the gripping area; The gripping device is controlled to grip the fabric to be gripped at the gripping operation point.

14. The control method according to claim 13, characterized in that, The coordinates of the image to be processed are based on the second image coordinate system, and the control of the grasping device is based on the mechanical coordinate system; The preset direction is along the horizontal axis of the second image coordinate system, and the horizontal coordinate of the graspable area is fixed. The step of determining the gripping operation point when the target gripping position moves to the grippable area based on the target gripping position of the fabric to be gripped includes: Based on the target gripping position of the fabric to be gripped, determine the coordinates of the gripping operation point in the second image coordinate system; By transforming the coordinate system, the coordinates of the grasping operation point in the second image coordinate system are converted into coordinates in the machine coordinate system; Wherein, the abscissa of the grabbing operation point in the second image coordinate system is the abscissa of the grabbable area in the second image coordinate system, and the ordinate of the grabbing operation point in the second image coordinate system is the ordinate of the target grabbing position in the second image coordinate system.

15. A control system for a gripping device, characterized in that, It includes one or more processors for implementing the control method of the gripping device as described in any one of claims 1-14.

16. A gripping device, characterized in that, include: Grab the device body, and The control system of the gripping device as described in claim 15 is disposed on the gripping device body.

Images