Cloth grabbing method, device, equipment, robot, medium and product

Abstract

The disclosure provides a cloth grabbing method, device, equipment, robot, medium and product. The method comprises the following steps: in response to obtaining a contour point set of target cloth on a conveying device, performing polygon approximation processing on the contour point set of the target cloth according to a contour geometric shape corresponding to the contour point set to obtain an approximation polygon, taking an area surrounded by the approximation polygon as a main area of the target cloth, and determining a position in the main area that meets a safety interval condition as a visual grabbing point of the target cloth; based on a mapping relationship between a preset pixel coordinate system and a robot coordinate system, converting the visual grabbing point from the pixel coordinate system to the robot coordinate system to obtain an execution grabbing point; and controlling a mechanical arm of the robot to grab the target cloth according to the execution grabbing point. The adaptive grabbing point planning of the cloth is realized, and the grabbing success rate of the irregularly shaped flexible cloth is improved.

Description

Technical Field

[0001] This disclosure relates to the field of robot detection and control technology, and in particular to a method, apparatus, equipment, robot, medium and product for grasping fabric. Background Technology

[0002] Automated gripping of flexible fabrics is a crucial step in textile manufacturing, garment processing, smart warehousing, and flexible material production. Unlike rigid workpieces, flexible fabrics are characterized by their softness, deformability, wrinkling, and non-fixed boundary shapes. During transport, they are prone to curling, folding, piling, and random posture changes, making it difficult for robot positioning and gripping methods to achieve stable automated operation. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides the following technical solutions: Firstly, a method for grasping fabric is provided, the method comprising: In response to acquiring the contour point set of the target fabric on the conveying device, the contour point set of the target fabric is subjected to polygon approximation processing according to the contour geometry corresponding to the contour point set to obtain an approximation polygon, and the area enclosed by the approximation polygon is taken as the main area of ​​the target fabric. The position in the main area that meets the safety interval condition is determined as the visual grasping point of the target fabric. Based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, the visual grasping point is transformed from the pixel coordinate system to the robot coordinate system to obtain the execution grasping point; The robotic arm of the robot is controlled to grasp the target fabric according to the grasping point.

[0004] In this embodiment, after obtaining the contour point set of the target fabric, the approximate polygon of the target fabric contour can be obtained, which improves the accuracy of obtaining the main area of ​​the target fabric, thereby realizing adaptive gripping point planning for irregularly shaped fabrics, ensuring that the gripping points are located in the safe stress area, and improving the gripping success rate of irregularly shaped flexible fabrics.

[0005] In one embodiment, obtaining the contour point set of the target fabric on the conveying device includes: In response to acquiring a captured image of the target fabric on the conveying device, a multi-stage visual inspection is performed on the captured image to obtain a segmentation mask of the target fabric; The contour information of the target fabric is obtained according to the segmentation mask. The contour information of the target fabric includes the contour area, centroid position, and contour point set.

[0006] In this embodiment, by performing multi-stage visual inspection on the acquired images, the success rate and stability of fabric contour detection under complex working conditions can be improved.

[0007] In one embodiment, multi-stage visual detection is performed on the acquired image to obtain a segmentation mask for the target fabric, including: Background difference detection is performed on the acquired image to obtain the background region composed of pixels that meet the first preset condition; Based on the outline of the background region, connected regions with an area greater than or equal to a preset area threshold are determined as moving foreground regions; the moving foreground regions serve as foreground cue information for a preset semantic segmentation model; the foreground cue information includes at least one of the following: a target bounding box cue generated based on the bounding rectangle of the connected regions of the foreground region, and a point cue generated based on the centroid coordinates of the connected regions of the foreground region. The moving foreground region and the acquired image are input into the semantic segmentation model for segmentation processing to obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0008] In this embodiment, background subtraction can quickly filter candidate foreground regions, reduce the processing range of subsequent segmentation models, and improve overall detection efficiency; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0009] In one embodiment, the method further includes: In response to the background difference detection failing to obtain the moving foreground region, a difference image is acquired between the acquired image and a preset device template; the device template refers to an image taken when there is no fabric on the conveying device; the difference image is filtered and then segmented using an adaptive threshold to obtain candidate fabric regions; The candidate fabric region is used as the foreground cue information of the semantic segmentation model and input into the preset semantic segmentation model for segmentation processing to obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0010] In this embodiment, template difference is used as a supplementary detection method to still identify the fabric area when background difference fails, thereby improving detection robustness; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive segmentation from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0011] In one embodiment, the method further includes: In response to the absence of a candidate fabric region, a bounding box is generated based on the long side length, short side length, and centroid position of the target fabric in the detection results of the previous frame image; the bounding box serves as foreground cue information for a preset semantic segmentation model. The bounding box and the acquired image are input into a preset semantic segmentation model for segmentation processing to obtain a segmentation mask of the target fabric output by the semantic segmentation model.

[0012] In this embodiment, the historical frame information rollback mechanism maintains the continuity of target cloth tracking when the detection of the current frame image fails, thus avoiding the loss of target cloth; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0013] In one embodiment, after obtaining the segmentation mask of the target fabric, the method further includes: The target fabric corresponding to the currently acquired image is associated and matched with the target fabric corresponding to the historical acquired images to obtain the continuous motion trajectory of the target fabric.

[0014] In this embodiment, by tracking the target fabric in association with multiple frames of acquired images, independent tracking and motion prediction of each target fabric can be achieved.

[0015] In one embodiment, the contour point set of the target fabric is approximated by a polygon based on the contour geometry corresponding to the contour point set to obtain an approximate polygon, and the area enclosed by the approximate polygon is taken as the main area of ​​the target fabric, including: Obtain the smallest convex polygon that encloses all contour points within the set of contour points; The minimum bounding polygon or the maximum inscribed polygon is iteratively searched using multiple approximation accuracies to obtain the approximation polygons corresponding to each approximation accuracies. The region containing the approximate polygon that has the largest coverage of the target fabric is determined as the main region of the target fabric.

[0016] In this embodiment, the main area that fits the actual shape of the fabric is obtained by iterative approximation with multiple approximation accuracies, thereby improving the accuracy of subsequent gripping point planning.

[0017] In one embodiment, the method further includes: In response to the absence of a minimum bounding polygon or a maximum inscribed polygon, point filling is performed based on the geometry of the contour to generate the boundary key points of the contour. The boundary key points of the contour are determined as the visual grasping points.

[0018] In this embodiment, point supplementation processing ensures that effective gripping points can be generated for all types of fabric, thereby improving the robustness of the system.

[0019] In one embodiment, determining the location within the main body area that meets the safety interval condition as the visual grasping point of the target fabric includes: Obtain the minimum bounding rectangle of the main region, wherein the minimum bounding rectangle includes the long side and the short side; Obtain the midpoints of the two short sides of the minimum bounding rectangle as the initial gripping points of the target fabric; Perform distance transformation processing on the main body region to obtain the shortest distance from each pixel position within the main body region to the contour boundary; Within a preset neighborhood of the initial grasping point, the location with the largest shortest distance is searched as the visual grasping point.

[0020] In this embodiment, by using distance transformation calculation, the internal point farthest from the boundary can be searched within the neighborhood of the initial gripping point, ensuring that the gripping point is located in a safe and stable area of ​​the target fabric, thereby improving the success rate of gripping irregularly shaped flexible fabrics.

[0021] In one embodiment, the ratio of the radius of the preset neighborhood range to the length of the shorter side is [10%, 30%].

[0022] In this embodiment, by setting a reasonable search radius, search efficiency and optimization effect can be balanced.

[0023] In one embodiment, the method further includes: In response to obtaining multiple execution crawl points, calculate the crawl point distance between adjacent execution crawl points; In response to the fact that the distance between the execution grab points is less than a preset safety distance threshold, adjacent execution grab points are merged, and the merged grab points are used to replace the execution grab points.

[0024] In this embodiment, by optimizing the grab point merging process, interference from the end effector can be avoided, ensuring the safety of multi-grab point collaborative operation.

[0025] In one embodiment, the ratio of the preset safety distance threshold to the size of the end effector is [1.2, 1.5].

[0026] In one embodiment, the robot's robotic arm includes a first robotic arm and a second robotic arm; controlling the robot's robotic arm to grasp the target fabric according to the execution gripping point includes: Obtain the rotation angle of the minimum bounding rectangle corresponding to the target fabric, and determine the attitude angle of the target fabric; The orientation and pose of the end effectors on the first and second robotic arms are adjusted according to the attitude angle of the target fabric so that the gripping direction of the end effectors is aligned with the short side direction of the target fabric; the attitude angle of the target fabric is determined based on the rotation angle of the minimum circumscribed rectangle. Based on the pose instructions in the grasping task queue, spatial interference detection is performed based on the current poses and planned trajectories of the first and second robotic arms; In response to spatial interference detection, the first robotic arm and the second robotic arm are controlled to change from a safe pose to a grasping pose to move to the execution grasping point of the target fabric. In response to grasping the target fabric, the first robotic arm and the second robotic arm are controlled to change from the grasping posture to the placement posture, and finally return to the safe posture transformation.

[0027] In this embodiment, by equipping the robot with two robotic arms, the collaborative scheduling and collision prevention of the two robotic arms can be utilized to improve grasping efficiency while ensuring operational safety.

[0028] In one embodiment, the end effector of the robotic arm includes a lead screw and multiple gripping mechanisms, and the method further includes: Based on the length of the long side of the target fabric and the zero-center distance of the lead screw, the positioning distance of the lead screw is determined, and the multiple gripping mechanisms are evenly distributed on both sides of the long side of the target fabric.

[0029] In this embodiment, the gripping direction is aligned with the fabric texture direction by adaptive adjustment of the attitude angle, thereby improving gripping stability; the gripping operation of different sized fabrics is automatically adapted by adaptive positioning of the lead screw, without the need for frequent manual adjustment of the end effector spacing, thus improving the flexibility of the production line and gripping stability.

[0030] Secondly, a fabric gripping device is provided, the device comprising: The visual grasping point determination module is used to respond to the acquisition of the contour point set of the target fabric on the conveying device, perform polygon approximation processing on the contour point set of the target fabric according to the contour geometry corresponding to the contour point set to obtain the approximate polygon, and take the area enclosed by the approximate polygon as the main area of ​​the target fabric. The visual grasping point determination module is used to determine the positions within the main body area that meet the safety interval conditions as the visual grasping points of the target fabric; The execution gripping point determination module is used to transform the visual gripping point from the pixel coordinate system to the robot coordinate system based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, so as to obtain the execution gripping point; The target fabric gripping module is used to control the robot's robotic arm to grip the target fabric according to the gripping point.

[0031] In this embodiment, after obtaining the contour point set of the target fabric, the approximate polygon of the target fabric contour can be obtained, which improves the accuracy of obtaining the main area of ​​the target fabric, thereby realizing adaptive gripping point planning for irregularly shaped fabrics, ensuring that the gripping points are located in the safe stress area, and improving the gripping success rate of irregularly shaped flexible fabrics.

[0032] In one embodiment, the visual grasping point determination module includes: The segmentation mask acquisition submodule is used to perform multi-stage visual inspection on the acquired image of the target fabric on the conveying device in response to acquiring the image of the target fabric to obtain the segmentation mask of the target fabric. The fabric contour acquisition submodule is used to acquire the contour information of the target fabric according to the segmentation mask. The contour information of the target fabric includes the contour area, centroid position, and contour point set.

[0033] In this embodiment, by performing multi-stage visual inspection on the acquired images, the success rate and stability of fabric contour detection under complex working conditions can be improved.

[0034] In one embodiment, the segmentation mask acquisition submodule includes: A moving foreground region determination unit is used to perform background difference detection on the acquired image to obtain pixels that meet a first preset condition to form a moving foreground region; the moving foreground region serves as foreground prompt information of a preset semantic segmentation model; the foreground prompt information includes at least one of the following: a target box prompt generated based on the bounding rectangle of the connected domain of the foreground region, and a point prompt generated based on the centroid coordinates of the connected domain of the foreground region; The segmentation mask determination unit is used to input the moving foreground region and the acquired image into the semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0035] In this embodiment, background subtraction can quickly filter candidate foreground regions, reduce the processing range of subsequent segmentation models, and improve overall detection efficiency; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0036] In one embodiment, the segmentation mask acquisition submodule further includes: A differential image acquisition unit is used to acquire a differential image between the acquired image and a preset device template in response to the background differential detection not obtaining the moving foreground region; the device template refers to an image taken when there is no fabric on the conveying device; the differential image is filtered and then segmented using an adaptive threshold to obtain candidate fabric regions; The segmentation mask acquisition unit is used to input the candidate fabric region as foreground cue information of the semantic segmentation model into the preset semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0037] In this embodiment, template difference is used as a supplementary detection method to still identify the fabric area when background difference fails, thereby improving detection robustness; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive segmentation from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0038] In one embodiment, the segmentation mask acquisition submodule further includes: The bounding box generation unit is used to generate a bounding box based on the length of the long side, the length of the short side, and the centroid position of the target fabric in the detection result corresponding to the previous frame image in response to the absence of a candidate fabric region. The bounding box serves as foreground cue information for a preset semantic segmentation model. The segmentation mask acquisition unit is used to input the bounding box and the acquired image into a preset semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0039] In this embodiment, the historical frame information rollback mechanism maintains the continuity of target cloth tracking when the detection of the current frame image fails, thus avoiding the loss of target cloth; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0040] In one embodiment, after obtaining the segmentation mask of the target fabric, the visual grasping point determination module includes: The motion trajectory determination module is used to associate and match the target fabric corresponding to the currently acquired image with the target fabric corresponding to the historical acquired images to obtain the continuous motion trajectory of the target fabric.

[0041] In this embodiment, by tracking the target fabric in association with multiple frames of acquired images, independent tracking and motion prediction of each target fabric can be achieved.

[0042] In one embodiment, the visual grasping point determination module includes: The minimum convex polygon acquisition submodule is used to acquire the minimum convex polygon that encloses all contour points within the set of contour points. The approximation polygon determination submodule is used to iteratively search the minimum bounding polygon or the maximum inscribed polygon using multiple approximation accuracies to obtain the approximation polygons corresponding to each approximation accuracies. The main area determination module is used to determine the area containing the approximate polygon that has the largest coverage of the target fabric as the main area of ​​the target fabric.

[0043] In this embodiment, the main area that fits the actual shape of the fabric is obtained by iterative approximation with multiple approximation accuracies, thereby improving the accuracy of subsequent gripping point planning.

[0044] In one embodiment, the visual grasping point determination module includes: The boundary key point determination submodule is used to generate the boundary key points of the contour by performing point filling processing based on the geometry of the contour in response to the absence of a minimum bounding polygon or a maximum inscribed polygon. The visual grasping point determination submodule is used to determine the boundary key points of the contour as the visual grasping points.

[0045] In this embodiment, point supplementation processing ensures that effective gripping points can be generated for all types of fabric, thereby improving the robustness of the system.

[0046] In one embodiment, the visual grasping point determination module includes: The minimum bounding rectangle acquisition submodule is used to acquire the minimum bounding rectangle of the main area, wherein the minimum bounding rectangle includes the long side and the short side; The initial gripping point acquisition submodule is used to acquire the midpoints of the two short sides of the minimum bounding rectangle as the initial gripping points of the target fabric. The shortest distance determination submodule is used to perform distance transformation processing on the main body region to obtain the shortest distance from each pixel position in the main body region to the contour boundary. The visual grasping point determination submodule is used to search for the position with the largest shortest distance within a preset neighborhood of the initial grasping point as the visual grasping point.

[0047] In this embodiment, by using distance transformation calculation, the internal point farthest from the boundary can be searched within the neighborhood of the initial gripping point, ensuring that the gripping point is located in a safe and stable area of ​​the target fabric, thereby improving the success rate of gripping irregularly shaped flexible fabrics.

[0048] In one embodiment, the ratio of the radius of the preset neighborhood range to the length of the shorter side is [10%, 30%].

[0049] In this embodiment, by setting a reasonable search radius, search efficiency and optimization effect can be balanced.

[0050] In one embodiment, the device further includes: The crawl point distance acquisition module is used to calculate the crawl point distance between adjacent crawl points in response to the acquisition of multiple crawl points; The execution grab point determination module is used to merge adjacent execution grab points in response to the distance between the execution grab points being less than a preset safety distance threshold, and to replace the execution grab points with the merged grab points.

[0051] In this embodiment, by optimizing the grab point merging process, interference from the end effector can be avoided, ensuring the safety of multi-grab point collaborative operation.

[0052] In one embodiment, the ratio of the preset safety distance threshold to the size of the end effector is [1.2, 1.5].

[0053] In one embodiment, the robot's robotic arm includes a first robotic arm and a second robotic arm; the target fabric gripping module includes: The attitude angle acquisition submodule is used to acquire the rotation angle of the minimum bounding rectangle corresponding to the target fabric and determine the attitude angle of the target fabric. The orientation and pose adjustment submodule is used to adjust the orientation and pose of the end effectors on the first and second robotic arms according to the orientation angle of the target fabric, so that the gripping direction of the end effector is aligned with the short side direction of the target fabric; the orientation angle of the target fabric is determined based on the rotation angle of the minimum bounding rectangle; The spatial interference detection submodule is used to perform spatial interference detection based on the pose instructions in the grasping task queue and the current pose and planned trajectory of the first robotic arm and the second robotic arm. The grasping pose switching submodule is used to control the first robotic arm and the second robotic arm to change from a safe pose to a grasping pose in response to spatial interference detection so as to move to the execution grasping point of the target fabric. The pose transformation submodule is used to control the first robotic arm and the second robotic arm to transform from the grasping pose to the placement pose in response to the grasping of the target fabric, and finally return to the safe pose transformation.

[0054] In this embodiment, by equipping the robot with two robotic arms, the collaborative scheduling and collision prevention of the two robotic arms can be utilized to improve grasping efficiency while ensuring operational safety.

[0055] In one embodiment, the end effector of the robotic arm includes a lead screw and multiple gripping mechanisms, and the device further includes: The positioning distance determination module is used to determine the positioning distance of the lead screw based on the length of the long side of the target fabric and the zero-position center distance of the lead screw, and to evenly distribute the multiple gripping mechanisms on both sides of the long side of the target fabric.

[0056] In this embodiment, the gripping direction is aligned with the fabric texture direction by adaptive adjustment of the attitude angle, thereby improving gripping stability; the gripping operation of different sized fabrics is automatically adapted by adaptive positioning of the lead screw, without the need for frequent manual adjustment of the end effector spacing, thus improving the flexibility of the production line and gripping stability.

[0057] Thirdly, a robot is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the method as described in any of the first aspects when executing the computer program.

[0058] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the method as described in any of the first aspects.

[0059] Fifthly, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, performs the method as described in any of the first aspects.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0062] Figure 1 This is a flowchart of a fabric gripping method disclosed herein.

[0063] Figure 2 This is a schematic diagram of a robotic arm grasping fabric, as disclosed in this publication.

[0064] Figure 3 This is a flowchart of another fabric gripping method provided in this disclosure.

[0065] Figure 4 This is a block diagram of a fabric gripping device disclosed herein.

[0066] Figure 5 This is a schematic diagram of the structure of a computing device provided in this disclosure. Detailed Implementation

[0067] This disclosure describes multiple technical solutions with different concepts. Each concept has one or more embodiments, and different concepts can be combined to form more embodiments. Those skilled in the art, after reading this disclosure, can combine different concepts to obtain new technical solutions, and these new technical solutions should also fall within the scope of this disclosure.

[0068] The technical solutions of these different concepts will be introduced in turn below. Some concepts may appear in multiple technical solutions of different concepts. For these concepts, this disclosure will explain them when they first appear and will not repeat them in the following text.

[0069] In related technologies, the planning of gripping points for flexible fabrics usually uses methods such as minimum circumscribed rectangle, centroid of the outline, or fixed template matching to determine the gripping position. This method is well applicable to regular shaped targets, but for triangular, trapezoidal, L-shaped, and other irregularly shaped fabrics, it is not possible to accurately reflect the actual boundary characteristics of the fabric. This can easily cause the gripping point to deviate from the target force area, resulting in unstable gripping, fabric slippage, or clamping failure.

[0070] Therefore, this disclosure provides a fabric grasping method, apparatus, device, robot, medium, and product. The fabric grasping method can be adapted to a robot, which may include a humanoid robot, a serial multi-degree-of-freedom robotic arm, a planar articulated robot, or dual independent cooperative robotic arms, etc., and can be configured according to specific scenarios. For ease of description, the technical solution will be described as a robot in subsequent embodiments. The robot includes two robotic arms, namely a first robotic arm and a second robotic arm. The first and second robotic arms can perform grasping operations independently or collaboratively, depending on the specific scenario, and are not limited thereto. The robot also includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform a fabric grasping method.

[0071] In one embodiment, see Figure 1 A method for grasping fabric, comprising steps 11-14.

[0072] In step 11, in response to obtaining the contour point set of the target fabric on the conveying device, the contour point set of the target fabric is subjected to polygon approximation processing according to the contour geometry corresponding to the contour point set to obtain an approximation polygon, and the area enclosed by the approximation polygon is taken as the main area of ​​the target fabric.

[0073] In this step, the robot's processor can acquire the contour point set of the target fabric on the conveying device.

[0074] See Figure 2 The robot is equipped with a vision sensor 21, which may include a camera or a webcam. The field of view of the vision sensor covers the moving parts of the conveyor device 23, which may be a conveyor belt. After the fabric is placed on the conveyor belt, it can move synchronously with the conveyor belt. The vision sensor can acquire images at preset intervals. The robot's robotic arm 22 includes an end effector 221, which can grasp the fabric placed on the conveyor device 23 after reaching the grasping point.

[0075] The processor can communicate with the aforementioned visual sensor and acquire the captured images transmitted by the visual sensor, specifically the captured images of the target fabric on the conveyor. In response to acquiring the captured images of the target fabric on the conveyor, the processor can perform multi-stage visual detection on the captured images to obtain a segmentation mask for the target fabric. The multi-stage visual detection may include at least two of the following: background subtraction detection, template subtraction detection, and bounding box detection of the fabric from the previous frame. The detection result of the current stage in the multi-stage visual detection process serves as a prompt input for the semantic segmentation model in the next stage. The segmentation mask of the target fabric is used to determine the boundary of the region where the target fabric is located within the captured image, and it represents an accurate boundary at the pixel level.

[0076] In one example, multi-stage visual detection may include background difference detection. In this scenario, the processor can perform background difference detection on the acquired image to obtain the background region composed of pixels that meet a first preset condition. The first preset condition is shown in equation (1).

[0077] (1) In equation (1), Indicates the sensitivity threshold; The background region is represented by a binary mask; Represents pixel coordinates, where u is the horizontal coordinate of the pixel and v is the vertical coordinate of the pixel; Represents the pixel values ​​of the captured image in the current frame; This represents the expected gray level of the background pixels in the image captured in frame t, i.e., the mean value of the background. This indicates the fluctuation range of the background pixel values.

[0078] Then, the processor can perform contour detection on the background region. A continuous region within the background region whose contour area is greater than a preset area threshold is considered a moving foreground region. Understandably, this moving foreground region serves as foreground cue information for a preset semantic segmentation model. The foreground cue information includes at least one of the following: a bounding box cue generated based on the bounding rectangle of the connected components of the foreground region, and a point cue generated based on the centroid coordinates of the connected components of the foreground region.

[0079] Then, the processor can input the moving foreground region and the acquired image into the semantic segmentation model for segmentation processing, obtaining a segmentation mask of the target fabric output by the semantic segmentation model. The aforementioned semantic segmentation model can be implemented using the SAM2 model, which is used to perform fine-grained segmentation of the acquired image based on foreground cues.

[0080] The processor can obtain the contour information of the target fabric based on the above segmentation mask. The contour information of the target fabric includes the contour area, centroid position, and contour point set.

[0081] It should be noted that the contour area and centroid position can be used to associate the same piece of cloth in the current image with historical images. Cloths whose centroid position offset is less than or equal to a preset offset and whose contour area difference is less than or equal to a preset area difference threshold can be identified as the same piece of cloth, allowing for association and matching, and the motion trajectory of the target cloth can be obtained. The centroid position is also used to determine the location of the grab point. The contour point set can be used to determine the contour geometry of the target cloth, the main areas of the target cloth, etc.

[0082] In this embodiment, by performing multi-stage visual detection processing on the acquired images, the success rate and stability of fabric contour detection under complex processes can be improved.

[0083] In another example, multi-stage visual detection may include template difference detection. In this scenario, if the moving foreground region is not obtained during the background difference detection process, for example, if the area of ​​the connected domain is less than the preset area threshold, the processor may use template difference detection, that is, to obtain the difference image between the acquired image and the preset device template, as shown in Equation (2).

[0084] (2) In equation (2), The difference image is the difference image obtained by subtracting the absolute value of the current image from the template image pixel by pixel. Represents the pixel coordinates within the image captured in the current frame. Pixel value at; Represents pixel coordinates within the template image Pixel value at; This indicates that the absolute value is being calculated.

[0085] Understandably, the template in question refers to an image taken when no fabric is placed on the conveyor.

[0086] Then, the processor can filter the aforementioned difference image and apply adaptive thresholding to segment the difference image to obtain candidate cloth regions. These candidate cloth regions serve as foreground cues for the semantic segmentation model. The foreground cues include at least one of the following: bounding box cues generated based on the bounding rectangle of the connected components of the foreground region, and point cues generated based on the centroid coordinates of the connected components of the foreground region. Next, the processor can input the candidate cloth regions and the acquired image into the semantic segmentation model for segmentation processing, obtaining a segmentation mask of the target cloth output by the semantic segmentation model. Finally, the processor can obtain the contour information of the target cloth based on the aforementioned segmentation mask. This contour information includes the contour area, centroid position, and contour point set.

[0087] In one example, the formula for calculating the contour area is shown in equation (3).

[0088] (3) In equation (3), represents the area enclosed by the target fabric outline; n represents the total number of outline points within the target outline; i represents the index of the outline point, with a value range of [0, n-1]; This represents the pixel coordinates of the i-th contour point in the image coordinate system. This represents the pixel coordinates of the first contour point following the i-th contour point, i.e., the (i+1)-th contour point, in the image coordinate system.

[0089] In one example, the centroid is calculated as shown in equations (4)-(8).

[0090] (4) (5) (6) (7) (8) In equations (4)-(8), This represents the sum of the pixel values ​​of the target fabric. This represents the sum of pixel values ​​in the x-direction of the target fabric; This represents the sum of pixel values ​​in the y-direction of the target fabric; and Let x and y represent the x and y coordinates of the centroid C, respectively.

[0091] In this embodiment, template difference is used as a supplementary detection method to still identify the fabric area when background difference fails, thereby improving detection robustness; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive segmentation from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0092] In another example, multi-stage visual detection may include bounding box detection. In this scenario, if template difference detection fails to identify a candidate fabric region, the processor can generate a bounding box based on the long side length, short side length, and centroid position of the target fabric in the detection results of the previous frame. This bounding box can be understood as the minimum bounding rectangle of the target fabric. The processor can then use the bounding box as foreground cue information for the semantic segmentation model, inputting the bounding box and the acquired image into a preset semantic segmentation model for segmentation processing, obtaining a segmentation mask of the target fabric output by the semantic segmentation model.

[0093] In this embodiment, the historical frame information rollback mechanism maintains the continuity of target cloth tracking when the detection of the current frame image fails, thus avoiding the loss of target cloth; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0094] In the above examples, the processor can obtain the contour point set of the target fabric on the conveying device, and then perform polygon approximation processing on the contour point set of the target fabric according to the contour geometry corresponding to the contour point set to obtain the approximate polygon.

[0095] In one example, the processor can obtain the contour geometry corresponding to the contour point set; the contour geometry includes regular geometry and irregular geometry. Regular geometry can include rectangles; irregular geometry can include triangles, L-shapes, or trapezoids. The classification of regular and irregular geometry can be set according to the specific scenario and is not limited herein. It is understood that, considering the semantic segmentation model involved in the above example, the geometry of the segmentation mask can be determined by the semantic segmentation model (e.g., outputting 1 for regular geometry and 0 for irregular geometry), and the geometry output by the semantic segmentation model can be used as the contour geometry, thus achieving the technical solution of this disclosure.

[0096] In response to the contour geometry being a regular geometry, the processor can obtain the smallest bounding polygon enclosing all contour points within the contour point set. The smallest bounding polygon can be implemented using a rectangle, pentagon, or hexagon. Alternatively, in response to the contour geometry being an irregular geometry, the processor can obtain the largest inscribed polygon of the contour formed by the contour point set. The largest inscribed polygon can be implemented using a rectangle, pentagon, or hexagon, etc. It is understood that by determining whether the contour geometry is a regular geometry to select the inscribed or bounding polygon method, the grasping requirements for target fabrics with different geometric shapes can be met.

[0097] After obtaining the minimum bounding polygon or the maximum inscribed polygon, the processor can use multiple approximation accuracies to iteratively search the minimum bounding polygon or the maximum inscribed polygon to obtain the approximate polygon corresponding to each approximation accuracies. That is, one approximate polygon can be searched under each approximation accuracies, and finally multiple approximate polygons are obtained.

[0098] When multiple approximating polygons are obtained, the processor can determine the area containing the approximating polygon with the largest coverage of the target cloth as the main area of ​​the target cloth.

[0099] In this embodiment, the main area that fits the actual shape of the fabric is obtained by iterative approximation with multiple approximation precisions, which can improve the accuracy of subsequent gripping point planning.

[0100] In another example, if the minimum bounding polygon or the maximum inscribed polygon cannot be obtained, the processor can perform point addition based on the contour geometry to generate boundary key points of the contour. For example, if the bounding or inscribed polygon is a quadrilateral, and the minimum bounding or maximum inscribed polygon cannot be obtained, then at least one vertex can be added between the vertices of the quadrilateral to increase the number of polygon vertices, continuing to obtain the minimum bounding or maximum inscribed polygon until it can be obtained. Furthermore, the processor can use the newly added vertices as boundary key points of the contour.

[0101] In step 12, the location within the main body area that meets the safety interval condition is determined as the visual grasping point of the target fabric.

[0102] In this step, after determining the main area, the processor can determine the location within the main area that meets the safety interval condition as the visual gripping point of the target fabric. The processor can obtain the minimum bounding rectangle of the main area, which includes the long side and the short side. Then, the processor can obtain the midpoint of the two short sides of the minimum bounding rectangle as the initial gripping point of the target fabric.

[0103] Suppose that the minimum bounding rectangle H has a width w, a height h and a placement angle theta, and w is greater than or equal to h. Define the center points P1 and P2 of the two short sides of the minimum bounding rectangle H. The four vertices of the minimum bounding rectangle H are arranged in clockwise order. The short sides are the two sides with a length of h that connect the two vertices. The formulas for calculating the midpoint of the short sides are shown in equations (9) and (10).

[0104] (9) (10) In equations (9) and (10), These are the two endpoints of the first short side; These are the two endpoints of the second short side; These are the midpoints of the shortest sides of the second shortest line of the first shortest line.

[0105] Then, the processor can perform distance transformation processing on the main body region to obtain the shortest distance from each pixel position in the main body region to the contour boundary; finally, within the preset neighborhood of the initial grasping point, the processor can search for the position with the largest shortest distance as the visual grasping point.

[0106] In one example, the ratio of the radius of the preset neighborhood range to the length of the short side of the main region is [10%, 30%]. By setting a reasonable search radius, the search efficiency and optimization effect can be balanced.

[0107] In this embodiment, by using distance transformation calculation, the internal point farthest from the boundary of the main area can be searched within the neighborhood of the initial gripping point, ensuring that the gripping point is located in a safe and stable area of ​​the target fabric, thereby improving the success rate of gripping irregularly shaped flexible fabrics.

[0108] In step 13, based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, the visual grasping point is transformed from the pixel coordinate system to the robot coordinate system to obtain the execution grasping point.

[0109] In this step, the robot can store a preset mapping relationship between the pixel coordinate system and the robot coordinate system, or in other words, a mapping relationship between the pixel coordinate system where the vision sensor is located and the coordinate system where the robot arm's end effector is located. It is understood that the above mapping relationship can be obtained by sequentially mapping the pixel coordinate system, camera coordinate system, world coordinate system, robot wall coordinate system, and end effector coordinate system, which will not be elaborated further here.

[0110] The processor can transform the visual grasping points from the pixel coordinate system to the robot coordinate system based on the mapping relationship between the visual grasping points (i.e., pixels on the acquired image) and the preset pixel coordinate system and robot coordinate system, thus obtaining the execution grasping points. It can be understood that the aforementioned execution grasping points refer to the spatial position points that the robot arm's end effector is expected to reach during its spatial movement.

[0111] In another example, the processor can acquire multiple execution grab points and calculate the grab point distance between adjacent execution grab points. In response to an execution grab point distance being less than a preset safety distance threshold, the processor merges adjacent execution grab points, for example, merging two adjacent execution grab points into one execution grab point, thereby ensuring that the distance between all merged execution grab points is greater than or equal to the preset safety distance threshold. The processor can then use the merged grab point to replace the original execution grab point.

[0112] In one example, considering the safe operating range of the robotic arm's end effector, the end effector is modeled as a rectangle based on the aforementioned gripping points. The Y-coordinates of all four corner points of this rectangle are checked to see if they all fall within the safe operating range. If they do, it indicates that the distance between all merged gripping points is greater than or equal to a preset safe distance threshold. By setting this preset safe distance threshold, it is ensured that the end effectors do not interfere with each other, guaranteeing the accuracy and reliability of the gripping operation.

[0113] In step 14, the robotic arm of the robot is controlled to grasp the target fabric according to the grasping point.

[0114] In one example, the robot includes a first robotic arm and a second robotic arm. A processor controls the robot's robotic arms to grasp the target fabric according to the grasping point. This can include: the processor obtaining the rotation angle of the minimum bounding rectangle to determine the orientation angle of the target fabric. Then, the processor can adjust the orientation and pose of the end effectors on the first and second robotic arms according to the orientation angle of the target fabric, aligning the grasping direction of the end effectors with the short side direction of the target fabric. It is understood that the orientation angle of the target fabric is determined based on the rotation angle of the minimum bounding rectangle. Adaptive adjustment of the orientation angle aligns the grasping direction with the fabric texture direction, improving grasping stability.

[0115] Then, based on the pose instructions in the grasping task queue, the processor can perform spatial interference detection based on the current poses and planned trajectories of the first and second robotic arms. In response to passing the spatial interference detection, the processor can control the first and second robotic arms to change from safe poses to grasping poses to move to the execution grasping point of the target fabric and grasp the target fabric. In response to failing the spatial interference detection, the processor skips the current pose instruction, i.e., skips the current grasping operation, and enters a locked state or returns to a safe position.

[0116] In one example, the end effector of the robotic arm includes a lead screw and multiple gripping mechanisms. Based on the length of the long side of the target fabric and the zero-center distance of the lead screw, the processor can determine the positioning distance of the lead screw and evenly distribute the multiple gripping mechanisms on both sides of the long side of the target fabric, and then control the end effector to perform the gripping action. The positioning distance of the lead screw is shown in Equation (11).

[0117] (11) In equation (11), Indicates the length of the long side of the target fabric; This indicates the distance between the center of the gripper and the zero position of the lead screw; This indicates the lead screw positioning distance; a positive value indicates the lead screw is extended, and a negative value indicates the lead screw is retracted. Through adaptive lead screw positioning, the gripping operation automatically adapts to different fabric sizes, eliminating the need for frequent manual adjustments to the end effector spacing, thus improving production line flexibility and gripping stability.

[0118] In this example, in response to grasping the target fabric, the processor can control the first and second robotic arms to change from a grasping pose to a placing pose, and finally return to a safe pose, thus completing one grasping process of the target fabric, and then waiting for the next grasping operation. By setting the robot to have two robotic arms, the collaborative scheduling and collision prevention of the two robotic arms can be used to improve grasping efficiency while ensuring operational safety.

[0119] Based on the above embodiments, this disclosure provides a fabric gripping method, see [link to relevant documentation]. Figure 3 This includes steps 31 to 36: In step 31, the acquired image and working area correction are performed.

[0120] The robot's vision sensors can acquire images of the fabric on the conveyor in real time. Understandably, the vision sensors and the conveyor have already undergone coordinate calibration before use, allowing the acquired images to be directly used for subsequent processing.

[0121] Considering that the positions of the conveying device and / or vision sensor may change, the acquired image can be corrected in this step. For example, a calibration point can be set in the working area of ​​the conveying device (i.e., the area where the target fabric is placed). The coordinates of the calibration point are known, or the calibration point can be marked with an identification code, which can be a QR code. The vision sensor can acquire the signal inside the QR code as the coordinate position of the working area within the conveying device. Then, the perspective transformation mapping relationship between the vision sensor and the working area is re-established, and image correction is performed based on the perspective transformation mapping relationship. The corrected acquired image will be used for subsequent steps. By establishing a perspective transformation mapping relationship through calibration points (or QR codes) and performing perspective correction on the acquired image, the image distortion caused by changes in the camera installation angle or the conveying device is eliminated, resulting in an accurate and reliable image of the working area, i.e., the acquired image.

[0122] In one example, the edge regions of the acquired image can be cropped to remove invalid background areas. The cropped image can then be used in subsequent steps to improve the accuracy of subsequent detection. In step 32, the flexible fabric grasps the target detection.

[0123] The corrected acquired image is then subjected to background segmentation. The moving foreground region is extracted using background subtraction. The background region M is... fgThe segmentation conditions are shown in equation (1).

[0124] After detecting the background region, contour detection can be performed on the background region according to equation (2) to obtain contour information, which can yield at least one connected component and the area of ​​each connected component. If there is a connected component with an area greater than or equal to a preset area threshold, it is taken as a moving foreground region; this moving foreground region serves as the foreground cue information of a preset semantic segmentation model; the foreground cue information includes at least one of the following: a target box cue generated based on the bounding rectangle of the connected components of the foreground region, and a point cue generated based on the centroid coordinates of the connected components of the foreground region. Furthermore, the moving foreground region and the acquired image can be input into the semantic segmentation model for semantic segmentation to obtain the cloth mask output by the semantic segmentation model.

[0125] If semantic segmentation fails, i.e., no effective fabric mask is obtained, then template difference is used to identify the fabric region. Template difference refers to pre-saving a template image, which is an image taken when the conveyor belt is transmitting without fabric. The calculation method of template difference is shown in equation (2), and the difference image is obtained. Then, Gaussian filtering can be performed on the difference image, and adaptive threshold segmentation and connected component area filtering can be used to determine the candidate fabric region.

[0126] If the template differencing method fails to obtain a candidate cloth region, the previous frame's captured image is retrieved. For example, if the current frame's captured image is It, then the previous frame's captured image is It-1. Assuming the target cloth in the previous frame has a long side length of Lp, a short side length of Wp, and a centroid of Cp, a bounding box can be generated and used as foreground cue information.

[0127] Understandably, this step employs a three-tiered progressive approach—background subtraction, template subtraction, and bounding box subtraction—to obtain foreground cue information, thereby acquiring the segmentation mask of the target fabric and thus its outline. This approach can improve the success rate of target recognition under complex conditions.

[0128] In step 33, the target fabric outline and target tracking can be extracted.

[0129] In this step, the contour information of the target fabric can be obtained based on the above segmentation mask, resulting in the contour area, centroid position, and contour point set. The contour area and centroid position are calculated using equations (3) and (4), respectively, and will not be elaborated here.

[0130] In this step, after acquiring the target cloth in the current frame's captured image, it can be correlated and matched with target cloth in historical captured images to achieve continuous tracking of the target cloth. For example, the difference in centroid positions of the target cloth in two adjacent captured images can be obtained. If the difference in centroid positions is less than or equal to a preset centroid difference threshold, it indicates that they may be the same target cloth. Simultaneously, the difference in the contour area of ​​the target cloth in two adjacent captured images can be obtained. If the area difference is less than or equal to a preset area threshold, it indicates that they are the same target cloth, and the target cloth in the two adjacent captured images can be correlated and matched. If either the centroid position or the contour area changes significantly, it indicates that they are not the same target cloth, and correlation and matching cannot be performed. After successful correlation and matching, the motion trajectory of the target cloth in multiple consecutive captured images can be recorded for subsequent motion prediction and grasping decisions.

[0131] In this step, convex hull calculation is performed on the fabric outline. When the target fabric outline geometry is regular, the minimum bounding polygon of the target fabric outline can be obtained; when the target fabric outline geometry is irregular, the maximum inscribed polygon of the target fabric outline can be obtained. Taking the target fabric outline geometry as a regular geometry as an example, a multi-scale contour approximation method is used to iteratively search the convex hull to obtain the minimum bounding quadrilateral that can maximally cover the area where the target fabric is located. When the target outline cannot form a minimum bounding quadrilateral, point filling is performed according to the outline geometry. For example, in scenarios where the fabric shape is too narrow or concave, causing quadrilateral degradation, and a minimum circumscribed quadrilateral with sufficient area cannot be formed, and only a triangle can be used for approximation, a point supplement can be triggered. The longest side of the circumscribed triangle is usually the long side / base of the fabric. By adding a midpoint to its longest side, the triangle becomes a circumscribed quadrilateral where "one side degenerates into two points," generating boundary key points, which are then used as the basis for grabbing.

[0132] In step 34, the grabbing red box and pose optimization are generated.

[0133] In this step, the long side of the minimum bounding quadrilateral of the target fabric represents the direction of the fabric's extension on the conveyor belt of the conveyor device. The minimum bounding rectangle of the segmentation mask M of the target fabric is calculated, and the center points P1 and P2 of the two short sides of the minimum bounding rectangle are calculated, as shown in equations (9) and (10).

[0134] In this step, the center points P1 and P2 are used as boundary key points, i.e., initial grasping points. A distance transformation operation is performed on the area where the target fabric is located, i.e., the main body area, to calculate the shortest distance from each pixel position within the main body area to the contour boundary. The position with the largest shortest distance is searched within the preset neighborhood of the initial grasping point, and the position with the largest shortest distance is used as the visual grasping point.

[0135] The aforementioned visual grasping points can be mapped to the end effector as execution grasping points. When the distance between adjacent execution grasping points is less than the safe distance of the end effector, the grasping points are merged and optimized to avoid interference between the grasping mechanisms.

[0136] In step 35, the robotic arm and end effector are controlled to grasp the object.

[0137] The processor can send the gripping point to the robotic arm, meaning the robotic arm receives the gripping task. After receiving the gripping task, the robotic arm can control the end effector to move from the safe position Ts (the safe pose) to the pre-grip position above the target fabric. Based on the attitude angle and gripping point obtained in step 4, the end effector's orientation and pose are adjusted. Then, it descends along the z-axis to the gripping height to complete the gripping action, i.e., switching to the gripping pose. After gripping, the robotic arm moves the target fabric to the designated placement area Tp and releases it. It then returns to the safe position Ts to await the next gripping task, where Ts and Tp represent fixed coordinates.

[0138] The travel distance of the lead screw is calculated using equation (11). This distance ensures that the grippers of the end effector are positioned at appropriate locations on both sides of the long side of the target fabric during gripping, achieving accurate and reliable gripping.

[0139] In step 36, the capture result is fed back and reset.

[0140] In this step, after the grabbing is completed, the grabbing result information can be recorded, including data such as the target fabric number, grabbing time, grabbing status and target location. The target database and system status information are updated according to the grabbing results, providing data support for subsequent target detection, trajectory prediction and task scheduling.

[0141] Compared with related technologies, the solution in this embodiment, by constructing a multi-stage visual inspection method, can cope with complex working conditions such as changes in lighting, background interference, and fabric wrinkles, achieving a fabric detection success rate of over 99%. Through convex hull inscribed or circumscribed polygon approximation and smooth tracking of gripping point planning, the gripping adaptability and posture estimation stability of irregular fabrics can be improved. It can be adapted to various flexible fabric gripping systems or robots, and is applicable to automated production line scenarios such as dual robotic arm collaboration, dynamic conveyor belt environments, and irregularly shaped fabrics.

[0142] Based on the aforementioned fabric gripping method, this disclosure also provides a fabric gripping device, see [link to relevant documentation]. Figure 4 The device includes: The main body region acquisition module 41 is used to respond to the acquisition of the outline point set of the target fabric on the conveying device, perform polygon approximation processing on the outline point set of the target fabric according to the outline geometry corresponding to the outline point set to obtain an approximation polygon, and take the area enclosed by the approximation polygon as the main body region of the target fabric. The visual grasping point determination module 42 is used to determine the position within the main body area that meets the safety interval condition as the visual grasping point of the target fabric; The execution gripping point determination module 43 is used to transform the visual gripping point from the pixel coordinate system to the robot coordinate system based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, so as to obtain the execution gripping point; The target fabric gripping module 44 is used to control the robot's robotic arm to grip the target fabric according to the gripping point.

[0143] In this embodiment, after obtaining the contour point set of the target fabric, the approximate polygon of the target fabric contour can be obtained, which improves the accuracy of obtaining the main area of ​​the target fabric, thereby realizing adaptive gripping point planning for irregularly shaped fabrics, ensuring that the gripping points are located in the safe stress area, and improving the gripping success rate of irregularly shaped flexible fabrics.

[0144] In one embodiment, the visual grasping point determination module includes: The segmentation mask acquisition submodule is used to perform multi-stage visual inspection on the acquired image of the target fabric on the conveying device in response to acquiring the image of the target fabric to obtain the segmentation mask of the target fabric. The fabric contour acquisition submodule is used to acquire the contour information of the target fabric according to the segmentation mask. The contour information of the target fabric includes the contour area, centroid position, and contour point set.

[0145] In this embodiment, by performing multi-stage visual inspection on the acquired images, the success rate and stability of fabric contour detection under complex working conditions can be improved.

[0146] In one embodiment, the segmentation mask acquisition submodule includes: A moving foreground region determination unit is used to perform background difference detection on the acquired image to obtain pixels that meet a first preset condition to form a moving foreground region; the moving foreground region serves as foreground prompt information of a preset semantic segmentation model; the foreground prompt information includes at least one of the following: a target box prompt generated based on the bounding rectangle of the connected domain of the foreground region, and a point prompt generated based on the centroid coordinates of the connected domain of the foreground region; The segmentation mask determination unit is used to input the moving foreground region and the acquired image into the semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0147] In this embodiment, background subtraction can quickly filter candidate foreground regions, reduce the processing range of subsequent segmentation models, and improve overall detection efficiency; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0148] In one embodiment, the segmentation mask acquisition submodule further includes: A differential image acquisition unit is used to acquire a differential image between the acquired image and a preset device template in response to the background differential detection not obtaining the moving foreground region; the device template refers to an image taken when there is no fabric on the conveying device; the differential image is filtered and then segmented using an adaptive threshold to obtain candidate fabric regions; The segmentation mask acquisition unit is used to input the candidate fabric region as foreground cue information of the semantic segmentation model into the preset semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0149] In this embodiment, template difference is used as a supplementary detection method to still identify the fabric area when background difference fails, thereby improving detection robustness; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive segmentation from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0150] In one embodiment, the segmentation mask acquisition submodule further includes: The bounding box generation unit is used to generate a bounding box based on the length of the long side, the length of the short side, and the centroid position of the target fabric in the detection result corresponding to the previous frame image in response to the absence of a candidate fabric region. The bounding box serves as foreground cue information for a preset semantic segmentation model. The segmentation mask acquisition unit is used to input the bounding box and the acquired image into a preset semantic segmentation model for segmentation processing, and obtain the segmentation mask of the target fabric output by the semantic segmentation model.

[0151] In this embodiment, the historical frame information rollback mechanism maintains the continuity of target cloth tracking when the detection of the current frame image fails, thus avoiding the loss of target cloth; and the foreground detection results are used to guide the semantic segmentation model to achieve progressive from coarse segmentation to fine segmentation in order to obtain an accurate and complete segmentation mask.

[0152] In one embodiment, after obtaining the segmentation mask of the target fabric, the visual grasping point determination module includes: The motion trajectory determination module is used to associate and match the target fabric corresponding to the currently acquired image with the target fabric corresponding to the historical acquired images to obtain the continuous motion trajectory of the target fabric.

[0153] In this embodiment, by tracking the target fabric in association with multiple frames of acquired images, independent tracking and motion prediction of each target fabric can be achieved.

[0154] In one embodiment, the association matching is based on at least one of the centroid distance and the profile intersection-to-union ratio of the target fabric.

[0155] In one embodiment, the visual grasping point determination module includes: The minimum convex polygon acquisition submodule is used to acquire the minimum convex polygon that encloses all contour points within the set of contour points. The approximation polygon determination submodule is used to iteratively search the minimum bounding polygon or the maximum inscribed polygon using multiple approximation accuracies to obtain the approximation polygons corresponding to each approximation accuracies. The main area determination module is used to determine the area containing the approximate polygon that has the largest coverage of the target fabric as the main area of ​​the target fabric.

[0156] In this embodiment, the main area that fits the actual shape of the fabric is obtained by iterative approximation with multiple approximation accuracies, thereby improving the accuracy of subsequent gripping point planning.

[0157] In one embodiment, the visual grasping point determination module includes: The boundary key point determination submodule is used to generate the boundary key points of the contour by performing point filling processing based on the geometry of the contour in response to the absence of a minimum bounding polygon or a maximum inscribed polygon. The visual grasping point determination submodule is used to determine the boundary key points of the contour as the visual grasping points.

[0158] In this embodiment, point supplementation processing ensures that effective gripping points can be generated for all types of fabric, thereby improving the robustness of the system.

[0159] In one embodiment, the visual grasping point determination module includes: The minimum bounding rectangle acquisition submodule is used to acquire the minimum bounding rectangle of the main area, wherein the minimum bounding rectangle includes the long side and the short side; The initial gripping point acquisition submodule is used to acquire the midpoints of the two short sides of the minimum bounding rectangle as the initial gripping points of the target fabric. The shortest distance determination submodule is used to perform distance transformation processing on the main body region to obtain the shortest distance from each pixel position in the main body region to the contour boundary. The visual grasping point determination submodule is used to search for the position with the largest shortest distance within a preset neighborhood of the initial grasping point as the visual grasping point.

[0160] In this embodiment, by using distance transformation calculation, the internal point farthest from the boundary can be searched within the neighborhood of the initial gripping point, ensuring that the gripping point is located in a safe and stable area of ​​the target fabric, thereby improving the success rate of gripping irregularly shaped flexible fabrics.

[0161] In one embodiment, the ratio of the radius of the preset neighborhood range to the length of the shorter side is [10%, 30%].

[0162] In this embodiment, by setting a reasonable search radius, search efficiency and optimization effect can be balanced.

[0163] In one embodiment, the device further includes: The crawl point distance acquisition module is used to calculate the crawl point distance between adjacent crawl points in response to the acquisition of multiple crawl points; The execution grab point determination module is used to merge adjacent execution grab points in response to the distance between the execution grab points being less than a preset safety distance threshold, and to replace the execution grab points with the merged grab points.

[0164] In this embodiment, by optimizing the grab point merging process, interference from the end effector can be avoided, ensuring the safety of multi-grab point collaborative operation.

[0165] In one embodiment, the ratio of the preset safety distance threshold to the size of the end effector is [1.2, 1.5].

[0166] In one embodiment, the robot's robotic arm includes a first robotic arm and a second robotic arm; the target fabric gripping module includes: The attitude angle acquisition submodule is used to acquire the rotation angle of the minimum bounding rectangle corresponding to the target fabric and determine the attitude angle of the target fabric. The orientation and pose adjustment submodule is used to adjust the orientation and pose of the end effectors on the first and second robotic arms according to the orientation angle of the target fabric, so that the gripping direction of the end effector is aligned with the short side direction of the target fabric; the orientation angle of the target fabric is determined based on the rotation angle of the minimum bounding rectangle; The spatial interference detection submodule is used to perform spatial interference detection based on the pose instructions in the grasping task queue and the current pose and planned trajectory of the first robotic arm and the second robotic arm. The grasping pose switching submodule is used to control the first robotic arm and the second robotic arm to change from a safe pose to a grasping pose in response to spatial interference detection so as to move to the execution grasping point of the target fabric. The pose transformation submodule is used to control the first robotic arm and the second robotic arm to transform from the grasping pose to the placement pose in response to the grasping of the target fabric, and finally return to the safe pose transformation.

[0167] In this embodiment, by equipping the robot with two robotic arms, the collaborative scheduling and collision prevention of the two robotic arms can be utilized to improve grasping efficiency while ensuring operational safety.

[0168] In one embodiment, the end effector of the robotic arm includes a lead screw and multiple gripping mechanisms, and the device further includes: The positioning distance determination module is used to determine the positioning distance of the lead screw based on the length of the long side of the target fabric and the zero-position center distance of the lead screw, and to evenly distribute the multiple gripping mechanisms on both sides of the long side of the target fabric.

[0169] In this embodiment, the gripping direction is aligned with the fabric texture direction by adaptive adjustment of the attitude angle, thereby improving gripping stability; the gripping operation of different sized fabrics is automatically adapted by adaptive positioning of the lead screw, without the need for frequent manual adjustment of the end effector spacing, thus improving the flexibility of the production line and gripping stability.

[0170] It is understood that the various embodiments of the fabric gripping device provided in this embodiment have been described when describing the fabric gripping method. The embodiments of the device can be referred to the content of the method embodiments, and will not be repeated here.

[0171] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of any of the above embodiments.

[0172] This disclosure also provides a computing device, including a memory and a processor; the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the methods of any of the above embodiments when executing the computer instructions.

[0173] Figure 5 This is a schematic diagram of the structure of a computing device provided in this disclosure, such as... Figure 5 As shown, the computing device 5 may include, but is not limited to: processor 51, memory 52, and bus 53 connecting different system components (including memory 52 and processor 51).

[0174] The memory 52 stores computer instructions that can be executed by the processor 51, enabling the processor 51 to perform the fabric gripping method of any of the above embodiments. The memory 52 may include a random access memory (RAM) 521, a cache memory 522, and / or a read-only memory (ROM) 523. The memory 52 may also include a program tool 525 having a set of program modules 524, including but not limited to: an operating system, one or more application programs, other program modules, and program data. One or more combinations of these program modules may include an implementation of a network environment.

[0175] Bus 53 may include, for example, a data bus, an address bus, and a control bus. The computing device 5 can also communicate with external devices 55 via I / O interface 54, such as a keyboard or a Bluetooth device. The computing device 5 can also communicate with one or more networks via network adapter 56, such as a local area network (LAN), a wide area network (WAN), or a public network. As shown in the figure, the network adapter 56 can also communicate with other modules of the computing device 5 via bus 53.

[0176] This disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to implement the methods described above.

[0177] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0178] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be noted that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for grasping fabric, characterized in that, The method includes: In response to obtaining the contour point set of the target fabric on the conveying device, the contour point set of the target fabric is subjected to polygon approximation processing according to the contour geometry corresponding to the contour point set to obtain an approximation polygon, and the area enclosed by the approximation polygon is taken as the main area of ​​the target fabric. The location within the main body area that meets the safety interval condition is determined as the visual grasping point of the target fabric; Based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, the visual grasping point is transformed from the pixel coordinate system to the robot coordinate system to obtain the execution grasping point; The robotic arm of the robot is controlled to grasp the target fabric according to the grasping point.

2. The method according to claim 1, characterized in that, Obtain the contour point set of the target fabric on the conveying device, including: In response to acquiring a captured image of the target fabric on the conveying device, a multi-stage visual inspection is performed on the captured image to obtain a segmentation mask of the target fabric; The contour information of the target fabric is obtained according to the segmentation mask. The contour information of the target fabric includes the contour area, centroid position, and contour point set.

3. The method according to claim 2, characterized in that, Multi-stage visual inspection is performed on the acquired image to obtain a segmentation mask for the target fabric, including: Background difference detection is performed on the acquired image to obtain the background region composed of pixels that meet the first preset condition; Based on the outline of the background region, connected regions with an area greater than or equal to a preset area threshold are determined as moving foreground regions; the moving foreground regions serve as foreground cue information for a preset semantic segmentation model; the foreground cue information includes at least one of the following: a target bounding box cue generated based on the bounding rectangle of the connected regions of the foreground region, and a point cue generated based on the centroid coordinates of the connected regions of the foreground region. The moving foreground region and the acquired image are input into the semantic segmentation model for segmentation processing to obtain the segmentation mask of the target fabric output by the semantic segmentation model.

4. The method according to claim 3, characterized in that, The method further includes: In response to the background difference detection failing to obtain the moving foreground region, a difference image is acquired between the acquired image and a preset device template; the device template refers to an image taken when there is no fabric on the conveying device; the difference image is filtered and then segmented using an adaptive threshold to obtain candidate fabric regions; The candidate fabric region is used as the foreground cue information of the semantic segmentation model and input into the preset semantic segmentation model for segmentation processing to obtain the segmentation mask of the target fabric output by the semantic segmentation model.

5. The method according to claim 4, characterized in that, The method further includes: In response to the absence of a candidate fabric region, a bounding box is generated based on the long side length, short side length, and centroid position of the target fabric in the detection results of the previous frame image; the bounding box serves as foreground cue information for a preset semantic segmentation model. The bounding box and the acquired image are input into a preset semantic segmentation model for segmentation processing to obtain a segmentation mask of the target fabric output by the semantic segmentation model.

6. The method according to claim 1, characterized in that, After obtaining the segmentation mask of the target fabric, the method further includes: The target fabric corresponding to the currently acquired image is associated and matched with the target fabric corresponding to the historical acquired images to obtain the continuous motion trajectory of the target fabric.

7. The method according to claim 1, characterized in that, Based on the contour geometry corresponding to the contour point set, the contour point set of the target fabric is subjected to polygon approximation processing to obtain an approximate polygon, and the area enclosed by the approximate polygon is taken as the main area of ​​the target fabric, including: Obtain the contour geometry corresponding to the contour point set; the contour geometry includes regular geometry and irregular geometry; In response to the contour geometry being a regular geometry, obtain the smallest bounding polygon that surrounds all contour points within the contour point set; or, in response to the contour geometry being an irregular geometry, obtain the largest inscribed polygon of the contour formed by the contour point set. The minimum bounding polygon or the maximum inscribed polygon is iteratively searched using multiple approximation accuracies to obtain the approximation polygons corresponding to each approximation accuracies. The region containing the approximate polygon that has the largest coverage of the target fabric is determined as the main region of the target fabric.

8. The method according to claim 7, characterized in that, The method further includes: In response to the absence of a minimum bounding polygon or a maximum inscribed polygon, point filling is performed based on the geometry of the contour to generate the boundary key points of the contour. The boundary key points of the contour are determined as the visual grasping points.

9. The method according to claim 1, 7 or 8, characterized in that, Determining the location within the main body area that meets the safety interval condition as the visual grasping point of the target fabric includes: Obtain the minimum bounding rectangle of the main region, wherein the minimum bounding rectangle includes the long side and the short side; Obtain the midpoints of the two short sides of the minimum bounding rectangle as the initial gripping points of the target fabric; Perform distance transformation processing on the main body region to obtain the shortest distance from each pixel position within the main body region to the contour boundary; Within a preset neighborhood of the initial grasping point, the location with the largest shortest distance is searched as the visual grasping point.

10. The method according to claim 9, characterized in that, The ratio of the radius of the preset neighborhood range to the length of the shorter side is [10%, 30%].

11. The method according to claim 1, characterized in that, The method further includes: In response to obtaining multiple execution crawl points, calculate the crawl point distance between adjacent execution crawl points; In response to the fact that the distance between the execution grab points is less than a preset safety distance threshold, adjacent execution grab points are merged, and the merged grab points are used to replace the execution grab points.

12. The method according to claim 1, characterized in that, The robot's robotic arm includes a first robotic arm and a second robotic arm; controlling the robot's robotic arm to grasp the target fabric according to the execution gripping point includes: Obtain the rotation angle of the minimum bounding rectangle corresponding to the target fabric, and determine the attitude angle of the target fabric; The orientation and pose of the end effectors on the first and second robotic arms are adjusted according to the attitude angle of the target fabric so that the gripping direction of the end effectors is aligned with the short side direction of the target fabric; the attitude angle of the target fabric is determined based on the rotation angle of the minimum circumscribed rectangle. Based on the pose instructions in the grasping task queue, spatial interference detection is performed based on the current poses and planned trajectories of the first and second robotic arms; In response to spatial interference detection, the first robotic arm and the second robotic arm are controlled to change from a safe pose to a grasping pose to move to the execution grasping point of the target fabric. In response to grasping the target fabric, the first robotic arm and the second robotic arm are controlled to change from the grasping posture to the placement posture, and finally return to the safe posture transformation.

13. The method according to claim 1, characterized in that, The end effector of the robotic arm includes a lead screw and multiple gripping mechanisms, and the method further includes: Based on the length of the long side of the target fabric and the zero-center distance of the lead screw, the positioning distance of the lead screw is determined, and the multiple gripping mechanisms are evenly distributed on both sides of the long side of the target fabric.

14. A fabric gripping device, characterized in that, The device includes: The main area acquisition module is used to respond to the acquisition of the outline point set of the target fabric on the conveying device, and to perform polygon approximation processing on the outline point set of the target fabric according to the outline geometry corresponding to the outline point set to obtain the approximate polygon. The visual grasping point determination module is used to take the area enclosed by the approximation polygon as the main area of ​​the target fabric; The visual grasping point determination module is used to determine the positions within the main body area that meet the safety interval conditions as the visual grasping points of the target fabric; The execution gripping point determination module is used to transform the visual gripping point from the pixel coordinate system to the robot coordinate system based on the preset mapping relationship between the pixel coordinate system and the robot coordinate system, so as to obtain the execution gripping point; The target fabric gripping module is used to control the robot's robotic arm to grip the target fabric according to the gripping point.

15. A robot, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the method of any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, performs the method described in any one of claims 1-13.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the method according to any one of claims 1-13.

Images