A visual identification and optimized sorting method and system for carbon fiber composite material cutting pieces

Abstract

The present application relates to the technical field of intelligent manufacturing of carbon fiber composite materials, in particular to a visual identification and optimized sorting method and system for carbon fiber composite material cutting pieces. The method comprises the following steps: S1: collecting visual data of the cutting piece stacking area; S2: based on the visual data, identifying and positioning multiple independent cutting pieces in the stacking state, and determining the pose and at least one grabbing point of the cutting pieces in the three-dimensional space; S3: according to the pose and grabbing point information of all cutting pieces, taking minimizing the total time consumption of robot sorting as the optimization goal, planning the grabbing sequence and motion path of the robot; S4: controlling the robot to sequentially perform grabbing and placing operations on each cutting piece according to the grabbing sequence and motion path. The corresponding system comprises a visual acquisition module, a calculation and control unit, and a robot execution module. The present application realizes the full-automatic closed-loop sorting of stacked carbon fiber composite material cutting pieces from perception, decision-making to execution, improves the sorting efficiency and adaptability, and ensures the lossless operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology for carbon fiber composite materials, and in particular to a visual recognition and optimized sorting method and system for carbon fiber composite material cut pieces. Background Technology

[0002] Carbon fiber composites are increasingly widely used in aerospace, rail transportation, new energy vehicles, wind turbine blades, and other fields due to their excellent properties such as high specific strength, high specific modulus, high temperature resistance, and fatigue resistance. The manufacturing process of carbon fiber composite parts typically includes multiple steps such as prepreg preparation, cutting, layup, curing, and machining. Among these, the sorting of prepreg sheets after cutting is a key process connecting cutting and layup.

[0003] Currently, the sorting and stacking of carbon fiber composite sheet material after cutting largely relies on manual labor, which presents the following technical problems: Inefficient and labor-intensive: Carbon fiber composite material pieces come in a variety of sizes and complex shapes (such as irregularly shaped pieces for aircraft skin). Manual sorting requires identifying, picking, handling and stacking them one by one, which is difficult to meet the needs of large-scale production, and long hours of work can easily lead to worker fatigue. Prone to damage during cutting: Before curing, carbon fiber prepreg is in a soft state, and the surface resin is sticky. Improper handling or pulling during manual sorting can easily cause fibers to bend, wrinkle, delaminate, or even break, resulting in material scrap. Carbon fiber composites are sensitive to damage; once microcracks occur, they will seriously affect the mechanical properties of the final part. Poor sorting consistency: Manual sorting makes it difficult to ensure the consistency of the placement direction and stacking accuracy of each piece of cut material, which affects the positioning accuracy of subsequent automatic lay-up processes. Cleanliness is difficult to guarantee: human contact can easily leave contaminants such as sweat and oil on the surface of the prepreg, affecting the quality of interlayer bonding.

[0004] There are already some automated sorting solutions in the existing technology, such as: Machine vision-based grasping systems use vision to locate the position of cut pieces and control the robotic arm to grasp them. However, these systems are mostly designed for regular, single-layer cut pieces and have poor adaptability to stacked, adhered, and differently shaped composite material cut pieces. Furthermore, they do not consider optimizing the grasping order among multiple cut pieces, resulting in redundant robotic arm paths and low efficiency. Rigid suction cup or gripper gripping method: Traditional end effectors mostly use suction cups or rigid grippers of fixed height, which cannot adapt to the slight undulations and warping of carbon fiber composite material cut pieces caused by stacking. When the cut piece is at the bottom of the stack, the surface may bend due to the pressure of the upper layer, and the rigid suction cup cannot fit tightly, resulting in adsorption failure or damage to the cut piece; Sorting methods based on optimized grasping order: These methods optimize the grasping order of cut pieces through algorithms, reducing the movement time of the robotic arm. However, these methods often assume that the position of the cut pieces is known and fixed, and they are not deeply integrated with real-time visual recognition, making them unable to cope with situations where the position of the cut pieces is random and their posture is variable in actual production. Multimodal sensing layup defect detection system (such as CN120778751A): This patent uses hyperspectral and laser displacement sensors to achieve real-time detection and closed-loop control of layup defects, reflecting the application trend of multi-sensor fusion and intelligent decision-making in composite material manufacturing. However, this system focuses on defect detection and compensation during the layup process and does not cover the sorting and stacking of cut pieces after cutting.

[0005] In summary, existing technologies have not yet achieved a fully automated closed loop from visual recognition to intelligent decision-making to robotic grasping of composite material cutting pieces. Especially in scenarios with multiple cutting pieces, random stacking, and varied postures, there is a lack of an automated sorting system that can perceive in real time, make intelligent decisions, and execute efficiently, making it difficult to meet the demand for high-quality and high-efficiency manufacturing of carbon fiber composite components. Summary of the Invention

[0006] The purpose of this invention is to solve the problems existing in the prior art, and to propose a visual recognition and optimized sorting method and system for carbon fiber composite material cut pieces.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a visual recognition and optimized sorting method for carbon fiber composite material cut pieces, comprising the following steps: S1: Collect visual data of the stacked area of ​​the cut pieces; S2: Based on the visual data, identify and locate multiple independent cut pieces in the stacked state, and determine the pose of each cut piece in three-dimensional space and the grasping point information of at least one grasping point; S3: Based on the pose and gripping point information of all the cut pieces, with the optimization goal of minimizing the total time spent by the robot sorting, plan the robot's gripping sequence and motion path; S4: Control the robot to perform grasping and placement operations on each piece of fabric in sequence according to the planned grasping order and movement path.

[0008] Further, in step S1, the visual data includes a two-dimensional color image acquired by a vertically positioned area array industrial camera, and depth information acquired by a 3D vision sensor; the two-dimensional color image and the depth information are registered and fused to form point cloud data containing color and three-dimensional coordinates.

[0009] Further, step S2 includes: S21: Input the two-dimensional color image into the trained instance segmentation deep learning model to obtain the pixel-level contour mask and category information of each clipping; S22: Combine the contour mask and the point cloud data to extract the three-dimensional point cloud clusters corresponding to each piece; S23: Perform principal component analysis on each 3D point cloud cluster to calculate its 3D spatial position, surface normal vector and rotation angle in the horizontal plane, as the pose of the cut piece; S24: Based on the geometric features of the cut piece contour mask and the surface flatness of its corresponding three-dimensional point cloud cluster, a stable region with a flat surface and close to the two-dimensional centroid of the contour mask is selected inside the contour mask, and the three-dimensional center point of the stable region is calculated as the grasping point.

[0010] Further, step S3 includes: S31: Construct a mixed-integer programming model, where the decision variables represent the grasping order and the objective function is the sum of the estimated time costs for the robot to complete all grasping, moving, and placing actions; S32: The sum of the estimated time costs includes at least the robot's motion time and the grasping action adjustment time t. a The grasping action adjustment time t a The stability evaluation value S of the gripping point calculated based on the surface flatness k Negative correlation; S33: Solve the mixed integer programming model using a genetic algorithm or simulated annealing algorithm, and output the globally optimal grasping order and the corresponding key points of the robot motion path.

[0011] Furthermore, the stability evaluation value S k Calculate using the following formula: S k = 1 / (1 + σ z 2 ), where σ z 2 The height variance of the 3D point cloud cluster in the region where the capture point is located.

[0012] Furthermore, in step S4, the robot performs a grasping operation through an end effector, which includes a mounting base and at least three suction cup units; each suction cup unit is independently mounted on the mounting base, and each suction cup unit has an independent vertical drive mechanism and a vacuum adsorption channel.

[0013] Furthermore, each of the suction cup units includes: A miniature electric actuator, the cylinder of which is fixed to the mounting base, constitutes the independent vertical drive mechanism; A flexible corrugated tube, the upper end of which is connected to the push rod end of the miniature electric push rod; A suction head is connected to the lower end of the flexible corrugated tube, and the inner cavity of the suction head is connected to the vacuum adsorption channel through the interior of the flexible corrugated tube. A force sensor is installed at the end of the miniature electric push rod or on the suction cup head to detect the contact force between the suction cup head and the surface of the cut piece. The miniature electric actuator can drive the suction head to extend and retract independently in the vertical direction, and the feedback signal from the force sensor is used to control the stop position of the miniature electric actuator.

[0014] Furthermore, during the grasping operation in step S4, the control method for the end effector includes: S41: Control each of the aforementioned miniature electric push rods to synchronously drive the suction cup head to move downwards; S42: When the contact force detected by the force sensor of any suction cup unit reaches the preset safety threshold, control the miniature electric push rod of that suction cup unit to stop. S43: After all suction heads have stopped, start vacuum suction; S44: After confirming that the adsorption is stable, control the robot to perform lifting and transfer actions.

[0015] A visual recognition and optimized sorting system for carbon fiber composite material cut pieces, implementing the visual recognition and optimized sorting method for carbon fiber composite material cut pieces as described in any of the above claims, comprising: A visual acquisition module is fixed above the fabric piece stacking area to acquire visual data of the stacked fabric pieces in the fabric piece stacking area; The computational control unit is used to run visual recognition algorithms, grasping sequence optimization algorithms, and generate robot control commands; A robot execution module includes a robot and the end effector mounted at its end; The computing control unit is communicatively connected to both the vision acquisition module and the robot execution module.

[0016] Furthermore, the vision acquisition module includes a rigid bracket and a high-resolution area array industrial camera and a 3D vision sensor fixed to the rigid bracket by an adjustable mounting mechanism. The adjustable mounting mechanism can adjust and lock the posture of the area array industrial camera and the 3D vision sensor to ensure that their field of view centers coincide and the optical axis parallelism error is less than 0.1°. The 3D vision sensor is a 3D structured light projector or a binocular camera.

[0017] Compared with existing technologies, the advantages of this invention are as follows: 1. Fully automated closed-loop sorting: It connects the entire chain of "visual recognition → intelligent decision-making → robot grasping", realizing fully automated closed-loop operation from perception of the stacked area of ​​cut pieces, identification and positioning of multiple cut pieces, optimization planning of grasping order to autonomous execution by the robot, completely replacing manual sorting; 2. Highly adaptable recognition of stacked pieces: The two-dimensional color image is registered and fused with depth information to form RGB-D point cloud data. The instance segmentation deep learning model is used to obtain the contour mask of each piece. Based on the surface flatness, a stable area is selected within the contour mask as the grasping point. The positioning accuracy is high, which can effectively avoid adsorption failure and can effectively handle stacked, adhered, and differently shaped pieces. 3. Global path optimization that integrates grasping reliability: A mixed integer programming model is constructed with the goal of minimizing the total time spent by the robot in sorting. Genetic algorithms or simulated annealing algorithms are used to solve for the globally optimal grasping order and motion path, thereby reducing the total time spent by the robot in sorting. 4. Adaptive force-controlled flexible gripping: The end effector contains at least three independent suction cup units. During gripping, each suction cup unit descends synchronously. Once any suction cup contacts the cut piece, the descent of that suction cup stops. After all suction cups have stopped, vacuum adsorption is activated to ensure that each suction cup gently adheres to the surface of the cut piece, achieving non-destructive and reliable gripping. 5. The system has strong scalability: By training instance segmentation deep learning models and adjusting key parameters such as the safety threshold, flatness threshold, and weight coefficient of the gripping order optimization algorithm of the suction cup unit contact force, it can quickly adapt to various composite materials such as carbon fiber, glass fiber, and basalt fiber, as well as various cut pieces such as rectangles and irregular shapes. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the mechanical structure of the sorting system of the present invention. Detailed Implementation

[0019] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings. First, some key terms involved in this invention will be uniformly explained: Contour mask: A contour mask is a binary image representing a pixel-level region of a single crop in a 2D color image, obtained after processing the image using a deep learning model for instance segmentation. In this contour mask, pixels belonging to the crop have a value of 1 (or 255), and others have a value of 0.

[0021] Point cloud data refers to a collection of coordinates of a large number of three-dimensional points, converted from depth information acquired by a 3D vision sensor. Each point contains (X, Y, Z) three-dimensional coordinates. After registration and fusion with a two-dimensional color image, point cloud data containing both color (R, G, B) and (X, Y, Z) three-dimensional coordinates can be formed, i.e., RGB-D point cloud data.

[0022] 3D point cloud cluster: refers to a subset of all 3D points belonging to a certain piece of the scene, extracted from the point cloud data of the entire scene based on the contour mask of that piece.

[0023] Pose: In this invention, pose refers to the complete spatial state of the cut piece in three-dimensional space, including its position (usually represented by three-dimensional coordinates) and orientation (usually represented by rotation angle or direction vector). For sheet-like cut pieces, their pose can be defined by the coordinates of the center point (three-dimensional spatial position), the surface normal vector (characterizing the orientation of the plane), and the rotation angle in the plane (characterizing rotation about the normal).

[0024] Example 1, as Figure 1 As shown, a visual recognition and optimized sorting system for carbon fiber composite material cut pieces includes: A visual acquisition module is fixed above the fabric piece stacking area to acquire visual data of the stacked fabric pieces in the fabric piece stacking area; The computational control unit is used to run visual recognition algorithms, grasping sequence optimization algorithms, and generate robot control commands; A robot execution module includes a robot and the end effector mounted at its end; A workbench is used to place carbon fiber composite material cut pieces that are stacked in a disordered state and are to be sorted. The computing control unit is communicatively connected to both the vision acquisition module and the robot execution module.

[0025] The vision acquisition module includes a rigid bracket and a high-resolution area array industrial camera and a 3D vision sensor fixed to the rigid bracket by an adjustable mounting mechanism. The adjustable mounting mechanism can adjust and lock the posture of the area array industrial camera and the 3D vision sensor to ensure that the centers of their fields of view coincide and the optical axis parallelism error is less than 0.1°. The 3D vision sensor is a 3D structured light projector or a binocular camera.

[0026] In this embodiment, the specific system hardware configuration and mechanical structure are as follows: 1. Visual Acquisition Module: Structure: The vision acquisition module is constructed from a robust aluminum profile gantry frame (rigid support), spanning above the worktable. On the crossbeam of the gantry frame, a high-precision 3D adjustable pan-tilt head (adjustable mounting mechanism) is used to fix the following components in parallel: Area array industrial camera: A 20-megapixel area array industrial camera is used to acquire high-definition two-dimensional color images (i.e., RGB color images). 3D vision sensor: As a specific implementation method for acquiring depth information, this embodiment uses a 3D structured light projector (or binocular camera) based on the principle of structured light, which integrates a structured light projector and an image sensor.

[0027] Connectivity and Functionality: The area scan industrial camera is used to acquire high-resolution RGB color images, providing rich color, texture, and edge information. The 3D structured light projector actively projects an coded grating and receives the grating, which is deformed by the object's surface shape, through its built-in image sensor. Using triangulation principles, it quickly calculates the depth value (Z coordinate) of each point within the field of view, generating a depth map. Both are connected to the computing control unit via a GigE or USB 3.0 data cable. Through pre-performed hand-eye calibration and binocular (RGB-D) calibration, a precise mapping relationship between the 2D image pixel coordinates and the 3D world coordinates is established, enabling real-time fusion of the 2D color image and the 3D point cloud data.

[0028] The three-dimensional adjustable gimbal has adjustment and locking functions with three degrees of freedom: pitch, yaw, and roll. It is used to finely adjust the attitude of the area array industrial camera and the 3D structured light projector during installation and calibration, ensuring that the centers of their fields of view coincide and the optical axis parallelism error is less than 0.1°.

[0029] 2. Workbench: Composition: Countertop: Made of flat 304 stainless steel plate, with a wear-resistant and anti-static coating sprayed on the surface; Support structure: A sturdy frame welded from structural steel, with adjustable feet installed at the bottom; Positioning device: The edge of the table is equipped with a limiting baffle, which can be adjusted to accommodate cut pieces of different sizes in a stacked state; Function: Used to place randomly stacked cut pieces to be sorted. Its flat and stable surface ensures the accuracy of visual acquisition and robot operation.

[0030] 3. Robot Execution Module: Main body: A six-axis articulated industrial robot with a repeatability of ±0.02mm and a payload of over 5kg. The robot's base is fixed to the ground.

[0031] Core components: such as Figure 1 As shown, its detailed structure is as follows: Mounting base: It is a circular aluminum alloy flange with a standard interface (such as ISO 9409-1) in the center for docking with the sixth axis flange of the robot. The inside of the flange is machined with an annular main air passage and a cable tray for wiring.

[0032] Suction cup units: Four units are evenly distributed at 90° angles to the center of the mounting base. Each suction cup unit is an independent functional module, containing: Miniature electric linear actuator: A stepper or servo electric linear actuator with a stroke of 30mm is selected, and its housing is vertically fixed to the mounting base by screws.

[0033] Flexible corrugated tube: Made of oil-resistant and fatigue-resistant silicone material, the upper end is connected to the push rod head of the miniature electric actuator via a clamp, and the lower end is connected to the suction cup head. Its unique corrugated design allows it to withstand a certain degree of lateral bending while undergoing axial compression / extension.

[0034] Suction head: Made of soft polyurethane material, with an internal cavity and a vent at the top that connects to the main air passage through a corrugated tube.

[0035] Force sensor: Located at the end of the miniature electric actuator or the suction cup head. In this embodiment, it is integrated at the end of the miniature electric actuator, specifically at the connection between the actuator end and the bellows. It is a miniature strain gauge sensor with a range of 0-20N, used for real-time measurement of contact pressure.

[0036] Air and electrical circuits: The external vacuum generator connects to the air inlet of the mounting base via a quick connector, and the air is then distributed to four independent solenoid valves, which in turn connect to each suction head, enabling independent control of the suction force. Force sensor signal lines and electric actuator control lines converge at the aviation connector on the mounting base, and then connect to the robot control cabinet.

[0037] 4. Computing and Control Unit: This is a high-performance industrial computer equipped with an Intel i7 processor, 32GB of memory, and an NVIDIA RTX4060 GPU. It is responsible for running visual recognition algorithms and grasping sequence optimization algorithms, and acts as the robot's host computer for trajectory planning and command issuance.

[0038] Example 2: A visual recognition and optimized sorting method for carbon fiber composite material cut pieces, applied to a visual recognition and optimized sorting system for carbon fiber composite material cut pieces in the above examples, includes the following steps: S1: Collect visual data of the stacked area of ​​the cut pieces; S2: Based on the visual data, identify and locate multiple independent cut pieces in the stacked state, and determine the pose of each cut piece in three-dimensional space and the grasping point information of at least one grasping point; S3: Based on the pose and gripping point information of all the cut pieces, with the optimization goal of minimizing the total time spent by the robot sorting, plan the robot's gripping sequence and motion path; S4: Control the robot to perform grasping and placement operations on each piece of fabric in sequence according to the planned grasping order and movement path.

[0039] In step S1, the visual data includes a two-dimensional color image acquired by a vertically positioned area array industrial camera and depth information acquired by a 3D vision sensor; the two-dimensional color image and the depth information are registered and fused to form point cloud data containing color and three-dimensional coordinates.

[0040] Step S2 includes: S21: Input the two-dimensional color image into the trained instance segmentation deep learning model to obtain the pixel-level contour mask and category information of each clipping; S22: Combine the contour mask and the point cloud data to extract the three-dimensional point cloud clusters corresponding to each piece; S23: Perform principal component analysis on each 3D point cloud cluster to calculate its 3D spatial position, surface normal vector and rotation angle in the horizontal plane, as the pose of the cut piece; S24: Based on the geometric features of the cut piece contour mask and the surface flatness of its corresponding three-dimensional point cloud cluster, a stable region with a flat surface and close to the two-dimensional centroid of the contour mask is selected inside the contour mask, and the three-dimensional center point of the stable region is calculated as the grasping point.

[0041] Step S3 includes: S31: Construct a mixed-integer programming model, where the decision variables represent the grasping order and the objective function is the sum of the estimated time costs for the robot to complete all grasping, moving, and placing actions; S32: The sum of the estimated time costs includes at least the robot's motion time and the grasping action adjustment time t. a The grasping action adjustment time t a The stability evaluation value S of the gripping point calculated based on the surface flatness k Negative correlation; the stability assessment value S k Calculate using the following formula: S k = 1 / (1 + σ z 2 ), where σ z 2 The height variance of the 3D point cloud cluster in the region where the capture point is located; S33: Solve the mixed integer programming model using a genetic algorithm or simulated annealing algorithm, and output the globally optimal grasping order and the corresponding key points of the robot motion path.

[0042] In step S4, the robot performs a grasping operation through an end effector, which includes a mounting base and at least three suction cup units; each suction cup unit is independently mounted on the mounting base, and each suction cup unit has an independent vertical drive mechanism and a vacuum adsorption channel.

[0043] Each of the suction cup units includes: A miniature electric actuator, the cylinder of which is fixed to the mounting base, constitutes the independent vertical drive mechanism; A flexible corrugated tube, the upper end of which is connected to the push rod end of the miniature electric push rod; A suction head is connected to the lower end of the flexible corrugated tube, and the inner cavity of the suction head is connected to the vacuum adsorption channel through the interior of the flexible corrugated tube. A force sensor is installed at the end of the miniature electric push rod or on the suction cup head to detect the contact force between the suction cup head and the surface of the cut piece. The miniature electric actuator can drive the suction head to extend and retract independently in the vertical direction, and the feedback signal from the force sensor is used to control the stop position of the miniature electric actuator.

[0044] During the grasping operation in step S4, the control method for the end effector includes: S41: Control each of the aforementioned miniature electric push rods to synchronously drive the suction cup head to move downwards; S42: When the contact force detected by the force sensor of any suction cup unit reaches the preset safety threshold, control the miniature electric push rod of that suction cup unit to stop. S43: After all suction heads have stopped, start vacuum suction; S44: After confirming that the adsorption is stable, control the robot to perform lifting and transfer actions.

[0045] Example 3: Based on Examples 1-2 above, the detailed steps and visual recognition algorithm principle of a visual recognition and optimized sorting method for carbon fiber composite material pieces are as follows: Step S1: Collect visual data of the stacked cut pieces: The computational control unit synchronously triggers the area array industrial camera and the 3D structured light projector to acquire a frame of synchronous data of the stacked cut pieces: Area array industrial camera: captures two-dimensional color images with a resolution of 2448×2048; 3D structured light projector: Simultaneously acquires depth information under the same field of view and generates a depth map; Using calibration parameters obtained beforehand through hand-eye and RGB-D calibration, including the intrinsic matrix of the area array industrial camera (focal length, principal point, etc.), the extrinsic matrix of the area array industrial camera (the transformation relationship between the area array industrial camera coordinate system and the world coordinate system), the registration matrix from the depth map to the 2D color image, and the transformation matrix from the robot coordinate system to the area array industrial camera coordinate system (hand-eye matrix), each pixel of the 2D color image is registered with the corresponding pixel of the depth map to generate RGB-D point cloud data. Each point in this RGB-D point cloud data contains 3D coordinates (X, Y, Z) and color information (R, G, B), and the coordinates of all points are unified in the world coordinate system.

[0046] Step S2: Based on the visual data, identify multiple independent cut pieces in the stacked state, and determine the pose of each cut piece in three-dimensional space and at least one gripping point: This step specifically includes the following sub-steps: S21: In this embodiment, the instance segmentation deep learning model used is the Mask R-CNN model. During the training phase, this model uses approximately 5000 two-dimensional color images labeled with carbon fiber rectangular clippings. Each clipping in each image is labeled with pixel-level contour masks and category information. The trained Mask R-CNN model can receive two-dimensional color images as input and output the following information for each detected clipping: Contour mask: A binary image where pixels belonging to the crop have a value of 1 and background pixels have a value of 0; Category information: For example, "carbon fiber rectangular cut piece - 0° layup", "carbon fiber rectangular cut piece - ±45° layup", etc. Confidence score: This indicates the confidence level of the Mask R-CNN model in the cropping recognition results, and is used for subsequent screening.

[0047] S22: For the contour mask of each piece output in step S21, combine it with the RGB-D point cloud data generated in step S1. Specifically, for each pixel in the contour mask with a value of 1 (i.e., belonging to that piece), based on the calibration parameters in step S1 (especially the transformation matrix mapping two-dimensional pixel coordinates to three-dimensional world coordinates), find the corresponding three-dimensional point in the RGB-D point cloud data for each pixel within the contour mask. The set of all these three-dimensional points constitutes the three-dimensional point cloud cluster corresponding to that piece. Thus, each piece corresponds to a three-dimensional point cloud cluster, denoted as P. i , where i is the cut piece number.

[0048] S23: For each 3D point cloud cluster P i Principal component analysis was performed, and the calculation process is as follows: 1. Calculate the 3D point cloud cluster P iThe covariance matrix; 2. Perform eigenvalue decomposition on the covariance matrix to obtain three eigenvalues ​​and their corresponding eigenvectors. Sort the eigenvalues ​​in ascending order, and denote the corresponding eigenvector as the smallest eigenvalue v. min intermediate eigenvalues ​​v mid The largest eigenvalue v max ; 3. Take the smallest eigenvalue v min N, the surface normal vector of the cut piece i This is because for a piece of fabric that is approximately planar, the direction corresponding to the minimum characteristic value is the thickness direction (normal). 4. The three-dimensional spatial position C of the cut piece i Take the three-dimensional point cloud cluster P i The geometric center of all points, i.e., the average coordinate: ,in, Representing the three-dimensional point cloud cluster P i The average of the coordinates of all points on the X, Y, and Z axes; 5. The rotation angle θ of the cut piece in the horizontal plane i Calculate the largest eigenvalue v in the following way: max Projecting onto the horizontal plane (i.e., the XOY plane of the world coordinate system) yields the projection vector v. proj Then calculate v proj The angle between the fabric piece and the positive X-axis of the world coordinate system (counterclockwise is positive) is the rotation angle θ of the fabric piece in the horizontal plane. i , representing the cut piece's position around the surface normal vector N in the horizontal plane. i The rotation angle.

[0049] Thus, the pose of piece i is obtained, represented by a 6-tuple: (C i.x C i.y C i.z , N i , θ i ), where C i.x C i.y C i.z , representing the X, Y, and Z coordinates of the center point of the cut piece, respectively; N i The unit normal vector representing the surface of the cut piece is a three-dimensional vector; θ i This represents the shape of the cut piece around the normal vector N in the horizontal plane. i The rotation angle.

[0050] S24: The “stable region” mentioned here refers to a region that simultaneously meets the following two conditions: first, the center of the region is close to the two-dimensional centroid of the contour mask; second, after the region is mapped to the three-dimensional point cloud, its height variance is less than the preset flatness threshold.

[0051] The specific steps are as follows: 1. Calculate the 2D centroid of the contour mask: Calculate the pixel coordinate centroid G of the contour mask. 2d (u,v), where u is the x-coordinate of a pixel and v is the y-coordinate of a pixel; 2. Define the candidate circular region: Let G be an example. 2d With the center as the center, and 1 / 4 of the length of the short side of the smallest bounding rectangle of the cut piece as the initial radius R, define a circular candidate region A within the contour mask; 3. Mapping to a 3D point cloud cluster: Using calibration parameters, the circular candidate region A is mapped to the 3D point cloud cluster P. i In the process, the set of three-dimensional points Q corresponding to the circular candidate region A is obtained. i Let Q i It contains n three-dimensional points; 4. Calculate surface flatness (height variance): Calculate Q i The variance σ of the height (i.e., Z coordinate) of all points z 2 The formula is: , in: n: represents the set of three-dimensional points Q i The total number of midpoints (positive integer). k: Represents the index number of the point, ranging from 1 to n (in the three-dimensional point set Q). i (Total number of midpoints) z k : Represents the Z-coordinate (height) value of the k-th point. : represents the average Z-coordinate of these n points, calculated using the following formula: , σ z 2 The calculated height variance is expressed in mm². A smaller height variance indicates a more concentrated height distribution of points and a smoother surface. 5. Check flatness: If the height variance is less than the preset flatness threshold (e.g., 0.25mm) 2 If the area is flat, then the area is considered flat; otherwise, gradually reduce the radius of the circular candidate area A (for example, by 10% each time), repeat steps 2-4, and recalculate the height variance until the height variance is less than the preset flatness threshold or the radius is less than the preset minimum radius threshold (for example, 10mm). 6. Determine the grab point: Select the area Q whose height variance is less than the preset flattening threshold. i The average of the three-dimensional point coordinates is used to obtain the grab point G. i (Three-dimensional coordinates): ,in, Q i The average of the X coordinates of all points. Q i The average Y-coordinate of all points. Q i The average Z-coordinate of all points.

[0052] Step S3: Based on the poses and gripping point information of all cut pieces, and with the optimization objective of minimizing the total robot sorting time, plan the robot's gripping sequence and motion path: S31: Constructing a mixed-integer programming model: This step constructs a mathematical model for the grabbing order planning problem. The core of this model is defining the decision variables and the objective function.

[0053] Decision variable x jk This is a binary variable (it can only take the value 0 or 1). x jk =1 indicates that after the robot finishes picking up piece j, it moves on to pick up the next piece k; x jk =0 indicates that the operation is not performed in this order. The model introduces virtual "starting point" and "ending point" to represent the robot's initial and final positions.

[0054] Objective function: The objective is to minimize the total time T for the robot to complete all tasks. total (i.e., minimizing the total time spent by the robot sorting). Total time T total It is the sum of the time of all robot actions, which can be expressed concisely by the following formula: , Among them, t jk Let x represent the sum of the estimated time costs of transitioning from state j to state k, and Σ represent the summation over all possible combinations of j and k. The goal is to find a set of x. jk The value of T is such that total Minimum.

[0055] In the mixed-integer programming model of this invention, "state" specifically refers to the step identifier of the robot in completing one full "grasp-place" operation cycle. It is not a physical quantity, but rather a task node number. Each state is associated with a specific spatial location and an action to be performed. Specifically, the set of states includes: Virtual starting point (state 0): Represents the initial standby position of the robot before it performs a task; There are N piece-grabbing states (state 1, state 2, ..., state N): each state is associated with a specific piece of fabric. When the system is in "state 2", it means that the robot has completed the placement of the previous piece of fabric and is preparing to move to the location of the next piece of fabric to perform the grabbing. Virtual endpoint (state N+1): Represents the final state after the robot has finished picking up and placing all the cut pieces.

[0056] S32: Define the sum of estimated time costs: The sum of estimated time costs t jk It is a key input to the mixed-integer programming model, determining the direction of optimization. Its calculation comprehensively considers: Robot motion time: Based on the robot's kinematics model, estimate the time required for it to move from one position to the next. Grasp motion settling time: This time is not a fixed value. It consists of a base time and a grasp motion settling time that is negatively correlated with grasp point stability. Grasp point stability evaluation value S k From region Q in step S24 k The height variance determines stability (the smaller the height variance, the higher the stability). Specifically, S k = 1 / (1 + σ z 2 ), where σ z 2 The height variance. Grasping action adjustment time t a = β * (1 - S k ), where β is a weighting coefficient, which can be calibrated experimentally based on experience or the robot's response characteristics, for example, with a value of 0.5 seconds. This guides the grasping order optimization algorithm to prioritize the selection of pieces with high grasping stability; Placement action time: a fixed value; By defining t in this way jk The crawling order optimization algorithm automatically balances the risk and efficiency of crawling operations while searching for the shortest path.

[0057] S33: Solved using a genetic algorithm: Because the optimization problem described above is computationally complex when the number of cut pieces is large, this embodiment employs a genetic algorithm, an efficient heuristic algorithm, to find an approximate optimal solution. The process simulates biological evolution: 1. Initialize the population: Randomly generate multiple possible grabbing orders, each grabbing order is represented by a number sequence (e.g., [2,1,4,3], [4,2,3,1]), each sequence is called a "chromosome"; 2. Fitness assessment: Calculate the sum of the estimated time costs T for each chromosome (grasping order). totalT total The shorter the chromosome, the higher its "fitness," indicating a better capture order. 3. Iterative evolution: Selection: Chromosomes with high fitness are preferred as "parents". Crossover: Exchanging partial sequences of two parent chromosomes to produce a new "offspring" sequence. Mutation: Randomly changing the position of a piece in a sequence with a small probability, introducing new possibilities; 4. Output Results: After repeating the selection, crossover, and mutation process hundreds of times, the best chromosome in the population is the globally optimal fetching sequence π. * This refers to the final grasping sequence of the robot. Subsequently, based on this globally optimal grasping sequence and the three-dimensional coordinates of each piece grasping point determined in step S24, combined with the robot's kinematic model, the smooth motion path of the robot from the starting point to each grasping point and then to the placement point, as well as the coordinates of intermediate critical path points, are interpolated and calculated for use in step S4.

[0058] Step S4: Control the robot to perform grasping and placement operations on each piece of fabric sequentially according to the planned grasping order and movement path: S41: The robot moves to the target piece grasping point G. i In the safe position above, adjust the attitude of the end effector so that the plane of the mounting base is aligned with the surface normal vector N of the cut piece calculated in step S23. i Parallel. Four miniature electric actuators extend synchronously, driving the suction head at a constant speed v. down (e.g., 10 mm / s) moving downwards; S42: The force sensor of each suction cup unit detects the contact force F between the suction cup head and the fabric piece in real time. When the contact force F detected by the force sensor of a certain suction cup unit is greater than or equal to a preset safety threshold (e.g., 2N), the miniature electric push rod of that suction cup unit stops immediately. Due to the uneven surface of the fabric piece, the time it takes for the contact force F of each suction cup head to reach the safety threshold varies.

[0059] S43: When all suction heads have stopped (or the maximum waiting time t has been reached) max =3 seconds later, open the independent solenoid valve to start vacuum adsorption. Monitor the vacuum pressure P, and when P≤P t (P) t The adsorption is determined to be stable when the preset safe vacuum pressure threshold (e.g., -60 kPa) is reached.

[0060] S44: After confirming stable adsorption, the robot moves at a speed v up (e.g., 20mm / s) Vertically lift to safe height H safe(e.g., 100mm), then move it along the planned path to the placement point, lower it and release the cut piece.

[0061] After each grab, return to step S1 to grab again until all cut pieces are sorted.

[0062] The visual recognition and optimized sorting system of the present invention has good scalability and can be quickly adapted to different composite materials and cut piece shapes in the following ways: 1. Visual model adaptation: For different layup directions or piece shapes (such as rectangular or irregular shapes) of carbon fiber composite materials, simply change the training dataset of the instance segmentation deep learning model in step S21, and retrain the instance segmentation deep learning model using the corresponding two-dimensional color images of the material and shape to achieve accurate identification and segmentation of new pieces. 2. Grasping parameter adaptation: Based on the physical properties of carbon fiber composite materials (such as surface friction coefficient, softness, and fragility), key parameters such as the flatness threshold in step S24, the weighting coefficient β in step S32, and the safety threshold in step S42 can be adjusted to optimize the grasping effect.

[0063] Furthermore, by changing the training dataset and adjusting key parameters, this system is also applicable to the sorting of other composite material pieces such as glass fiber and basalt fiber, demonstrating good versatility.

[0064] Example 4: Based on the above examples, taking a rectangular carbon fiber sheet as an example, the specific sorting process is as follows: Assumption: Approximately 50 rectangular carbon fiber pieces, each 300mm x 200mm, are randomly stacked on the workbench, with a stack height of approximately 150mm.

[0065] System operation: Visual recognition: The Mask R-CNN model accurately identified all 50 fabric pieces, with a recognition time of approximately 1.2 seconds. For one slightly occluded fabric piece, the positioning error was measured to be less than 1.5mm in center position and less than 2° in rotation angle.

[0066] Optimized planning: The genetic algorithm finds the optimized grasping order within 3 seconds. Compared to a simple "top-to-bottom, left-to-right" traversal sequence, the optimized grasping order reduces the robot's total sorting time by approximately 35%.

[0067] Robot execution: The suction cup units successfully picked up all the cut pieces. When picking up a slightly curved piece located at the bottom of the stack of cut pieces, the miniature motorized push rods of the two suction cup units located above the warped area extended by about 2mm more than the other two, ensuring that all four suction cup heads made close contact with the cut piece simultaneously. The entire sorting process took approximately 210 seconds, with no damaged or missed pieces.

[0068] Technical Results: This embodiment verifies the system's robust identification capability for stacked cut pieces, the significant efficiency improvement effect of the grasping order optimization algorithm, and the good compensation capability of the end effector for irregular surfaces, thus achieving efficient, reliable, and non-destructive automated sorting.

[0069] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for visual recognition and optimized sorting of carbon fiber composite material cut pieces, characterized in that, Includes the following steps: S1: Collect visual data of the stacked area of ​​the cut pieces; S2: Based on the visual data, identify and locate multiple independent cut pieces in the stacked state, and determine the pose of each cut piece in three-dimensional space and the grasping point information of at least one grasping point; S3: Based on the pose and gripping point information of all the cut pieces, with the optimization goal of minimizing the total time spent by the robot sorting, plan the robot's gripping sequence and motion path; S4: Control the robot to perform grasping and placement operations on each piece of fabric in sequence according to the planned grasping order and movement path.

2. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 1, characterized in that, In step S1, the visual data includes a two-dimensional color image acquired by a vertically positioned area array industrial camera and depth information acquired by a 3D vision sensor; the two-dimensional color image and the depth information are registered and fused to form point cloud data containing color and three-dimensional coordinates.

3. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 2, characterized in that, Step S2 includes: S21: Input the two-dimensional color image into the trained instance segmentation deep learning model to obtain the pixel-level contour mask and category information of each clipping; S22: Combine the contour mask and the point cloud data to extract the three-dimensional point cloud clusters corresponding to each piece; S23: Perform principal component analysis on each 3D point cloud cluster to calculate its 3D spatial position, surface normal vector and rotation angle in the horizontal plane, as the pose of the cut piece; S24: Based on the geometric features of the cut piece contour mask and the surface flatness of its corresponding three-dimensional point cloud cluster, a stable region with a flat surface and close to the two-dimensional centroid of the contour mask is selected inside the contour mask, and the three-dimensional center point of the stable region is calculated as the grasping point.

4. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 3, characterized in that, Step S3 includes: S31: Construct a mixed-integer programming model, where the decision variables represent the grasping order and the objective function is the sum of the estimated time costs for the robot to complete all grasping, moving, and placing actions; S32: The sum of the estimated time costs includes at least the robot's motion time and the grasping action adjustment time t. a The grasping action adjustment time t a The stability evaluation value S of the gripping point calculated based on the surface flatness k Negative correlation; S33: Solve the mixed integer programming model using a genetic algorithm or simulated annealing algorithm, and output the globally optimal grasping order and the corresponding key points of the robot motion path.

5. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 4, characterized in that, The stability assessment value S k Calculate using the following formula: S k = 1 / (1 + σ z 2 ), where σ z 2 The height variance of the 3D point cloud cluster in the region where the capture point is located.

6. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 1, characterized in that, In step S4, the robot performs a grasping operation through an end effector, which includes a mounting base and at least three suction cup units; each suction cup unit is independently mounted on the mounting base, and each suction cup unit has an independent vertical drive mechanism and a vacuum adsorption channel.

7. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 6, characterized in that, Each of the suction cup units includes: A miniature electric actuator, the cylinder of which is fixed to the mounting base, constitutes the independent vertical drive mechanism; A flexible corrugated tube, the upper end of which is connected to the push rod end of the miniature electric push rod; A suction head is connected to the lower end of the flexible corrugated tube, and the inner cavity of the suction head is connected to the vacuum adsorption channel through the interior of the flexible corrugated tube. A force sensor is installed at the end of the miniature electric push rod or on the suction cup head to detect the contact force between the suction cup head and the surface of the cut piece. The miniature electric actuator can drive the suction head to extend and retract independently in the vertical direction, and the feedback signal from the force sensor is used to control the stop position of the miniature electric actuator.

8. The method for visual recognition and optimized sorting of carbon fiber composite material cut pieces according to claim 7, characterized in that, During the grasping operation in step S4, the control method for the end effector includes: S41: Control each of the aforementioned miniature electric push rods to synchronously drive the suction cup head to move downwards; S42: When the contact force detected by the force sensor of any suction cup unit reaches the preset safety threshold, control the miniature electric push rod of that suction cup unit to stop. S43: After all suction heads have stopped, start vacuum suction; S44: After confirming that the adsorption is stable, control the robot to perform lifting and transfer actions.

9. A visual recognition and optimized sorting system for carbon fiber composite material cut pieces, implementing the visual recognition and optimized sorting method for carbon fiber composite material cut pieces as described in any one of claims 1-8, characterized in that, include: A visual acquisition module is fixed above the fabric piece stacking area to acquire visual data of the stacked fabric pieces in the fabric piece stacking area; The computational control unit is used to run visual recognition algorithms, grasping sequence optimization algorithms, and generate robot control commands; A robot execution module includes a robot and the end effector mounted at its end; The computing control unit is communicatively connected to both the vision acquisition module and the robot execution module.

10. The visual recognition and optimized sorting system for carbon fiber composite material cut pieces according to claim 9, characterized in that, The vision acquisition module includes a rigid bracket and a high-resolution area array industrial camera and a 3D vision sensor fixed to the rigid bracket by an adjustable mounting mechanism. The adjustable mounting mechanism can adjust and lock the posture of the area array industrial camera and the 3D vision sensor to ensure that the centers of their fields of view coincide and the optical axis parallelism error is less than 0.1°. The 3D vision sensor is a 3D structured light projector or a binocular camera.

Images