Multimodal sensing AI-adaptive adsorption and grasping control system and method for abalone

Abstract

This invention discloses a multi-modal sensing AI-adaptive adsorption and grasping control system and method for abalone, relating to the field of AI control. The system includes: acquiring point cloud data of the production line in real time using a stereo vision sensor to construct a 3D scene map; identifying individual abalone and acquiring their positional contour point cloud data; constructing a 3D model of the abalone; building an adsorption area selection model using a convolutional neural network model to obtain the optimal adsorption contact area; generating an approach path and optimal adsorption pose through inverse kinematics calculations; making adsorption and grasping decisions using pressure sensors and stereo vision sensors and triggering grasping commands; and outputting the stability of the current adsorption state and the probability of slippage risk in real time using a slip prediction model. The advantages of this invention are: by integrating stereo vision, laser scanning, and deep learning technologies, it achieves high-precision, intelligent abalone grasping control, real-time target identification, and optimized adsorption area, improving grasping efficiency and safety.

Description

Technical Field

[0001] This invention relates to the field of AI control, and in particular to a multimodal sensing AI adaptive adsorption and grasping control system and method for abalone. Background Technology

[0002] In the abalone deep processing production line, the abalone meat after blanching has a slippery and irregular surface due to protein denaturation and moisture loss, and the individual sizes and shapes vary significantly. This poses a great challenge to the subsequent handling and transfer to processes such as shelling and viscera removal. In traditional handling methods, rigid mechanical clamps are prone to crushing the fragile abalone meat due to excessive clamping force, or the shape may not be properly clamped and the meat may fall off. Vacuum adsorption devices with fixed parameters cannot adapt to the dynamic changes in the humidity and roughness of the abalone meat surface after blanching, often resulting in insufficient or excessive adsorption force, which seriously affects production efficiency and product integrity.

[0003] While current multimodal sensing abalone AI adaptive adsorption and grasping control systems and methods on the market incorporate some sensors, they still have significant limitations: First, they lack the data fusion capability of stereo vision and high-precision laser scanning, making it difficult to construct a high-precision 3D model of the abalone shell, resulting in large errors in shape recognition and pose localization; Second, the selection of adsorption areas relies heavily on preset rules or simple algorithms, without incorporating deep learning optimization based on historical grasping data, making it unable to adapt to the diversity of individual abalone, and the grasping success rate is limited by fixed strategies; Third, grasping decisions are based solely on single pressure or visual parameters, without integrating real-time pressure sensors and stereo vision for comprehensive analysis, lacking dynamic monitoring of the adsorption state; Fourth, the absence of a slippage prediction model makes it impossible to assess adsorption stability and slippage risk in real time, easily leading to grasping failure or abalone damage due to unstable adsorption. Summary of the Invention

[0004] To improve existing systems and methods, a multi-modal sensing AI adaptive adsorption and grasping control system and method for abalone is provided. This method integrates technologies such as stereo vision, laser scanning and deep learning to achieve high-precision and intelligent abalone grasping control. It can identify targets in real time, optimize the adsorption area and dynamically evaluate grasping stability, thereby improving grasping efficiency and safety.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multimodal sensing AI-based adaptive adsorption and grasping control method for abalone includes:

[0007] Real-time point cloud data of the production line is acquired through stereo vision sensors to construct a 3D scene map. A segmentation model based on deep learning is used to identify individual abalone within the work area and obtain the location contour point cloud data of the abalone.

[0008] The abalone's location contour point cloud data is fused with the high-precision abalone morphology data obtained by a laser scanner to construct a high-precision three-dimensional model of the abalone shell.

[0009] Based on historical data, a convolutional neural network model is trained to build an adsorption region screening model. Combined with a high-precision three-dimensional model of abalone shell, the optimal adsorption contact area is obtained.

[0010] Based on the optimal adsorption contact area, the approach path and optimal adsorption posture of the adsorption device are generated through inverse kinematics calculation. The device moves to the vicinity of the shell at a preset safe approach speed and aligns each adsorption unit of the flexible adsorption head with the optimal adsorption area.

[0011] Before adsorption and grasping, the adsorption and grasping decision is made by pressure sensor and stereo vision sensor. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the effective adsorption expectation, the grasping command is triggered.

[0012] After adsorption is established, the sensor data stream is continuously collected and fused at high speed. Through a slip prediction model pre-trained based on historical data, the stability of the current adsorption state and the probability of slip risk are output in real time until the gripping ends and the negative pressure is safely released.

[0013] Preferably, the step of acquiring point cloud data of the production line in real time through a stereo vision sensor, constructing a 3D scene map, identifying individual abalone within the work area based on a deep learning segmentation model, and acquiring the location contour point cloud data of the abalone specifically includes:

[0014] The system uses binocular cameras to capture left and right eye image pairs on the production line in real time, collects image data, and records the pose information of the sensors.

[0015] Stereo matching is performed on the acquired left and right eye images to calculate the disparity map. Based on the disparity map and calibration parameters, the depth information of each pixel is calculated to generate three-dimensional point cloud data.

[0016] By combining real-time generated point cloud data with sensor pose information, and using synchronous positioning and mapping technology, a three-dimensional scene map of the production line is constructed.

[0017] A deep learning segmentation model is trained on a labeled abalone point cloud dataset. Real-time generated point cloud data is input into the deep learning segmentation model. The model performs semantic segmentation on the point cloud, identifies the subset of point cloud belonging to individual abalone, and removes background point cloud.

[0018] For each identified abalone point cloud subset, its spatial position is calculated as the center position of the abalone, and the boundary contour of the abalone point cloud is extracted.

[0019] Preferably, the step of fusing the abalone's positional contour point cloud data with the high-precision abalone morphology data acquired by a laser scanner to construct a high-precision three-dimensional model of the abalone shell specifically includes:

[0020] Based on the acquired abalone location data, a high-precision morphological scan of the abalone is performed using a laser scanner to obtain detailed three-dimensional shape data;

[0021] Using relative pose data from a global positioning system and sensors, the abalone location contour point cloud is spatially aligned with the data source of the laser scan point cloud;

[0022] High-precision laser scanning data and position contour point cloud data are fused using a multi-view fusion method based on point cloud matching.

[0023] The fused point cloud data is synthesized and homogenized, and the synthesized point cloud data is converted into a triangular mesh model. The surface of the generated triangular mesh is smoothed to construct a high-precision three-dimensional model of the abalone shell.

[0024] Preferably, the step of constructing an adsorption region screening model based on historical data through training a convolutional neural network model, and combining it with a high-precision three-dimensional model of the abalone shell to obtain the optimal adsorption contact region specifically includes:

[0025] Based on the historical data of abalone grabbing in different environments, the possible adsorption areas of abalone are marked. A model for adsorption area screening is constructed by training the historical data through a convolutional neural network based on point cloud data processing.

[0026] The three-dimensional model of abalone is processed by a trained adsorption region screening model to obtain possible adsorption regions and add scores.

[0027] Based on all possible adsorption regions, the region with the highest score is selected as the optimal adsorption contact region.

[0028] Preferably, the step of generating the approach path and optimal adsorption posture of the adsorption device based on the optimal adsorption contact area through inverse kinematics calculation, moving to the vicinity of the housing at a preset safe approach speed, and aligning each adsorption unit of the flexible adsorption head with the optimal adsorption area specifically includes:

[0029] Based on the structure of the adsorption device, a kinematic model is constructed to obtain the range of motion, speed limits, and geometric constraints of each joint.

[0030] Calculate the centroid coordinates and surface normal vector of the optimal adsorption contact region, and define the target pose of the adsorption device;

[0031] The inverse kinematics algorithm is used to calculate the joint angles and displacements required for the adsorption device to reach the target pose from the current position, and to generate a continuous motion path from the initial position to the target pose.

[0032] Based on the surface normal vector and geometric features of the optimal adsorption region, the orientation of the adsorption device is finely adjusted to align the adsorption unit of the flexible adsorption head with the optimal adsorption region of the abalone shell.

[0033] Preferably, the adsorption and grasping decision is made by a pressure sensor and a stereo vision sensor before adsorption and grasping. When the pressure distribution reaches a preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, the grasping command is triggered, specifically including:

[0034] The micro-pressure changes at the bottom of each adsorption unit are monitored in real time by a high-frequency micro pressure sensor at the end of the adsorption device.

[0035] Microscopic deformation images of the contact area between the adsorption head and the shell are obtained using a stereo vision sensor;

[0036] Decision-making is based on micro-pressure changes and micro-deformation images. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, it is determined that the grasping conditions have been met and the grasping command is triggered.

[0037] Preferably, after adsorption is established, the sensor data stream is continuously and rapidly acquired and fused. A slip prediction model pre-trained based on historical data is used to output the stability of the current adsorption state and the probability of slip risk in real time, until the gripping process ends and the negative pressure is safely released. Specifically, this includes:

[0038] After adsorption is established, sensor data streams are continuously collected and the data is integrated into a unified feature vector.

[0039] The slip prediction model, which is pre-trained based on historical capture data, inputs the feature vector of the fused sensor data into the slip prediction model and outputs the stability of the previous adsorption state and the probability of slippage of the adsorption device in the current state in a probabilistic form.

[0040] Thresholds for adsorption stability and slippage risk are set based on historical data. Throughout the grasping process, sensor data is continuously collected and the adsorption status is monitored through a slippage prediction model.

[0041] At the end of the grasping process, check the sensor data and the output of the slip prediction model, check the status of the adsorption device, and reduce the negative pressure of the flexible adsorption head to separate the adsorption unit from the abalone shell.

[0042] Furthermore, a multimodal sensing AI-based adaptive adsorption and grasping control system for abalone is proposed, including:

[0043] Stereo vision sensing module: The module acquires point cloud data of the production line in real time through a stereo vision sensor and constructs a three-dimensional scene map;

[0044] Abalone identification module: The module processes point cloud data based on a deep learning segmentation model to identify individual abalone and extract location contour point clouds;

[0045] 3D model construction module: This module integrates abalone location contour point cloud data and laser scanning data to construct a high-precision 3D model of the abalone shell;

[0046] Adsorption region screening module: This module obtains the optimal adsorption contact region by training a model through a convolutional neural network based on historical data.

[0047] Approach path and pose module: This module generates the approach path and optimal adsorption pose of the adsorption device through inverse kinematics calculations;

[0048] Grasping decision-making module: The module judges the adsorption stability through pressure sensor and stereo vision sensor, and triggers a grasping command when the grasping conditions are met;

[0049] Slip Prediction and Stability Assessment Module: This module uses a slip prediction model pre-trained with historical data to output the stability of the adsorption state and the probability of slip risk in real time.

[0050] Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] By using stereo vision sensors and deep learning models to identify and precisely locate individual abalone on the production line in real time, a 3D scene map can be quickly constructed, ensuring the accuracy and efficiency of the grasping operation. After fusion with laser scanning data, a high-precision 3D model of the abalone shell is formed, providing a solid foundation for subsequent adsorption area selection and adsorption device path planning. The adsorption area selection model is optimized and trained using historical grasping data to ensure more accurate selection of adsorption areas, thereby improving the grasping success rate. By monitoring micro-pressure changes and deformation characteristics during the adsorption process in real time, the system intelligently determines the grasping timing, avoiding mis-grabbing or damage to the target. Furthermore, the slip prediction model can dynamically assess adsorption stability and adjust operating strategies in real time to minimize risks and ensure the safety and reliability of the grasping process. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method proposed in this invention;

[0054] Figure 2This is a schematic diagram illustrating the point cloud data for obtaining the location contour of an abalone, as proposed in this invention.

[0055] Figure 3 This is a schematic diagram of the high-precision three-dimensional model of the abalone shell proposed in this invention;

[0056] Figure 4 This is a schematic diagram illustrating the optimal adsorption contact area proposed in this invention.

[0057] Figure 5 This is a schematic diagram of the alignment of the flexible adsorption head proposed in this invention;

[0058] Figure 6 This is a schematic diagram of the triggering and grabbing command proposed in this invention;

[0059] Figure 7 This is a schematic diagram of the slip prediction model proposed in this invention. Detailed Implementation

[0060] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0061] A multimodal sensing AI-driven adaptive adsorption and grasping control system for abalone includes:

[0062] Stereo vision sensing module: The module acquires point cloud data of the production line in real time through a stereo vision sensor and constructs a three-dimensional scene map;

[0063] Abalone identification module: The module processes point cloud data based on a deep learning segmentation model to identify individual abalone and extract location contour point clouds;

[0064] 3D model construction module: This module integrates abalone location contour point cloud data and laser scanning data to construct a high-precision 3D model of the abalone shell;

[0065] Adsorption region screening module: This module obtains the optimal adsorption contact region by training a model through a convolutional neural network based on historical data.

[0066] Approach path and pose module: This module generates the approach path and optimal adsorption pose of the adsorption device through inverse kinematics calculations;

[0067] Grasping decision-making module: The module judges the adsorption stability through pressure sensor and stereo vision sensor, and triggers a grasping command when the grasping conditions are met;

[0068] Slip Prediction and Stability Assessment Module: This module uses a slip prediction model pre-trained with historical data to output the stability of the adsorption state and the probability of slip risk in real time.

[0069] Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0070] See Figure 1 As shown, the multimodal sensing AI-based adaptive adsorption and grasping control method for abalone includes:

[0071] Step 1: Real-time acquisition of point cloud data of the production line through stereo vision sensors to construct a 3D scene map; identification of individual abalone within the work area based on a deep learning segmentation model; acquisition of point cloud data of the abalone's location contours.

[0072] Step 2: The point cloud data of the abalone's location contour is fused with the high-precision abalone morphology data obtained by the laser scanner to construct a high-precision 3D model of the abalone shell;

[0073] Step 3: Based on historical data, a convolutional neural network model is trained to build an adsorption region screening model. Combined with a high-precision 3D model of abalone shell, the optimal adsorption contact area is obtained.

[0074] Step 4: Based on the optimal adsorption contact area, the approach path and optimal adsorption posture of the adsorption device are generated through inverse kinematics calculation. The device moves to the vicinity of the shell at a preset safe approach speed and aligns each adsorption unit of the flexible adsorption head with the optimal adsorption area.

[0075] Step 5: Before adsorption and grasping, the pressure sensor and stereo vision sensor make adsorption and grasping decisions. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, the grasping command is triggered.

[0076] Step Six: After adsorption is established, continuously collect and fuse sensor data streams at high speed. Using a slip prediction model pre-trained based on historical data, output the stability of the current adsorption state and the probability of slip risk in real time until the gripping ends and the negative pressure is safely released.

[0077] See Figure 2 As shown, point cloud data of the production line is acquired in real time using a stereo vision sensor to construct a 3D scene map. A segmentation model based on deep learning is used to identify individual abalone within the work area. The acquisition of the abalone's location contour point cloud data specifically includes:

[0078] The system uses binocular cameras to capture left and right eye image pairs on the production line in real time, collects image data, and records the pose information of the sensors.

[0079] Stereo matching is performed on the acquired left and right eye images to calculate the disparity map. Based on the disparity map and calibration parameters, the depth information of each pixel is calculated to generate three-dimensional point cloud data.

[0080] By combining real-time generated point cloud data with sensor pose information, and using synchronous positioning and mapping technology, a three-dimensional scene map of the production line is constructed.

[0081] A deep learning segmentation model is trained on a labeled abalone point cloud dataset. Real-time generated point cloud data is input into the deep learning segmentation model. The model performs semantic segmentation on the point cloud, identifies the subset of point cloud belonging to individual abalone, and removes background point cloud.

[0082] For each identified abalone point cloud subset, its spatial position is calculated as the center position of the abalone, and the boundary contour of the abalone point cloud is extracted.

[0083] Specifically, binocular cameras are used to capture left and right eye images of the production line in real time. and , The image pixel coordinates are used to obtain the pose information of the stereo camera in the world coordinate system through the inertial measurement unit;

[0084] Images of the left and right eyes and Perform stereo matching and calculate disparity map ,in , representing the pixels in the left image Pixels corresponding to the image on the right Horizontal displacement;

[0085] Based on the principle of binocular vision, the depth of each pixel is calculated based on the disparity map, camera focal length, and baseline distance. For each pixel, its three-dimensional coordinates in the camera coordinate system are calculated by combining the camera intrinsic parameters. The three-dimensional coordinates of all pixels are combined to generate point cloud data.

[0086] The point cloud in the camera coordinate system is transformed to the world coordinate system to obtain the point cloud in the world coordinate system. SLAM technology is used to combine binocular images and sensor pose information to optimize the camera pose and map point cloud in real time and build a 3D scene map of the production line.

[0087] A point cloud semantic segmentation model is trained on a labeled abalone point cloud dataset. The real-time generated point cloud is input into the trained segmentation model, which predicts the label of each point and generates a segmented point cloud subset. For each abalone point cloud subset, its geometric center is calculated as the spatial location of the abalone. For each abalone point cloud subset, the point cloud is projected onto the principal plane. In the projection plane, the boundary point set is calculated using the convex hull algorithm, and the boundary contour is represented as an ordered point set.

[0088] See Figure 3 As shown, the high-precision 3D model of the abalone shell is constructed by fusing the abalone's positional contour point cloud data with high-precision abalone morphology data acquired by a laser scanner. Specifically, this includes:

[0089] Based on the acquired abalone location data, a high-precision morphological scan of the abalone is performed using a laser scanner to obtain detailed three-dimensional shape data;

[0090] Using relative pose data from a global positioning system and sensors, the abalone location contour point cloud is spatially aligned with the data source of the laser scan point cloud;

[0091] High-precision laser scanning data and position contour point cloud data are fused using a multi-view fusion method based on point cloud matching.

[0092] The fused point cloud data is synthesized and homogenized, and the synthesized point cloud data is converted into a triangular mesh model. The surface of the generated triangular mesh is smoothed to construct a high-precision three-dimensional model of the abalone shell.

[0093] Specifically, a laser scanner is used to perform high-precision morphological scanning of abalone to obtain detailed three-dimensional shape data. A global positioning system is used to obtain global coordinate references for the abalone's position contour point cloud. Based on the pose of the laser scanner and the pose of the binocular camera, the laser point cloud is transformed into the world coordinate system. Iterative nearest point calculation is used for fine registration to solve for the optimal transformation matrix.

[0094] The aligned laser point cloud and abalone outline point cloud are fused to generate a unified point cloud. For points in overlapping areas, the weighted average coordinates are calculated, and for non-overlapping areas, their respective point cloud data are directly retained.

[0095] The fused point cloud is converted into a triangular mesh model by Poisson reconstruction. By solving the implicit surface function of the point cloud, a smooth triangular mesh is generated. The surface of the triangular mesh is smoothed. Through several iterations, the surface is ensured to be smooth but the geometric features of the abalone shell are preserved.

[0096] See Figure 4 As shown, based on historical data, a convolutional neural network model is trained to construct an adsorption region selection model. Combined with a high-precision 3D model of abalone shells, the optimal adsorption contact region is obtained, specifically including:

[0097] Based on the historical data of abalone grabbing in different environments, the possible adsorption areas of abalone are marked. A model for adsorption area screening is constructed by training the historical data through a convolutional neural network based on point cloud data processing.

[0098] The three-dimensional model of abalone is processed by a trained adsorption region screening model to obtain possible adsorption regions and add scores.

[0099] Based on all possible adsorption regions, the region with the highest score is selected as the optimal adsorption contact region.

[0100] Specifically, a point cloud processing network is used to build an adsorption region screening model. The input is point cloud data and its features, and the output is the adsorption probability of each point, which represents the probability that the point belongs to the adsorption region. The trained model can receive point cloud input, output the adsorption probability of each point, and generate a subset of point cloud adsorption regions.

[0101] Based on the probability threshold, a subset of point clouds of possible adsorption regions is extracted. For each adsorption region, its geometric features are calculated, and a comprehensive score is calculated for each adsorption region. Taking into account the adsorption probability, geometric features, and grasping stability, the scores of all adsorption regions are compared, and the region with the highest score is selected as the optimal adsorption region. Its center position is used as the contact point of the grasping device.

[0102] See Figure 5 As shown, based on the optimal adsorption contact area, the approach path and optimal adsorption posture of the adsorption device are generated through inverse kinematics calculations. The device moves to the vicinity of the housing at a preset safe approach speed, and aligns each adsorption unit of the flexible adsorption head with the optimal adsorption area. Specifically, this includes:

[0103] Based on the structure of the adsorption device, a kinematic model is constructed to obtain the range of motion, speed limits, and geometric constraints of each joint.

[0104] Calculate the centroid coordinates and surface normal vector of the optimal adsorption contact region, and define the target pose of the adsorption device;

[0105] The inverse kinematics algorithm is used to calculate the joint angles and displacements required for the adsorption device to reach the target pose from the current position, and to generate a continuous motion path from the initial position to the target pose.

[0106] Based on the surface normal vector and geometric features of the optimal adsorption region, the orientation of the adsorption device is finely adjusted to align the adsorption unit of the flexible adsorption head with the optimal adsorption region of the abalone shell.

[0107] Specifically, assuming the adsorption device is a multi-degree-of-freedom robotic arm with n joints, the joint parameters are expressed as follows: ,in For the angle or displacement of the i-th joint, construct a positive kinematic model to describe the mapping from joint space to the pose of the adsorption head, as shown in the formula:

[0108]

[0109] in, This represents the pose of the adsorption head in the world coordinate system. The mathematical representation of positive kinematics. Let be a rotation matrix. It is a translation vector. Let be the transformation matrix of the i-th joint;

[0110] Motion constraints include the range of motion of each joint: for example, the angular range of a rotary joint or the displacement range of a translational joint; velocity limits: the maximum angular velocity or linear velocity of each joint; and geometric constraints: the maximum angular velocity or linear velocity of each joint.

[0111] The centroid of the optimal adsorption region is calculated as the target contact point. Principal component analysis is performed on the subset of point cloud to extract the principal plane normal vector of the region. Inverse kinematics calculation is performed by minimizing the pose error to generate a continuous motion path.

[0112] See Figure 6 As shown, before adsorption and grasping, a pressure sensor and a stereo vision sensor make adsorption and grasping decisions. When the pressure distribution reaches a preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, the grasping command is triggered, specifically including:

[0113] The micro-pressure changes at the bottom of each adsorption unit are monitored in real time by a high-frequency micro pressure sensor at the end of the adsorption device.

[0114] Microscopic deformation images of the contact area between the adsorption head and the shell are obtained using a stereo vision sensor;

[0115] Decision-making is based on micro-pressure changes and micro-deformation images. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, it is determined that the grasping conditions have been met and the grasping command is triggered.

[0116] Specifically, a high-frequency micro pressure sensor array is arranged on the flexible adsorption head at the end of the adsorption device. Each sensor collects the micro pressure value at the bottom of the adsorption unit in real time. Based on the pressure data of all sensors, a global pressure distribution vector is constructed, and the mean and spatial consistency of the global pressure distribution are calculated.

[0117] The image pairs of the contact area between the adsorption head and the abalone shell are captured in real time using a stereo vision sensor to generate a high-resolution three-dimensional deformation point cloud. The three-dimensional shape of the contact area is reconstructed for each pair of images through a stereo matching algorithm. The deformation displacement of the contact area is calculated. Based on the point cloud at the initial contact time, the deformation amplitude distribution is calculated, the global deformation features are statistically analyzed, and the local features of the deformation point cloud are extracted into feature vectors.

[0118] Set a stable threshold for pressure distribution and a threshold for deformation characteristics. Combine the pressure distribution and deformation characteristics to define a scoring function for gripping conditions. If the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, it is determined that the gripping conditions have been met, triggering a gripping command, activating the adsorption head to apply negative pressure, and completing the abalone gripping.

[0119] See Figure 7As shown, after adsorption is established, the sensor data stream is continuously and rapidly collected and fused. A slip prediction model pre-trained based on historical data is used to output the stability of the current adsorption state and the probability of slip risk in real time, until the gripping process ends and the negative pressure is safely released. Specifically, this includes:

[0120] After adsorption is established, sensor data streams are continuously collected and the data is integrated into a unified feature vector.

[0121] The slip prediction model, which is pre-trained based on historical capture data, inputs the feature vector of the fused sensor data into the slip prediction model and outputs the stability of the previous adsorption state and the probability of slippage of the adsorption device in the current state in a probabilistic form.

[0122] Thresholds for adsorption stability and slippage risk are set based on historical data. Throughout the grasping process, sensor data is continuously collected and the adsorption status is monitored through a slippage prediction model.

[0123] At the end of the grasping process, check the sensor data and the output of the slip prediction model, check the status of the adsorption device, and reduce the negative pressure of the flexible adsorption head to separate the adsorption unit from the abalone shell.

[0124] Specifically, a high-frequency micro pressure sensor array is used to continuously collect micro pressure data at the bottom of the adsorption unit, a stereo vision sensor is used to collect three-dimensional deformation point cloud of the contact area, and pressure data streams with timestamps aligned are collected simultaneously.

[0125] Features are extracted from pressure data, including average pressure, standard deviation of pressure distribution, and rate of change of pressure; features are extracted from deformation data, including average deformation, average deformation, and rate of change of deformation; and the features are integrated into a unified feature vector.

[0126] Adsorption stability and slip risk are evaluated based on a pre-trained slip prediction model. Real-time feature vectors are input into the slip prediction model, and the output probability is a high stability probability, which indicates that the adsorption state is stable and the adsorption head is in good contact with the abalone shell.

[0127] Based on historical grasping data, the feature distribution of successful grasping and slip failure is statistically analyzed, and adsorption stability and slip risk thresholds are set. During the grasping process, sensor data is continuously collected to generate feature vectors, which are then input into the slip prediction model to calculate stability and slip probability in real time, and to make slip probability judgments.

[0128] At the end of the grasping process, check the sensor data at the final moment to verify the negative pressure system status of the adsorption device, ensure that the negative pressure value is stable within the target range, gradually reduce the negative pressure of the flexible adsorption head, record the sensor data at the end of the grasping process, and store the data in the historical database for updating the slip prediction model or optimizing the threshold.

[0129] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0130] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal sensing AI-based adaptive adsorption and grasping control method for abalone, characterized in that, include: Real-time point cloud data of the production line is acquired through stereo vision sensors to construct a 3D scene map. A segmentation model based on deep learning is used to identify individual abalone within the work area and obtain the location contour point cloud data of the abalone. The abalone's location contour point cloud data is fused with the high-precision abalone morphology data obtained by a laser scanner to construct a high-precision three-dimensional model of the abalone shell. Based on historical data, a convolutional neural network model is trained to build an adsorption region screening model. Combined with a high-precision three-dimensional model of abalone shell, the optimal adsorption contact area is obtained. Based on the optimal adsorption contact area, the approach path and optimal adsorption posture of the adsorption device are generated through inverse kinematics calculation. The device moves to the vicinity of the shell at a preset safe approach speed and aligns each adsorption unit of the flexible adsorption head with the optimal adsorption area. Before adsorption and grasping, the adsorption and grasping decision is made by pressure sensor and stereo vision sensor. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the effective adsorption expectation, the grasping command is triggered. After adsorption is established, the sensor data stream is continuously and rapidly collected and fused. A slip prediction model based on historical data is used to output the stability of the current adsorption state and the probability of slip risk in real time until the gripping ends and the negative pressure is safely released. Specifically, this includes: using a high-frequency micro pressure sensor array to continuously collect micro-pressure data at the bottom of the adsorption unit, using a stereo vision sensor to collect the three-dimensional deformation point cloud of the contact area, and simultaneously collecting pressure data streams aligned with timestamps; extracting features from the pressure data and deformation data and integrating them into a unified feature vector, which is then input into the slip prediction model to output the stability of the current adsorption state and the probability of slip risk.

2. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, The process of acquiring point cloud data of the production line in real time through a stereo vision sensor, constructing a 3D scene map, identifying individual abalone within the work area based on a deep learning segmentation model, and obtaining the location contour point cloud data of the abalone specifically includes: The system uses binocular cameras to capture left and right eye image pairs on the production line in real time, collects image data, and records the pose information of the sensors. Stereo matching is performed on the acquired left and right eye images to calculate the disparity map. Based on the disparity map and calibration parameters, the depth information of each pixel is calculated to generate three-dimensional point cloud data. By combining real-time generated point cloud data with sensor pose information, and using synchronous positioning and mapping technology, a three-dimensional scene map of the production line is constructed. A deep learning segmentation model is trained on a labeled abalone point cloud dataset. Real-time generated point cloud data is input into the deep learning segmentation model. The segmentation model performs semantic segmentation on the point cloud, identifies the subset of point cloud belonging to individual abalone, and removes background point cloud. For each identified abalone point cloud subset, its spatial position is calculated as the center position of the abalone, and the boundary contour of the abalone point cloud is extracted.

3. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, The step of fusing the abalone's positional contour point cloud data with the high-precision abalone morphology data acquired by a laser scanner to construct a high-precision three-dimensional model of the abalone shell specifically includes: Based on the acquired abalone location data, a high-precision morphological scan of the abalone is performed using a laser scanner to obtain detailed three-dimensional shape data; Using relative pose data from a global positioning system and sensors, the abalone location contour point cloud is spatially aligned with the data source of the laser scan point cloud; High-precision laser scanning data and position contour point cloud data are fused using a multi-view fusion method based on point cloud matching. The fused point cloud data is synthesized and homogenized, and the synthesized point cloud data is converted into a triangular mesh model. The surface of the generated triangular mesh is smoothed to construct a high-precision three-dimensional model of the abalone shell.

4. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, The process of constructing an adsorption region screening model based on historical data, trained using a convolutional neural network model, and combining this model with a high-precision 3D model of the abalone shell to obtain the optimal adsorption contact area specifically includes: Based on the historical data of abalone grabbing in different environments, the possible adsorption areas of abalone are marked. A model for adsorption area screening is constructed by training the historical data through a convolutional neural network based on point cloud data processing. The three-dimensional model of abalone is processed by a trained adsorption region screening model to obtain possible adsorption regions and add scores. Based on all possible adsorption regions, the region with the highest score is selected as the optimal adsorption contact region.

5. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, The process of generating the approach path and optimal adsorption posture of the adsorption device based on the optimal adsorption contact area through inverse kinematics calculation, moving it to the vicinity of the housing at a preset safe approach speed, and aligning each adsorption unit of the flexible adsorption head with the optimal adsorption area specifically includes: Based on the structure of the adsorption device, a kinematic model is constructed to obtain the range of motion, speed limits, and geometric constraints of each joint. Calculate the centroid coordinates and surface normal vector of the optimal adsorption contact region, and define the target pose of the adsorption device; The inverse kinematics algorithm is used to calculate the joint angles and displacements required for the adsorption device to reach the target pose from the current position, and to generate a continuous motion path from the initial position to the target pose. Based on the surface normal vector and geometric features of the optimal adsorption region, the orientation of the adsorption device is finely adjusted to align the adsorption unit of the flexible adsorption head with the optimal adsorption region of the abalone shell.

6. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, Before adsorption and grasping, a pressure sensor and a stereo vision sensor are used to make adsorption and grasping decisions. When the pressure distribution reaches a preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, the grasping command is triggered, specifically including: The micro-pressure changes at the bottom of each adsorption unit are monitored in real time by a high-frequency micro pressure sensor at the end of the adsorption device. Microscopic deformation images of the contact area between the adsorption head and the shell are obtained using a stereo vision sensor; Decision-making is based on micro-pressure changes and micro-deformation images. When the pressure distribution reaches the preset stable contact threshold and the visual deformation characteristics meet the expected effective adsorption, it is determined that the grasping conditions have been met and the grasping command is triggered.

7. The multi-modal sensing abalone AI adaptive adsorption and grasping control method according to claim 1, characterized in that, After adsorption is established, the system continuously collects and fuses sensor data streams at high speed. Using a slip prediction model pre-trained based on historical data, it outputs in real-time the stability of the current adsorption state and the probability of slip risk, until the gripping process ends and the negative pressure is safely released. Specifically, this includes: After adsorption is established, sensor data streams are continuously collected and the data is integrated into a unified feature vector. The slip prediction model, which is pre-trained based on historical capture data, inputs the feature vector of the fused sensor data into the slip prediction model and outputs the stability of the current adsorption state and the probability of slippage of the adsorption device in the current state in a probabilistic form. Thresholds for adsorption stability and slippage risk are set based on historical data. Throughout the grasping process, sensor data is continuously collected and the adsorption status is monitored through a slippage prediction model. At the end of the grasping process, check the sensor data and the output of the slip prediction model, check the status of the adsorption device, and reduce the negative pressure of the flexible adsorption head to separate the adsorption unit from the abalone shell.

8. A multimodal sensing abalone AI adaptive adsorption and grasping control system, used to implement the multimodal sensing abalone AI adaptive adsorption and grasping control method as described in any one of claims 1-7, characterized in that, include: Stereo vision sensing module: used to acquire point cloud data of the production line in real time through stereo vision sensors and build a 3D scene map; Abalone identification module: Used to process point cloud data based on a deep learning-based segmentation model, identify individual abalone and extract location contour point clouds; 3D model building module: used to fuse abalone location contour point cloud data with laser scan data to build a high-precision 3D model of the abalone shell; Adsorption region screening module: used to obtain the optimal adsorption contact region by training a model through a convolutional neural network based on historical data; Approach path and pose module: used to generate the approach path and optimal adsorption pose of the adsorption device through inverse kinematics calculations; Grasping decision-making module: used to determine the adsorption stability through pressure sensor and stereo vision sensor, and trigger grasping command when the grasping conditions are met; Slip Prediction and Stability Assessment Module: This module is used to output the stability of the adsorption state and the probability of slip risk in real time using a slip prediction model pre-trained with historical data. Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

Images