An AI vision-based man-machine collaborative unloading and intelligent logistics scheduling system

Abstract

This invention relates to the field of image recognition technology, specifically to a human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision. A line laser scanner acquires 3D point cloud data of workpieces within a material basket, and a multi-scale feature extraction network extracts multi-scale point cloud features. A physical prior calculation module generates a 3D tensor mask for physical stability based on the projected support area of ​​the point cloud clusters, the centroid offset, and the angle between the point cloud normal vector and the direction of gravity. This mask is multiplied element-wise with the multi-scale point cloud features to generate fused point cloud features. A pose regression head outputs the six-degree-of-freedom pose of candidate workpieces based on the fused point cloud features. A grasping interference evaluation module combines this with the geometric model of the end-effector. This invention introduces physical stability priors into the visual feature extraction process, suppressing pseudo-feature responses from suspended and unstable workpieces, eliminating pseudo-poses that are not feasible for grasping, and using intersection volume calculation to eliminate collision interference poses, thus avoiding physical collisions between the fixture and surrounding workpieces.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision. Background Technology

[0002] In existing technologies, when loading and unloading workpieces that are randomly stacked in a basket, a line laser 3D scanner is typically used to acquire 3D point cloud data of the workpieces within the basket. This 3D point cloud data is then input into a deep learning-based pose estimation network. This network extracts the geometric structure and local curvature features of the point cloud through a multi-scale feature extraction network, and then outputs the six-DOF pose of the candidate workpiece through a pose regression head. After acquiring the six-DOF pose, the industrial robot directly generates the motion trajectory of the end effector gripper based on this pose and performs the grasping action. The entire visual perception and pose calculation process relies solely on the geometric spatial distribution of the point cloud data itself.

[0003] In real-world industrial scenarios, workpieces within a basket are stacked in a disordered manner, with complex physical occlusions and adhesions between them. Because existing pose estimation networks rely solely on matching the geometric structure and local curvature features of point clouds, when the edges of multiple workpieces are stitched together to form a geometric contour resembling a single workpiece, or when a workpiece is suspended and supported by other workpieces in an unstable state, the geometric feature extraction process cannot distinguish between the actual workpiece capable of being grasped and the stitched surface that generates false features. This causes the candidate pose output by the pose regression head to deviate from the true physical state of the workpiece. When the robotic arm's end effector performs a grasping action based on this deviated pose, it will physically collide with other surrounding workpieces within the basket. Summary of the Invention

[0004] The purpose of this invention is to provide a human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision, which can solve the problems in the background art mentioned above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision includes a line laser 3D scanner, a multi-scale feature extraction network, a physical prior calculation module, a pose regression head, a grasping interference evaluation module, and a robot controller. The line laser 3D scanner acquires 3D point cloud data of randomly stacked workpieces within a material basket. The multi-scale feature extraction network receives the 3D point cloud data and outputs multi-scale point cloud features including geometric structure and local curvature. The physical prior calculation module receives the 3D point cloud data and calculates the projected support area, center of gravity offset, and other parameters of the point cloud clusters in the gravity direction. The angle between the point cloud normal vector and the gravity vector is used to generate a three-dimensional tensor mask characterizing physical stability. The three-dimensional tensor mask is applied to the multi-scale point cloud features through element-wise multiplication to output fused point cloud features. The pose regression head receives the fused point cloud features and outputs the six-degree-of-freedom pose of the candidate workpiece. The grasping interference evaluation module, combined with the pre-built geometric model of the end effector of the robotic arm, calculates the intersection volume between the bounding box of the gripper and the point cloud surrounding the candidate pose in three-dimensional space, eliminates candidate poses with collision interference, and outputs the optimal interference-free pose and the corresponding gripper approach vector to the robot controller.

[0006] Preferably, when calculating the projected support area, the physical prior calculation module projects the three-dimensional point cloud data onto a horizontal reference plane along the gravity direction, performs convex hull operations on the points within the projection area, and solves for the area of ​​the convex hull polygon. When calculating the centroid offset, it extracts the point cloud cluster of a single workpiece and calculates the three-dimensional geometric center coordinates of the point cloud cluster, obtains the centroid coordinates of the bottom support point set of the point cloud cluster, and uses the Euclidean distance vector between the three-dimensional geometric center coordinates and the centroid coordinates as the centroid offset. When calculating the included angle, it uses principal component analysis to extract the normal vector of the point cloud cluster, calculates the cosine value between the normal vector and the reverse vector of the gravity direction, and inputs the projected support area, the centroid offset, and the cosine value into a pre-constructed three-layer fully connected layer to generate the three-dimensional tensor mask aligned with the feature space dimension of the multi-scale point cloud.

[0007] Preferably, the multi-scale feature extraction network includes a bottom sparse convolutional layer, a middle sparse convolutional layer, and a high sparse convolutional layer connected in series. The bottom sparse convolutional layer performs initial feature mapping on the input 3D point cloud data to extract local edge geometric features. The middle sparse convolutional layer receives the local edge geometric features and expands the receptive field to extract the overall structural features of the workpiece surface. The high sparse convolutional layer receives the overall structural features and extracts global semantic features containing contextual information. The multi-scale feature extraction network also includes a feature concatenation layer. The feature concatenation layer concatenates the local edge geometric features, the overall structural features, and the global semantic features in the channel dimension, and inputs the concatenated features into a one-dimensional convolutional layer for dimensionality reduction, outputting the multi-scale point cloud features containing geometric structure and local curvature.

[0008] Preferably, after receiving the six-degree-of-freedom pose of the candidate workpiece, the grasping interference evaluation module retrieves the pre-built geometric model of the end effector of the robotic arm, applies the translation vector and rotation matrix of the six-degree-of-freedom pose to the base coordinate point set of the geometric model of the end effector of the robotic arm, obtains the vertex coordinates of the bounding box of the clamp in three-dimensional space, uses the smallest circumscribed cylinder of the clamping bounding box as the collision detection body, extracts the local point cloud within a set radius around the candidate pose, calculates the number of points in the local point cloud that fall into the smallest circumscribed cylinder, and when the number of points exceeds a preset safety threshold, it is determined that the candidate pose has collision interference and a rejection operation is performed.

[0009] Preferably, the pose regression head includes parallel translation and rotation branches. The translation branch performs 3D bounding box regression on the fused point cloud features to output the 3D center coordinates of the candidate workpiece in the camera coordinate system. The rotation branch performs classification processing on the fused point cloud features to output the Euler angles of the candidate workpiece around the three coordinate axes. The 3D center coordinates and the Euler angles are combined to form the six-degree-of-freedom pose of the candidate workpiece. When generating the fixture approach vector, the 3D center coordinates are used as the starting point, and a set distance is extended along the workpiece surface normal direction determined by the Euler angles to obtain the ending point. The spatial vector from the starting point to the ending point is normalized to generate the fixture approach vector.

[0010] Preferably, before the 3D tensor mask is applied to the multi-scale point cloud features through element-wise multiplication, a dimension alignment operation is performed on the multi-scale point cloud features and the 3D tensor mask to extract the spatial resolution size of the multi-scale point cloud features. A trilinear interpolation algorithm is then used to upsample the 3D tensor mask so that the spatial resolution size of the upsampled 3D tensor mask is the same as that of the multi-scale point cloud features. A channel dimension copying and expansion operation is then performed on the upsampled 3D tensor mask so that the number of channels in the expanded 3D tensor mask is the same as the number of channels in the multi-scale point cloud features. Finally, the channel-expanded 3D tensor mask is multiplied element-wise with the multi-scale point cloud features to output the fused point cloud features.

[0011] Preferably, before extracting the point cloud cluster of a single workpiece, the three-dimensional point cloud data is subjected to voxel rasterization downsampling to filter out redundant spatial points. The downsampled three-dimensional point cloud data is then subjected to a region growing-based segmentation operation. The point with the smallest curvature is selected as the seed point, and neighboring points with a normal vector angle less than a set threshold are merged into the same region to generate an independent point cloud cluster. Before extracting the normal vector using principal component analysis, a local neighborhood search sphere is constructed for each point in the point cloud cluster. A covariance matrix is ​​constructed based on the three-dimensional coordinates of the neighboring points in the local neighborhood search sphere. The covariance matrix is ​​then subjected to eigenvalue decomposition, and the eigenvector corresponding to the smallest eigenvalue is used as the normal vector of the corresponding point in the point cloud cluster.

[0012] Preferably, when the number of point clouds does not exceed the preset safety threshold, the vertex coordinates of the fixture bounding box are obtained to construct a three-dimensional axis-aligned bounding box. The three-dimensional axis-aligned bounding box is divided into multiple cubic voxels. The local point cloud is mapped to the cubic voxels to generate an occupying voxel set. The fixture bounding box is mapped to the cubic voxels to generate a fixture voxel set. A bitwise logical AND operation is performed on the occupying voxel set and the fixture voxel set. The total number of voxels in the operation result that are in the occupying state is counted. The total number of voxels is multiplied by the volume of a single cubic voxel to obtain the actual intersection volume. When the actual intersection volume exceeds the preset volume threshold, it is determined that there is a collision interference and a culling operation is performed.

[0013] Preferably, after generating the clamp approach vector, the length and width dimensions of the 3D bounding box output by the translation branch are obtained. The dimension with the smaller value among the length and width dimensions is taken as the dimension to be avoided. The rotation angle of the Euler angle around the coordinate axis corresponding to the dimension to be avoided is extracted as the initial deflection angle. The clamp approach vector is rotated in the opposite direction around the coordinate axis corresponding to the dimension to be avoided by the initial deflection angle to obtain a spatial adjustment vector. The angle between the spatial adjustment vector and the horizontal plane is calculated. When the angle is less than the set anti-collision elevation angle, the spatial adjustment vector is rotated in the positive direction around the horizontal axis perpendicular to the dimension to be avoided by the difference between the anti-collision elevation angle and the angle, and the final clamp approach vector is output.

[0014] Preferably, in the process of mapping the fixture bounding box to the cube voxel to generate a fixture voxel set, a local coordinate system is constructed with the center of the fixture bounding box as the origin and collinear with the three-dimensional axis-aligned bounding box. The direction vector of the fixture proximity vector in the local coordinate system is extracted. The fixture bounding box is extended by a set distance in the opposite direction of the direction vector to construct a motion sweep bounding box. The overlapping area of ​​the motion sweep bounding box and the three-dimensional axis-aligned bounding box is taken as the detection area. The mapping operation of the fixture bounding box is performed on the cube voxel only in the detection area to generate the fixture voxel set. The bitwise logical AND operation is performed on the occupied voxel set only in the detection area.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention obtains the projected support area, centroid offset, and angle between the point cloud normal vector and the gravity direction of a point cloud cluster through a physical prior calculation module, generating a three-dimensional tensor mask characterizing physical stability. This three-dimensional tensor mask is applied to multi-scale point cloud features through element-wise multiplication, enabling the pose regression head to exclude pseudo-feature point responses in suspended or unstable support states when calculating pose. Combined with the grasping interference evaluation module, the actual intersection volume between the fixture bounding box and the local point cloud is calculated, and candidate poses with collision interference are eliminated. This solves the problem of pose estimation divergence caused by occlusion and pseudo-feature interference in disordered stacking scenes, and avoids collision interference between the grasping action and surrounding workpieces.

[0016] 2. Before generating the 3D tensor mask, a dimensional alignment operation involving trilinear interpolation upsampling and channel dimension replication expansion is performed on the multi-scale point cloud features and the 3D tensor mask to ensure spatial and channel consistency when the mask performs element-wise multiplication of the multi-scale point cloud features. When calculating the actual intersection volume, a 3D axis-aligned bounding box is used to divide the voxels. The point cloud and fixture bounding boxes are mapped to voxels for bitwise logical AND operations, transforming the calculation of the spatial 3D intersection volume into a statistical analysis of the number of discrete voxels, reducing the computational load of the interference evaluation process. By constructing a motion-sweep bounding box to define the detection area, the search space for voxel logical AND operations is reduced. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall operation of the human-machine collaborative loading and unloading and intelligent logistics scheduling system of the present invention. Figure 2 This is a flowchart of the internal calculation process of the physical prior calculation module of the present invention. Figure 3 This is a flowchart illustrating the structure extraction process of the multi-scale feature extraction network of the present invention. Figure 4 This is a flowchart illustrating the dimensional alignment and fusion of 3D tensor masks and multi-scale point cloud features according to the present invention. Figure 5 This is a preliminary collision detection flowchart of the present invention; Figure 6 This is a flowchart illustrating the precise intersection volume calculation and sweeping region definition of the present invention. Detailed Implementation

[0018] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0019] Please refer to Figure 1 This embodiment provides a human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision, which is deployed in an industrial disordered loading and unloading operation scenario. The operation scenario is equipped with a basket for loading disordered stacked workpieces, a six-axis industrial robot, a line laser 3D scanner and a robot controller. The line laser 3D scanner is fixedly installed at a preset mounting position above the basket, and its scanning field of view completely covers the internal loading space of the basket.

[0020] Specifically, after receiving the scan trigger signal from the host computer, the line laser 3D scanner scans the internal space of the basket line by line along a preset scanning path, acquiring 3D point cloud data of the disordered stacked workpieces within the basket. Each point in the 3D point cloud data contains 3D coordinate values ​​in the camera coordinate system. The Z-axis of the camera coordinate system is parallel to the scanning direction, the Y-axis is parallel to the opposite direction of gravity, and the X-axis is parallel to the horizontal direction. The origin of the 3D point cloud data coincides with the optical center of the line laser 3D scanner. After completing a single scan, the line laser 3D scanner synchronously transmits the acquired 3D point cloud data to the multi-scale feature extraction network and physical prior calculation module.

[0021] After receiving 3D point cloud data, the multi-scale feature extraction network performs voxelization on the data, converting continuous 3D spatial coordinates into discrete voxel grid coordinates. A sparse feature tensor is constructed based on these voxel grid coordinates, where non-zero elements correspond to valid voxel positions containing the point cloud data. The multi-scale feature extraction network performs multi-order convolution operations on the sparse feature tensor to extract point cloud features from different receptive fields. These features are then fused to output multi-scale point cloud features that include geometric structure and local curvature. The multi-scale point cloud features are 3D feature tensors, with their spatial dimension corresponding to the 3D voxel grid coordinates and their channel dimension corresponding to the extracted geometric and curvature feature dimensions.

[0022] After receiving the 3D point cloud data, the physical prior calculation module performs clustering and segmentation on the data, dividing it into multiple independent point cloud clusters. Each cluster corresponds to the point cloud set of a single workpiece. For each independent point cloud cluster, the module calculates the projected support area, centroid offset, and angle between the point cloud normal vector and the gravity vector in the gravity direction. Based on these three parameters, a physical stability score is generated for the corresponding point cloud cluster. The physical stability scores of all point cloud clusters are then mapped onto a 3D spatial mesh to generate a 3D tensor mask representing physical stability. The spatial dimension of the 3D tensor mask is consistent with the voxel mesh spatial dimension of the 3D point cloud data. Each element in the mask ranges from 0 to 1; higher values ​​indicate stronger physical stability of the point cloud cluster at the corresponding spatial location, increasing its grasping feasibility. In the process of generating fused point cloud features, a 3D tensor mask is applied to the multi-scale point cloud features through element-wise multiplication. This suppresses the feature responses of unstable point cloud clusters and preserves the feature responses of stable point cloud clusters, outputting fused point cloud features that incorporate physical prior information. The spatial and channel dimensions of the fused point cloud features are completely consistent with those of the multi-scale point cloud features, and their feature response intensity is simultaneously correlated with the geometric structure features of the point cloud and the physical stability of the corresponding workpiece.

[0023] After receiving the fused point cloud features, the pose regression head performs global average pooling on the fused point cloud features, converting the 3D feature tensor into a 1D feature vector. Based on the 1D feature vector, it performs pose calculation operations and outputs at least one set of six-DOF poses for candidate workpieces. The six-DOF pose includes the 3D center coordinates of the candidate workpiece in the camera coordinate system and the rotation parameters about the three coordinate axes, which can completely characterize the position and orientation of the candidate workpiece in 3D space.

[0024] After receiving the six-DOF pose of a candidate workpiece, the grasping interference evaluation module retrieves a pre-built geometric model of the robotic arm's end effector. This model is a three-dimensional mesh model containing the fixture body, grippers, and gripping space, and is pre-stored in the system's local storage. The module applies the spatial transformation parameters corresponding to the six-DOF pose to the geometric model, completing the pose mapping of the fixture model in three-dimensional space and constructing the mapped fixture bounding box. It then calculates the intersection volume between the bounding box and the point cloud surrounding the candidate pose in three-dimensional space. The module has a preset safety threshold for the intersection volume. If the calculated intersection volume exceeds the safety threshold, the candidate pose is determined to have collision interference, and it is eliminated. If the intersection volume does not exceed the safety threshold, the candidate pose is determined to be interference-free. The grasping interference evaluation module sorts the physical stability scores corresponding to all interference-free poses, selects the interference-free pose with the highest score as the optimal interference-free pose, and generates a gripper approach vector corresponding to the optimal interference-free pose. The optimal interference-free pose and the gripper approach vector are then output to the robot controller.

[0025] After receiving the optimal interference-free pose and the gripper approach vector, the robot controller executes the hand-eye calibration parameter call operation, converting the pose parameters in the camera coordinate system into pose parameters in the robot base coordinate system. Based on the converted pose parameters and the gripper approach vector, the robot arm's motion trajectory is planned. The motion trajectory is divided into a pre-approach segment, a contact segment, a gripping segment, and an extraction segment. In the pre-approach segment, the robot arm moves from its current position to a pre-approach point at a preset distance from the workpiece along the gripper approach vector. In the contact segment, the robot arm moves linearly from the pre-approach point along the gripper approach vector to the gripping position. In the gripping segment, the grippers close according to a preset gripping force to complete the workpiece gripping. In the extraction segment, the robot arm moves in the opposite direction of the gripper approach vector to extract the workpiece from the basket, completing the loading and unloading operation of a single workpiece. After a single loading and unloading operation is completed, the system repeats the scanning, feature extraction, pose calculation, and interference evaluation process until all workpieces in the basket that are feasible to be gripped have completed the loading and unloading operation.

[0026] Table 1 Input / output and core computational logic at each stage of the 3D point cloud data processing workflow Table 1 fully presents the entire process logic of the 3D point cloud data in this embodiment, from acquisition to final output of interference-free grasping pose and completion of loading and unloading operations. It clarifies the correspondence between input and output data and the core calculation content of each processing stage. Those skilled in the art can use this table to clarify the execution order and data flow relationship of each processing stage.

[0027] This embodiment fully presents the basic operation process of the system. By integrating multi-scale feature extraction and physical prior information, it realizes the pose calculation of disordered stacked workpieces. By calculating the intersection volume, it completes the elimination of interference poses, ensuring the feasibility and safety of the gripping action during the loading and unloading operation.

[0028] In one alternative embodiment, refer to Figure 2 and Figure 4 The parameter calculation logic of the physical prior calculation module, the generation process of the three-dimensional tensor mask, and the dimension alignment operation before feature fusion are refined.

[0029] Specifically, after receiving the 3D point cloud data, the physical prior calculation module first performs voxel rasterization downsampling on the 3D point cloud data to filter out redundant spatial points. During voxel rasterization downsampling, the 3D space corresponding to the 3D point cloud data is divided into multiple cubic voxel grids of uniform size. The side length of the voxel grid is pre-set based on the minimum geometric feature size of the target workpiece. For each voxel grid containing at least one point cloud data point, the average value of all point cloud coordinates within that voxel grid is calculated. This average value is used as the unique representative point of that voxel grid, completing the downsampling process. The downsampled 3D point cloud data retains the overall geometric structure of the original point cloud while reducing the number of points in the point cloud data, thus reducing the computational load of subsequent operations.

[0030] Furthermore, a region-growing-based segmentation operation is performed on the downsampled 3D point cloud data to generate independent point cloud clusters. During the region-growing segmentation operation, firstly, for each point in the downsampled 3D point cloud data, a local neighborhood search sphere of a preset radius is constructed. All neighboring points within the local neighborhood search sphere are extracted, and a covariance matrix is ​​constructed based on the 3D coordinates of the neighboring points. Eigenvalue decomposition is then performed on the covariance matrix to obtain the normal vector and local curvature value of that point. All points are sorted according to their local curvature values ​​from smallest to largest, and the point with the smallest curvature is selected as the initial seed point. The point with the smallest curvature corresponds to a flat area on the workpiece surface, which can reduce error accumulation during the region growing process. A local neighborhood search sphere of a preset radius is constructed with the initial seed point as the center. All neighboring points within the local neighborhood search sphere are extracted, and the angle between the normal vector of the seed point and each neighboring point is calculated. Neighboring points with a normal vector angle less than a set threshold are merged into the same region as the seed point. For neighboring points that are merged into the same region, they are used as new seed points. The neighborhood search and neighbor point merging operations are repeated until no new neighboring points can be merged into the region, thus completing the growth of a single region. For the remaining points that have not been assigned to any region, the seed point selection and region growth operations are repeated until all points are assigned to their corresponding regions. Each independent region corresponds to an independent point cloud cluster, and each point cloud cluster corresponds to the point cloud set of a single workpiece.

[0031] When extracting the normal vectors of points within a point cloud cluster, for each point within the cluster, a local neighborhood search sphere with a preset radius is constructed. All neighboring points within the local neighborhood search sphere are extracted, and a covariance matrix is ​​constructed based on the 3D coordinates of the neighboring points. The formula for calculating the covariance matrix is ​​as follows: in, This represents the number of neighboring points within the local neighborhood search sphere. For the neighboring region Three-dimensional coordinate vectors of neighboring points Let be the centroid vector of the three-dimensional coordinates of all neighboring points within the neighborhood. It is a 3×3 covariance matrix.

[0032] The constructed covariance matrix is ​​subjected to eigenvalue decomposition, yielding three eigenvalues ​​and their corresponding eigenvectors arranged in descending order. These three eigenvalues ​​correspond to the coordinate distribution variances of three orthogonal directions within the local neighborhood of the point cloud. The eigenvector corresponding to the smallest eigenvalue is the normal direction of the local plane of the point cloud, and this eigenvector is used as the normal vector for that point. This covariance matrix construction and eigenvalue decomposition operation is repeated for all points within the point cloud cluster to obtain the normal vectors for all points within the cluster.

[0033] Furthermore, for each independent point cloud cluster, the physics prior calculation module calculates the projected support area, centroid offset, and angle between the point cloud normal vector and the gravity vector of the point cloud cluster in the gravity direction.

[0034] When calculating the projected support area, all 3D points of the point cloud cluster are projected onto a horizontal reference plane along the direction of gravity. The horizontal reference plane is the plane formed by the X and Z axes in the camera coordinate system. After projection, the 2D plane coordinates corresponding to each 3D point are obtained. During the projection process, the coordinate values ​​of the point cloud on the X and Z axes are retained, while the coordinate values ​​in the Y direction are removed. A convex hull operation is performed on all 2D coordinate points within the projection area to obtain the convex hull polygon that encloses all projected points. The area of ​​the convex hull polygon is calculated based on the vertex coordinates of the convex hull polygon. This area is the projected support area of ​​the point cloud cluster. The formula for calculating the area of ​​the convex hull polygon is: in, This represents the number of vertices of the convex hull polygon. For the convex hull polygon The two-dimensional planar coordinates of the vertices, and satisfying , , The area of ​​the convex hull polygon is the projected support area. The larger the projected support area, the larger the contact area between the workpiece corresponding to the point cloud cluster and the support surface below, and the stronger the physical stability.

[0035] When calculating the centroid offset, the three-dimensional coordinates of all points within the point cloud cluster are extracted, and the average of all three-dimensional coordinates is calculated to obtain the three-dimensional geometric center coordinates of the point cloud cluster. The set of points with the smallest coordinate values ​​along the gravity direction within the point cloud cluster is extracted as the bottom support point set. The bottom support point set is the contact area between the point cloud cluster and the supporting surface below or other workpieces. The average three-dimensional coordinates of all points within the bottom support point set are calculated to obtain the centroid coordinates of the bottom support point set. The Euclidean distance vector between the coordinates of the three-dimensional geometric center and the coordinates of the centroid is taken as the centroid offset. The formula for calculating the centroid offset is: in, The magnitude of the vector represents the degree of offset between the geometric center and the supporting centroid. The larger the magnitude, the higher the risk of the point cloud cluster overturning and the worse its physical stability.

[0036] When calculating the angle between the point cloud normal vector and the gravity vector, the gravity direction vector is extracted. The gravity direction vector is parallel to the negative Y-axis of the camera coordinate system, and the unit gravity direction vector is... For the normal vectors of all points within the point cloud cluster, calculate the cosine value between the normal vector and the vector opposite to the direction of gravity. The formula is: in, The normal vector of a single point within a point cloud cluster. It is the unit vector in the opposite direction to the direction of gravity. The angle between the normal vector and the vector opposite to the direction of gravity. The value of the cosine of the angle is 1. The closer the cosine value is to 1, the smaller the angle between the normal vector of the corresponding point and the opposite vector of gravity, the closer the supporting surface of the point cloud cluster is to horizontal, and the stronger the physical stability.

[0037] Furthermore, the projected support area, the magnitude of the centroid offset, and the average cosine value of all points within a single point cloud cluster are input into a pre-constructed three-layer fully connected layer. The three-layer fully connected layer has an input dimension of 3, a hidden layer dimension of 16, and an output dimension of 1. The activation function is the Sigmoid function, and the output is a physical stability score between 0 and 1, which is positively correlated with the physical stability of the point cloud cluster. During the training of the three-layer fully connected layer, a workpiece point cloud dataset labeled with physical stability is used as the training set. The samples in the training set include workpiece point clouds in different stacking states. The labeled physical stability score is determined based on the results of actual grasping experiments; samples that are successfully grasped are assigned a score of 1, and samples that overturn during grasping are assigned a score of 0. The weight parameters of the fully connected layer are optimized using the backpropagation algorithm to minimize the mean square error between the physical stability score output by the fully connected layer and the labeled value. The physical stability scores of all point cloud clusters are mapped to a 3D spatial grid. The mask element of the spatial grid position corresponding to each point cloud cluster is the physical stability score of that point cloud cluster. The mask element of the spatial grid position without point cloud data is 0, thus generating the initial 3D tensor mask.

[0038] Before applying the 3D tensor mask to the multi-scale point cloud features through element-wise multiplication, a dimension alignment operation is performed on the multi-scale point cloud features and the 3D tensor mask. Specifically, the spatial resolution size of the multi-scale point cloud features is extracted, which is the number of voxel grids in the X, Y, and Z axes of the 3D feature tensor. A trilinear interpolation algorithm is then used to upsample the initial 3D tensor mask, ensuring that the spatial resolution size of the upsampled 3D tensor mask is identical to that of the multi-scale point cloud features. The formula for trilinear interpolation is: in, The mask values ​​are the values ​​of the eight neighboring voxels around the point to be interpolated in the initial 3D tensor mask. The interpolation weights for the corresponding neighborhood voxels are negatively correlated with the distance from the point to be interpolated to the neighborhood voxels. This is the mask value obtained after interpolation.

[0039] Furthermore, a channel-dimensional copy expansion operation is performed on the upsampled 3D tensor mask to extract the number of channels of the multi-scale point cloud features. The upsampled single-channel 3D tensor mask is then copied and expanded along the channel dimension until the number of channels in the expanded 3D tensor mask is exactly the same as the number of channels in the multi-scale point cloud features, resulting in a dimensionally aligned 3D tensor mask. The channel-expanded 3D tensor mask is then multiplied element-wise with the multi-scale point cloud features to output the fused point cloud features. The formula for element-wise multiplication is as follows: in, For multi-scale point cloud features, This is the dimension-aligned 3D tensor mask. This represents element-wise multiplication. The output is the fused point cloud feature.

[0040] Table 2 Mapping rules between physical stability characteristic parameters and 3D tensor mask weights Table 2 clarifies the physical stability determination results and the weighting rules of the three-dimensional tensor mask corresponding to different combinations of the three types of physical stability characteristic parameters. The preset area threshold, first offset threshold, second offset threshold, and third offset threshold are all preset based on the actual geometric dimensions of the target workpiece. Those skilled in the art can adjust the corresponding thresholds and weighting ranges according to the specifications of the target workpiece.

[0041] This embodiment refines the entire process of point cloud preprocessing, clustering and segmentation, physical stability parameter calculation, 3D tensor mask generation and dimension alignment. It achieves accurate characterization of the physical stability of the workpiece through the quantitative calculation of three types of physical parameters, and ensures the consistency of space and channels during feature fusion through dimension alignment operation. It effectively suppresses the feature response of unstable workpieces and avoids the interference of pseudo-features on the pose calculation process.

[0042] In one alternative embodiment, refer to Figure 3 The network structure of the multi-scale feature extraction network, the pose calculation logic of the pose regression head, and the generation and adjustment process of the fixture proximity vector are refined.

[0043] Specifically, the multi-scale feature extraction network comprises a series of cascaded sparse convolutional layers: a bottom-level sparse convolutional layer, a middle-level sparse convolutional layer, a high-level sparse convolutional layer, and a feature concatenation layer. The output of the feature concatenation layer is connected to a one-dimensional convolutional layer. After receiving 3D point cloud data, the multi-scale feature extraction network first performs voxelization on the 3D point cloud data to construct a feature concatenation layer with a size of [missing information]. The sparse feature tensor, in which These represent the dimensions of the voxel grid along the X, Y, and Z axes, respectively. The non-zero elements of the sparse feature tensor correspond to the valid voxel positions containing point cloud data, while the zero elements correspond to the empty voxel positions without point cloud data.

[0044] The bottom sparse convolutional layer receives a sparse feature tensor and performs a 3×3×3 sparse convolution operation on the input sparse feature tensor. The convolution stride is set to 1, padding is set to 1, and the number of output channels is set to 16. This performs initial feature mapping on the input 3D point cloud data and extracts local edge geometric features. The sparse convolution operation only performs convolution calculations at non-zero voxel positions containing valid point cloud data, skipping zero voxel positions without point cloud data, thus reducing computational cost while preserving the local geometric details of the point cloud. The calculation formula for the sparse convolution operation is: in, Let be the input sparse feature tensor. For the output feature tensor, The weight parameters are those of the sparse convolution kernel. The convolution kernel has a spatial size of 3×3×3. The spatial coordinates of the effective voxels.

[0045] The middle sparse convolutional layer receives the local edge geometric features output by the bottom sparse convolutional layer, performs a 3×3×3 sparse convolution operation on the input features, sets the convolution stride to 2, padding to 1, and the number of output channels to 32. By expanding the receptive field of the features through the convolution operation with a stride of 2, the overall structural features of the workpiece surface are extracted.

[0046] The high-level sparse convolutional layer receives the overall structural features output by the middle-level sparse convolutional layer, performs a 3×3×3 sparse convolution operation on the input features, sets the convolution stride to 2, padding to 1, and the number of output channels to 64, further expanding the receptive field and extracting global semantic features containing workpiece context information.

[0047] The feature concatenation layer receives local edge geometric features from the bottom sparse convolutional layer, overall structural features from the middle sparse convolutional layer, and global semantic features from the high-level sparse convolutional layer. It performs upsampling on each of these three types of features to ensure they have the same spatial resolution. The three types of features are then concatenated along the channel dimension to obtain a concatenated feature with 112 channels. The one-dimensional convolutional layer receives the concatenated feature and performs a one-dimensional convolution operation with a kernel size of 1 on it, outputting 32 channels. It then performs dimensionality reduction on the concatenated high-dimensional feature, outputting multi-scale point cloud features that include geometric structure and local curvature. The formulas for feature concatenation and dimensionality reduction are as follows: in, This represents the local edge geometric features output by the bottom sparse convolutional layer. This represents the overall structural features output by the middle sparse convolutional layer. These are the global semantic features output by high-level sparse convolutional layers. This is a channel-level splicing operation. It is a one-dimensional convolutional layer. Number of output channels This is the output multi-scale point cloud feature.

[0048] During the training of the multi-scale feature extraction network, a publicly available industrial workpiece point cloud pose estimation dataset is used as the pre-training dataset, and an unordered stacked workpiece point cloud dataset collected in actual operation scenarios is used as the fine-tuning dataset. During training, the average error of pose estimation is used as the loss function, and the weight parameters of sparse convolutional layers and one-dimensional convolutional layers are optimized through the backpropagation algorithm to improve the accuracy of feature extraction.

[0049] The pose regression head receives fused point cloud features and includes parallel translation and rotation branches. The inputs of both the translation and rotation branches are connected to a global average pooling layer. The global average pooling layer performs a global average pooling operation on the fused point cloud features, converting the 3D feature tensor into a 32-dimensional feature vector. This 3D feature vector is then input into the translation and rotation branches, respectively.

[0050] The translation branch consists of two fully connected layers connected in series. The first fully connected layer has an input dimension of 32 and an output dimension of 16. The second fully connected layer has an input dimension of 16 and an output dimension of 3. The translation branch performs 3D bounding box regression on the input one-dimensional feature vector and outputs the 3D center coordinates of the candidate workpiece in the camera coordinate system. The 3D center coordinates are the geometric center of the 3D bounding box of the candidate workpiece.

[0051] The rotation branch consists of two fully connected layers connected in series. The first fully connected layer has an input dimension of 32 and an output dimension of 16, while the second fully connected layer has an input dimension of 16 and an output dimension of 3. The rotation branch classifies the input one-dimensional feature vector and outputs the Euler angles of the candidate workpiece around the X, Y, and Z coordinate axes, which are the rotation angles around the X-axis. Rotation angle about the Y-axis Rotation angle about the Z-axis The six-degree-of-freedom pose of the candidate workpiece is formed by combining the three-dimensional center coordinates and three Euler angles. The formula for calculating the rotation matrix corresponding to the Euler angles is: in, Let X be the rotation matrix about the X-axis. Let be the rotation matrix about the Y-axis. Let Z be the rotation matrix about the Z-axis. The combined 3×3 rotation matrix can transform the workpiece model from the model coordinate system to the camera coordinate system.

[0052] During the training of the pose regression head, the translation branch uses the L1 loss function, the rotation branch uses the mean squared error loss function, and the total loss function is the weighted sum of the translation branch loss and the rotation branch loss. The weight parameters of the fully connected layer are optimized through training to improve the accuracy of pose regression.

[0053] When generating the fixture approach vector, the three-dimensional center coordinates output by the translation branch are used as the starting point. The endpoint is obtained by extending a set distance along the normal direction of the workpiece surface determined by Euler angles. The normal direction of the workpiece surface is parallel to the normal vector of the candidate workpiece clamping surface. The clamping surface is a pre-defined flat area that can be clamped on the workpiece. The spatial vector from the starting point to the ending point is normalized to generate the fixture approach vector. The normalization calculation formula is: in, This is the normalized initial clamp proximity vector. The distance between the starting point and the ending point.

[0054] Furthermore, after generating the initial fixture approach vector, collision avoidance adjustment processing is performed on the fixture approach vector. Specifically, the length and width dimensions of the 3D bounding box output by the translation branch are obtained. The length and width dimensions of the 3D bounding box are the dimensions upward along the X and Z axes. The dimension with the smaller value among the length and width dimensions is selected as the dimension to be avoided. The dimension to be avoided corresponds to the narrow side direction of the workpiece, where the avoidance space is smaller and the collision risk is higher. The rotation angle of the coordinate axis corresponding to the dimension to be avoided is extracted as the initial deflection angle. The initial fixture approach vector is rotated in the opposite direction around the coordinate axis corresponding to the dimension to be avoided by the initial deflection angle to obtain the spatial adjustment vector. The formula for calculating the vector rotation is: in, This is the rotation matrix corresponding to the reverse rotation. Let this be the initial clamp proximity vector. This is the spatial adjustment vector obtained after rotation.

[0055] The calculation determines the angle between the spatial adjustment vector and the horizontal plane, which is the plane formed by the X and Z axes in the camera coordinate system. When the calculated angle is less than the set anti-collision elevation angle, the spatial adjustment vector is rotated positively around the horizontal axis perpendicular to the dimension to be avoided. The difference between the anti-collision elevation angle and the current angle is then used to output the final fixture approach vector. The anti-collision elevation angle is preset based on the stacking height of the workpieces in the basket and the dimensions of the fixture, ensuring that the fixture maintains a sufficient safe distance from the stacked workpieces below during the approach process.

[0056] Table 3. Operational logic and feature output dimensions of each layer of the multi-scale feature extraction network. Table 3 clarifies the input and output dimensions, core computational parameters, and extracted feature types for each layer of the multi-scale feature extraction network. To determine the basic dimensions of the voxel mesh corresponding to the input 3D point cloud data in three axes, those skilled in the art can adjust parameters such as the convolution kernel size, the number of output channels, and the convolution stride according to the geometric complexity of the target workpiece.

[0057] This embodiment refines the hierarchical structure and computational logic of the multi-scale feature extraction network. It achieves the extraction and fusion of features under different receptive fields through three sparse convolutional layers, ensuring the integrity and discriminability of point cloud features. It also refines the parallel branch structure of the pose regression head and the solution process of the six-degree-of-freedom pose, achieving accurate regression of the workpiece pose. Furthermore, it refines the generation of the fixture approach vector and the anti-collision adjustment process, optimizes the fixture approach path, and further reduces the risk of collision with surrounding workpieces during the gripping process.

[0058] In one alternative embodiment, refer to Figure 5 and Figure 6The collision detection logic, intersection volume calculation process, and detection area limitation method of the interference assessment module are refined.

[0059] Specifically, after receiving the six-DOF pose of the candidate workpiece, the grasping interference evaluation module retrieves the pre-built geometric model of the robotic arm's end effector. This model is a three-dimensional mesh model containing the fixture body, grippers, and clamping space. The model's base coordinate point set is located in the fixture model coordinate system, with the origin of the model coordinate system coinciding with the fixture's TCP center point, and the Z-axis parallel to the fixture's clamping direction. The grasping interference evaluation module applies the translation vector and rotation matrix from the six-DOF pose to the base coordinate point set of the robotic arm's end effector's geometric model. Through spatial transformation, it converts the base coordinate point set in the fixture model coordinate system to the camera coordinate system, obtaining the vertex coordinates of the fixture's bounding box in three-dimensional space. The fixture bounding box is the smallest hexahedral bounding box enclosing the fixture's geometric model, and its vertex coordinates completely characterize the fixture's position and orientation in three-dimensional space.

[0060] In the first-level collision detection process, the smallest circumscribed cylinder of the fixture bounding box is used as the collision detection body. The central axis of the smallest circumscribed cylinder is parallel to the clamping direction of the fixture, and the clamping direction is consistent with the direction of the fixture's approach vector. The radius of the cylinder is the maximum distance from the vertex of the fixture bounding box to the central axis, and the height of the cylinder is the difference between the maximum and minimum lengths of the fixture bounding box along the central axis, which can completely enclose the fixture bounding box. Local point clouds within a set radius around the candidate pose are extracted. The local point cloud consists of all point cloud data within a set radius around the three-dimensional center coordinates of the candidate workpiece's six-DOF pose. The set radius is pre-set based on the maximum size of the fixture and the size of the workpiece, which can completely cover the surrounding space during the fixture's movement. The number of points in the local point cloud that fall inside the smallest circumscribed cylinder is calculated. During the point cloud fall-in determination process, the three-dimensional coordinates of the point cloud are substituted into the spatial equation of the cylinder. If the equation is satisfied, the point is determined to fall inside the cylinder. When the number of points in the calculated point cloud exceeds the preset safety threshold, the candidate pose is determined to have collision interference, and the candidate pose is directly eliminated without performing the subsequent second-level detection. When the number of points in the point cloud does not exceed the preset safety threshold, the candidate pose is determined to pass the first-level detection, and the subsequent second-level fine detection is performed.

[0061] In the second-level fine inspection process, the vertex coordinates of the fixture bounding box are obtained. Based on these vertex coordinates, a three-dimensional axis-aligned bounding box is constructed. The three sides of the three-dimensional axis-aligned bounding box are parallel to the X, Y, and Z axes of the camera coordinate system, respectively, and can completely enclose the fixture bounding box. The three-dimensional axis-aligned bounding box is divided into multiple cubic voxels of the same size. The side length of each cubic voxel is preset based on the inspection accuracy requirements, and the voxel side length is set to 1 / 10 of the minimum feature size of the target workpiece, thus controlling the computational load while ensuring inspection accuracy.

[0062] Local point clouds are mapped onto the partitioned cubic voxels. Cubic voxels containing at least one point cloud data are counted to generate an occupied voxel set. Each voxel in the occupied voxel set is marked as occupied, and voxels without point cloud data are marked as idle. Fixture bounding boxes are mapped onto the partitioned cubic voxels. Cubic voxels overlapping with fixture bounding boxes are counted to generate a fixture voxel set. Each voxel in the fixture voxel set is marked as fixture occupied.

[0063] Perform a bitwise AND operation on the set of occupied voxels and the set of clamp voxels. Count the total number of voxels in the result that are in the occupied state. Multiply the total number of voxels by the volume of a single cubic voxel to obtain the actual intersection volume between the clamp bounding box and the local point cloud. The formula for calculating the actual intersection volume is: in, The state after a bitwise AND operation is the total number of voxels occupied. For the volume of a single cubic voxel, This represents the calculated actual intersection volume.

[0064] A preset volume threshold for intersecting volumes is established. When the actual intersecting volume exceeds the preset volume threshold, the candidate pose is determined to have collision interference, and the candidate pose is rejected. When the actual intersecting volume does not exceed the preset volume threshold, the candidate pose is determined to be an interference-free pose. The preset volume threshold is based on the pre-set mating clearance between the fixture and the workpiece, allowing for slight clearance errors between the fixture and the workpiece to avoid accidental rejection.

[0065] Furthermore, in the process of mapping the fixture bounding box to a cube voxel to generate a fixture element set, a local coordinate system is constructed with the center of the fixture bounding box as the origin and collinear with the 3D axis-aligned bounding box. The direction vector of the fixture approach vector in the local coordinate system is extracted, and the fixture bounding box is extended by a set distance in the opposite direction of the direction vector to construct a motion sweep bounding box. The extension distance is set as the movement distance of the fixture from the pre-approach point to the clamping position. The motion sweep bounding box can completely cover the entire motion space when the fixture performs the approach action along the fixture approach vector. The overlapping area of ​​the motion sweep bounding box and the 3D axis-aligned bounding box is used as the detection area. The mapping operation of the fixture bounding box is performed on the cube voxels only in the detection area to generate the fixture element set, and the bitwise logical AND operation is performed on the occupied voxel set and the fixture element set only in the detection area, which reduces the search space for collision detection and reduces the amount of computation.

[0066] The interference assessment module sorts all interference-free poses that pass the two-level detection according to the physical stability score of the corresponding point cloud cluster from high to low, selecting the top-ranked interference-free pose as the optimal interference-free pose. Simultaneously, it outputs the final gripper approach vector corresponding to the optimal interference-free pose, and then outputs both the optimal interference-free pose and the gripper approach vector to the robot controller. For all rejected interference poses, the system records their corresponding pose parameters and the cause of interference for subsequent network model optimization and training.

[0067] Table 4. Judgment Rules and Execution Logic for Two-Level Interference Detection Table 4 clarifies the input data, core detection methods, judgment conditions, and corresponding execution actions for the two-level detection of the grasping interference assessment module. The preset safety threshold and preset volume threshold are both preset based on the size of the target workpiece and the specifications of the fixture. Those skilled in the art can adjust the corresponding thresholds according to the accuracy requirements of the actual operation scenario.

[0068] This embodiment refines the entire process of two-level collision detection. The first-level coarse detection enables the rapid removal of obvious interfering poses, while the second-level voxel-based fine detection enables the accurate calculation of the intersection volume, ensuring the efficiency and accuracy of collision detection. By constructing a motion sweep bounding box, the detection area is limited, reducing the search space for collision detection and further reducing the computational load of the interference evaluation process, thus improving the system's operating efficiency.

Claims

1. A human-machine collaborative loading and unloading and intelligent logistics scheduling system based on AI vision, characterized in that, The system includes a line laser 3D scanner, a multi-scale feature extraction network, a physical prior calculation module, a pose regression head, a grasping interference evaluation module, and a robot controller. The line laser 3D scanner is used to acquire 3D point cloud data of disordered stacked workpieces in a basket. The multi-scale feature extraction network receives the 3D point cloud data and outputs multi-scale point cloud features containing geometric structure and local curvature. The physical prior calculation module receives the 3D point cloud data and calculates the projected support area of ​​the point cloud cluster in the gravity direction, the centroid offset, and the angle between the point cloud normal vector and the gravity vector to generate a 3D tensor mask characterizing physical stability. The 3D tensor mask is applied to the multi-scale point cloud features through element-wise multiplication to output fused point cloud features. The pose regression head receives the fused point cloud features and outputs the six-degree-of-freedom pose of the candidate workpiece. The grasping interference evaluation module, combined with a pre-built geometric model of the robotic arm end-effector, calculates the intersection volume of the bounding box of the fixture and the point cloud surrounding the candidate pose in 3D space.

2. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 1, characterized in that, When calculating the projected support area, the physical prior calculation module projects the three-dimensional point cloud data onto a horizontal reference plane along the gravity direction, performs convex hull operations on the points within the projection area, and solves for the area of ​​the convex hull polygon. When calculating the centroid offset, it extracts the point cloud cluster of a single workpiece and calculates the three-dimensional geometric center coordinates of the point cloud cluster, obtains the centroid coordinates of the bottom support point set of the point cloud cluster, and uses the Euclidean distance vector between the three-dimensional geometric center coordinates and the centroid coordinates as the centroid offset. When calculating the included angle, it uses principal component analysis to extract the normal vector of the point cloud cluster, calculates the cosine value between the normal vector and the reverse vector of the gravity direction, and inputs the projected support area, the centroid offset, and the cosine value into a pre-constructed three-layer fully connected layer to generate the three-dimensional tensor mask aligned with the feature space dimension of the multi-scale point cloud.

3. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 1, characterized in that, The multi-scale feature extraction network includes a bottom-level sparse convolutional layer, a middle-level sparse convolutional layer, and a high-level sparse convolutional layer connected in series. The bottom-level sparse convolutional layer performs initial feature mapping on the input 3D point cloud data to extract local edge geometric features. The middle-level sparse convolutional layer receives the local edge geometric features and expands the receptive field to extract the overall structural features of the workpiece surface. The high-level sparse convolutional layer receives the overall structural features and extracts global semantic features containing contextual information. The multi-scale feature extraction network also includes a feature concatenation layer, which concatenates the local edge geometric features, the overall structural features, and the global semantic features in the channel dimension. The concatenated features are then input into a one-dimensional convolutional layer for dimensionality reduction, and the multi-scale point cloud features containing geometric structure and local curvature are output.

4. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 1, characterized in that, After receiving the six-degree-of-freedom pose of the candidate workpiece, the grasping interference evaluation module retrieves the pre-built geometric model of the end effector of the robotic arm. It applies the translation vector and rotation matrix of the six-degree-of-freedom pose to the base coordinate point set of the geometric model of the end effector to obtain the vertex coordinates of the bounding box of the clamp in three-dimensional space. Using the smallest circumscribed cylinder of the bounding box as the collision detection body, it extracts the local point cloud within a set radius around the candidate pose and calculates the number of points in the local point cloud that fall into the smallest circumscribed cylinder. When the number of points exceeds a preset safety threshold, it determines that the candidate pose has collision interference and performs a rejection operation.

5. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 1, characterized in that, The pose regression head includes parallel translation and rotation branches. The translation branch performs 3D bounding box regression on the fused point cloud features to output the 3D center coordinates of the candidate workpiece in the camera coordinate system. The rotation branch performs classification processing on the fused point cloud features to output the Euler angles of the candidate workpiece around the three coordinate axes. The 3D center coordinates and the Euler angles are combined to form the six-degree-of-freedom pose of the candidate workpiece. When generating the fixture approach vector, the 3D center coordinates are used as the starting point, and a set distance is extended along the workpiece surface normal direction determined by the Euler angles to obtain the ending point. The spatial vector from the starting point to the ending point is normalized to generate the fixture approach vector.

6. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 1, characterized in that, Before the 3D tensor mask is applied to the multi-scale point cloud features through element-wise multiplication, a dimension alignment operation is performed on the multi-scale point cloud features and the 3D tensor mask to extract the spatial resolution size of the multi-scale point cloud features. A trilinear interpolation algorithm is used to perform upsampling on the 3D tensor mask so that the spatial resolution size of the upsampled 3D tensor mask is the same as that of the multi-scale point cloud features. A channel dimension copying and expansion operation is performed on the upsampled 3D tensor mask so that the number of channels of the expanded 3D tensor mask is the same as the number of channels of the multi-scale point cloud features. The channel-expanded 3D tensor mask is then multiplied element-wise with the multi-scale point cloud features to output the fused point cloud features.

7. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 2, characterized in that, Before extracting the point cloud cluster of a single workpiece, voxel rasterization downsampling is performed on the 3D point cloud data to filter out redundant spatial points. A region-growing-based segmentation operation is then performed on the downsampled 3D point cloud data, selecting the point with the smallest curvature as the seed point. Neighboring points with normal vector angles less than a set threshold are merged into the same region to generate independent point cloud clusters. Before extracting the normal vectors using principal component analysis, a local neighborhood search sphere is constructed for each point within the point cloud cluster. A covariance matrix is ​​constructed based on the 3D coordinates of neighboring points within the local neighborhood search sphere. Eigenvalue decomposition is performed on the covariance matrix, and the eigenvector corresponding to the smallest eigenvalue is used as the normal vector of the corresponding point within the point cloud cluster.

8. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 4, characterized in that, When the number of point clouds does not exceed the preset safety threshold, the vertex coordinates of the fixture bounding box are obtained to construct a three-dimensional axis-aligned bounding box. The three-dimensional axis-aligned bounding box is divided into multiple cubic voxels. The local point cloud is mapped to the cubic voxels to generate an occupying voxel set. The fixture bounding box is mapped to the cubic voxels to generate a fixture voxel set. A bitwise logical AND operation is performed on the occupying voxel set and the fixture voxel set. The total number of voxels in the operation result that are in the occupying state is counted. The total number of voxels is multiplied by the volume of a single cubic voxel to obtain the actual intersection volume. When the actual intersection volume exceeds the preset volume threshold, it is determined that there is a collision interference and a culling operation is performed.

9. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 5, characterized in that, After generating the clamp approach vector, the length and width dimensions of the 3D bounding box output by the translation branch are obtained. The dimension with the smaller value among the length and width dimensions is taken as the dimension to be avoided. The rotation angle of the Euler angle around the coordinate axis corresponding to the dimension to be avoided is extracted as the initial deflection angle. The clamp approach vector is rotated in the opposite direction around the coordinate axis corresponding to the dimension to be avoided by the initial deflection angle to obtain the spatial adjustment vector. The angle between the spatial adjustment vector and the horizontal plane is calculated. When the angle is less than the set anti-collision elevation angle, the spatial adjustment vector is rotated in the positive direction around the horizontal axis perpendicular to the dimension to be avoided by the difference between the anti-collision elevation angle and the angle, and the final clamp approach vector is output.

10. The AI ​​vision-based human-machine collaborative loading and unloading and intelligent logistics scheduling system according to claim 8, characterized in that, In the process of mapping the fixture bounding box to the cube voxel to generate the fixture voxel set, a local coordinate system is constructed with the center of the fixture bounding box as the origin and collinear with the 3D axis-aligned bounding box. The direction vector of the fixture proximity vector in the local coordinate system is extracted. The fixture bounding box is extended by a set distance in the opposite direction of the direction vector to construct a motion sweep bounding box. The overlapping area of ​​the motion sweep bounding box and the 3D axis-aligned bounding box is taken as the detection area. The mapping operation of the fixture bounding box is performed on the cube voxel only in the detection area to generate the fixture voxel set. The bitwise logical AND operation is performed on the occupied voxel set only in the detection area.

Images