Fresh milk cold chain conveying line multi-mechanical arm cooperative operation control method and system

Abstract

The present application relates to the technical field of data acquisition control, and discloses a kind of fresh milk cold chain conveying line multi-mechanical arm collaborative operation control method and system, wherein, method includes: the RGB image and depth map of fresh milk soft package are synchronously collected;YOLO light weight instance segmentation is formed soft package instance perception data by;Foreground pixel is mapped to three-dimensional space;Extraction collapse geometry feature and distinguish collapse type, form soft package collapse discrimination data;According to the thin film tension poisson compensation model of collapse type, obtain soft package form completion data;Boundary distance weighted fusion is carried out to measured point and compensation point;The grabbing pose data is obtained by support vector regression correction grabbing pose by;According to grabbing pose data, optimize multi-mechanical arm task allocation, and implement feedforward and closed-loop motion control in combination with conveying speed, output multi-mechanical arm collaborative control instruction.The present application can reduce the influence of soft package collapse and point cloud loss on grabbing positioning and multi-arm collaborative beat.

Description

Technical Field

[0001] This invention relates to the field of data acquisition and control technology, and in particular to a method and system for controlling the collaborative operation of multiple robotic arms in a fresh milk cold chain conveyor line. Background Technology

[0002] Currently, in scenarios where multiple robotic arms are deployed collaboratively to grasp soft bags of fresh milk on a cold chain conveyor line, the common practice is to use an RGB-D camera above the conveyor line to simultaneously acquire color images and 3D point cloud data. Instance segmentation or object detection algorithms are then used to locate the region of each soft bag. The depth point cloud of the corresponding region is then extracted, and after preprocessing such as filtering and downsampling, principal component analysis or standard template fitting is used to estimate the six-degree-of-freedom grasping pose of the soft bag based on the assumption of intact geometry. Finally, the pose results are sent to multiple robotic arms for task allocation and synchronized grasping. This processing path can quickly provide relatively stable grasping coordinates under the condition that the soft bag has an intact shape, a flat surface, and a dense point cloud. However, it heavily relies on the premise that the soft bag has a regular and unchanging spatial shape. It lacks an effective perception and feedback correction mechanism for the surface collapse and deformation of the soft bag caused by low-temperature hardening and stacking compression during cold chain conveying. Once non-rigid depressions appear on the actual surface, point cloud voids and shape distortions will be directly transmitted to the geometric estimation stage, resulting in unreliable pose calculation results.

[0003] For example, when bagged fresh milk is stored for extended periods in a cold chain environment of 0-4℃, the milk inside shrinks due to localized freezing or uneven temperature distribution. Under the pressure difference between the inside and outside, the packaging material collapses inward, resulting in a top center collapse. Alternatively, under multi-layer stacking and conveying conditions, the bottom layer of soft packaging bears continuous pressure from the upper layers, leading to corner crushing or continuous collapse in multiple areas. In these situations, the point cloud of the soft packaging captured by the 3D camera shows large areas of missing parts and severe contour distortion in the collapsed areas. Traditional processing algorithms still misjudge the void boundaries as the natural edges of the object, failing to distinguish between the false contours caused by the collapse and the true outer boundaries of the soft packaging. This results in a significant shift of the fitted geometric center towards the non-collapsed side, and an unacceptable tilt in the principal axis direction. Ultimately, the calculated gripping pose drifts by several millimeters or even tens of millimeters from the actual gripping position. This deviation directly causes end-effector suction cup misalignment, gripper slippage, or even empty gripping, resulting in frequent production line downtime and product loss, severely weakening the operating cycle and processing capacity of the automated cold chain conveying system.

[0004] Therefore, there is an urgent need for a method that can still perceive the collapse state in real time and dynamically compensate for its true three-dimensional geometric features even when there are point cloud voids and contour deformations on the surface of the soft package. By integrating collapse type discrimination with physical model-driven surface restoration, the method can correct the pose calculation deviation caused by collapse and stabilize the final gripping point positioning error within ±1.5mm. This will maintain a high success rate and work cycle of multi-robotic arm collaborative gripping during high-speed operation of the conveyor line, and significantly improve the operational reliability and overall efficiency of the automation of fresh milk cold chain logistics. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the present invention aims to propose a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line. This method addresses the technical problem in the prior art where, when multiple robotic arms collaboratively grasp soft packages of fresh milk in a cold chain conveyor line, especially under conditions where the soft packages collapse due to low temperatures or stacking, resulting in point cloud voids and contour deformation, real-time compensation for the collapse state cannot be achieved, leading to pose estimation errors and grasping failures.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line.

[0007] The method for controlling the collaborative operation of multiple robotic arms in a fresh milk cold chain conveyor line includes: Step S10: Obtain image data of fresh milk soft packages on the fresh milk cold chain conveyor line, and perform soft package instance perception task based on the image data using the YOLO lightweight instance segmentation mechanism to output soft package instance perception data. Step S20: Based on the soft package instance perception data, a point cloud spatial mapping and curvature feature classification mechanism is used to perform the soft package collapse discrimination task and output the soft package collapse discrimination data; Step S30: Based on the soft package collapse discrimination data, the soft package morphology completion task is performed using the thin film tension Poisson compensation mechanism, and the soft package morphology completion data is output. Step S40: Based on the soft package morphology completion data, the grasping pose estimation task is performed using a boundary distance weighted fusion and support vector regression pose correction mechanism, and the grasping pose data is output. Step S50: Based on the grasping pose data, a collaborative task cost optimization mechanism is used to allocate grasping tasks and control motion of multiple robotic arms, and output collaborative control commands for multiple robotic arms.

[0008] Preferably, step S10, which involves acquiring image data of fresh milk pouches on the fresh milk cold chain conveyor line, performing a pouch instance perception task based on the image data using the YOLO lightweight instance segmentation mechanism, and outputting pouch instance perception data, specifically includes: Step S101: Using multiple RGB-D cameras arranged above the fresh milk cold chain conveyor line, synchronous image acquisition is performed on the soft bags of fresh milk passing through the fresh milk cold chain conveyor line to obtain the image data including RGB images and depth maps. Step S102: Input the RGB image into the pre-trained YOLO lightweight detection network, extract multi-scale image features through the backbone network of the YOLO lightweight detection network, detect the bounding boxes of each fresh milk soft pack according to the multi-scale image features, and input the multi-scale image features and each bounding box into the lightweight mask decoder to generate a two-dimensional instance segmentation mask for each fresh milk soft pack. Step S103: The lightweight mask decoder is pre-trained using edge-aware contrastive loss, which is determined by accumulating the squared differences between the spatial gradients in the neighborhood of the predicted mask boundary and the spatial gradients in the neighborhood of the real mask boundary; each of the two-dimensional instance segmentation masks and the corresponding depth maps are pixel-aligned, and the pixel-aligned two-dimensional instance segmentation masks and depth maps are encapsulated into the soft-pack instance-aware data.

[0009] Preferably, step S20, which involves performing a soft-pack collapse discrimination task based on the soft-pack instance perception data using a point cloud spatial mapping and curvature feature classification mechanism, and outputting soft-pack collapse discrimination data, specifically includes: Step S201: Extract the two-dimensional instance segmentation mask of each fresh milk soft package and the depth map aligned with the pixels of the two-dimensional instance segmentation mask from the soft package instance perception data, and obtain the pre-calibrated intrinsic parameter matrix of each RGB-D camera; based on the pre-calibrated intrinsic parameter matrix and the depth value of each foreground pixel in the depth map, map the foreground pixels in the two-dimensional instance segmentation mask to three-dimensional space to obtain the fresh milk soft package point cloud subset corresponding to each fresh milk soft package; Step S202: The fresh milk soft bun point cloud subset is sequentially subjected to voxel downsampling and statistical outlier filtering. The best fitting plane of the fresh milk soft bun point cloud subset is determined by the random sampling consistency plane fitting method. The filtered fresh milk soft bun point cloud subset is projected onto the best fitting plane, and the outer contour of the soft bun and the collapsed boundary located within the outer contour of the soft bun are extracted by the Alpha Shapes algorithm. The ratio between the projected area of ​​the region surrounded by the collapsed boundary and the projected area of ​​the region surrounded by the outer contour of the soft bun is determined as the collapse rate. The mean of the maximum principal curvature, the variance of the Gaussian curvature and the normal scattering angle in the neighborhood of the collapsed boundary are extracted to obtain the collapsed geometric features. Step S203: Input the collapse rate and the collapse geometric features into a pre-trained fully connected neural network classifier to obtain the collapse type; divide the soft milk bun point cloud subset into collapsed region point clouds and original non-collapse point clouds according to the collapse boundary, and encapsulate the soft milk bun point cloud subset, the collapse boundary, the collapse type and the original non-collapse point cloud into the soft bun collapse discrimination data, wherein the collapse type includes top center depression, corner compression collapse and multi-region continuous collapse.

[0010] Preferably, step S202, the step of extracting the normal scattering angle within the neighborhood of the collapse boundary, specifically includes: Taking each boundary point on the collapsed boundary as the center, select neighboring points within a preset neighborhood radius from the cloud subset of fresh milk soft buns; Local tangent plane fitting is performed on the neighborhood points corresponding to each boundary point to obtain the local normal vector corresponding to each boundary point; Calculate the angle between each local normal vector and the reference normal vector of the best-fit plane to obtain the normal deflection angle corresponding to each boundary point; Calculate the standard deviation of each of the said normal deflection angles, and determine the standard deviation as the normal scattering angle.

[0011] Preferably, step S30, which involves using a thin-film tension Poisson compensation mechanism to perform soft package morphology completion based on the soft package collapse discrimination data and outputting soft package morphology completion data, specifically includes: Step S301: Extract the collapse boundary, the collapse type, and the original non-collapse point cloud from the soft-pack collapse discrimination data; project the original non-collapse point cloud onto the best-fit plane; and determine the area enclosed by the collapse boundary on the best-fit plane as the collapse projection region; equip the surface of the fresh milk soft-pack with a liquid-filled film tension shell, and use the measured surface height at the collapse boundary as a boundary constraint to establish a Poisson equation for recovering the soft-pack surface within the collapse projection region:

[0012] The boundary conditions for the Poisson equation are:

[0013] in, The coordinates of the collapsed projection area The height of the compensated curved surface at that location, The film tension coefficient, The equivalent areal density of the liquid inside the soft milk package within the collapsed projection area. It is the acceleration due to gravity. The coordinate position of the liquid inside the fresh milk soft pack The distributed pressure applied to the surface of the fresh milk soft package, The collapsed projection area, The boundary of the collapsed projection area. The coordinates of the collapsed boundary are... The corresponding measured surface height; Step S302: Obtain the pre-measured low-temperature tensile stiffness per unit width of the fresh milk soft packaging material, and set the initial value of the film tension coefficient according to the collapse type; when the collapse type is a top center depression, set the initial value of the film tension coefficient to 0.8 times the low-temperature tensile stiffness per unit width; when the collapse type is corner compression collapse, set the initial value of the film tension coefficient to the sum of 0.6 times the low-temperature tensile stiffness per unit width and the tangential friction tension correction amount, the tangential friction tension correction amount being determined based on the tangential friction force at the collapse boundary; when the collapse type is multi-region continuous collapse, set the initial value of the film tension coefficient corresponding to each continuous collapse region according to the preset angle interval where the normal dispersion angle corresponding to each continuous collapse region is located, and apply a continuous surface height constraint at the common boundary of adjacent continuous collapse regions; Step S303: Discretize the Poisson equation using a five-point difference scheme, and solve the sparse linear equation system formed after discretization using the conjugate gradient method to obtain discrete grid points of the compensation surface; according to the coordinate transformation relationship between the best-fit plane and the corresponding RGB-D camera coordinate system, map the discrete grid points to the corresponding RGB-D camera coordinate system to obtain the compensation point set, and encapsulate the compensation point set, the original non-collapsed point cloud, and the collapsed boundary into the soft-pack morphological completion data.

[0014] Preferably, in step S40, the step of performing the grasping pose estimation task based on the soft package morphology completion data and using a boundary distance weighted fusion and support vector regression pose correction mechanism to output the grasping pose data specifically includes: Step S401: Extract the compensation point set, the original non-collapsed point cloud, and the collapse boundary from the soft-pack morphological completion data. Within a preset neighborhood of the collapse boundary, pair the original points in the original non-collapsed point cloud with the nearest neighbor compensation points in the compensation point set point by point. Determine the shortest distance from each paired point to the collapse boundary. Based on the shortest distance and a preset smoothing factor, determine the exponential decay weight corresponding to the original point and the complementary weight corresponding to the nearest neighbor compensation point, respectively. Then, perform weighted fusion on each paired point based on the exponential decay weight and the complementary weight to obtain a fused point. Combine the fused point, the original points located outside the preset neighborhood, and the compensation points that did not participate in the pairing to obtain a complete point cloud. Step S402: Select a preset number of nearest neighbor points corresponding to each point from the complete point cloud. Estimate the local curvature of each point based on each point and its corresponding preset number of nearest neighbor points. Determine the reciprocal of the local curvature as the principal component analysis weight of the corresponding point. Perform weighted principal component analysis on the complete point cloud based on the principal component analysis weight. Determine the eigenvector corresponding to the largest eigenvalue as the length direction of the milk soft pack. Determine the eigenvector corresponding to the smallest eigenvalue as the normal direction of the surface of the milk soft pack. Establish a grasping reference coordinate system based on the length direction and the normal direction to obtain the initial value of the six-degree-of-freedom grasping pose. Step S403: Extract the local point cloud normal histogram of the grasping region corresponding to the initial six-DOF grasping pose value; determine the point cloud completion ratio based on the proportion of compensated points in the complete point cloud; and determine the surface fitting residual based on the distance deviation between the complete point cloud and the preset soft-packaged surface model; normalize the point cloud completion ratio and the surface fitting residual respectively; determine the normalized point cloud completion ratio as the first confidence component; and determine the value obtained by subtracting the normalized surface fitting residual from the first confidence component as the second confidence component, and then proceed according to the preset confidence... The first confidence component and the second confidence component are weighted and summed to obtain the point cloud completion confidence. The initial value of the six-degree-of-freedom grasping pose, the local point cloud normal histogram, and the point cloud completion confidence are input into a support vector regression model pre-trained based on historical grasping data to obtain translation adjustment and rotation correction. The initial value of the six-degree-of-freedom grasping pose is corrected according to the translation adjustment and rotation correction to obtain the final grasping pose. The final grasping pose and the point cloud completion confidence are encapsulated as the grasping pose data.

[0015] Preferably, step S50, which involves allocating and controlling the grasping tasks of multiple robotic arms using a collaborative task cost optimization mechanism based on the grasping pose data, and outputting collaborative control commands for the multiple robotic arms, specifically includes: Step S501: Extract the final grasping pose and point cloud completion confidence level of each fresh milk soft bag to be grasped from the grasping pose data; collect the current end position, current joint angle vector, and current load of each idle robotic arm; determine the sum of the ratios between the current load and the rated load of the corresponding idle robotic arm as the dimensionless load coefficient; and construct the cooperative task cost function:

[0016] in, The total cost of the collaborative task. Let be the set of indices for the soft milk packages to be grabbed. The set of indices for the idle robotic arms. The index of the soft milk package to be grabbed. For the index of the idle robotic arm, To indicate the first Have the uncollected fresh milk pouches been assigned to the first...? Binary decision variables for an idle robotic arm. For the first The current position of the end effector of the idle robotic arm is compared with that of the first idle robotic arm. The ratio between the actual Cartesian distance between the final gripping positions of each fresh milk pouch to the preset maximum working distance. For the first The normalized current joint angle vector of an idle robotic arm. For the first The idle robotic arm reaches the first Each soft bag of fresh milk to be grasped corresponds to the normalized target joint angle vector at the final grasping pose. Let L be the square of the L2 norm of the vector. For the first The reliability of the point cloud completion corresponding to the fresh milk soft package to be captured, and , For the first The dimensionless load coefficient of each idle robotic arm, and , Assuming a preset distance cost weight, To preset the joint movement cost weights, Preset load confidence cost weights; Step S502: With the goal of minimizing the total cost of the collaborative task, the Hungarian algorithm is used to solve the binary decision variables, and the binary decision variables are subject to the constraint that each fresh milk soft bag to be grabbed can be assigned to at most one idle robotic arm and each idle robotic arm can be assigned to at most one fresh milk soft bag to be grabbed in the current grabbing cycle, so as to obtain the optimal grabbing task allocation result. Step S503: Collect the conveying speed output by the encoder pre-set on the fresh milk cold chain conveyor line; for each idle robotic arm in the optimal grasping task allocation result, calculate the end-effector velocity feedforward based on the conveying speed, and calculate the position closed-loop correction based on the position error between the current end position of the corresponding idle robotic arm and the corresponding final grasping pose; superimpose the end-effector velocity feedforward and the position closed-loop correction, calculate the corresponding joint speed command through the inverse velocity kinematics of the robotic arm, and generate the multi-robotic arm cooperative control command based on the joint speed command and the optimal grasping task allocation result.

[0017] This invention also provides a multi-robotic arm collaborative operation control system for a fresh milk cold chain conveyor line, comprising: The visual perception module is used to acquire image data of fresh milk soft packages on the fresh milk cold chain conveyor line, and to perform soft package instance perception tasks based on the image data using the YOLO lightweight instance segmentation mechanism, and output soft package instance perception data. The collapse discrimination module is used to perform the soft package collapse discrimination task based on the soft package instance perception data, using a point cloud spatial mapping and curvature feature classification mechanism, and output soft package collapse discrimination data; The morphological completion module is used to perform the soft package morphological completion task based on the soft package collapse discrimination data and adopt the film tension Poisson compensation mechanism to output the soft package morphological completion data. The pose estimation module is used to complete the data based on the soft package shape, and to perform the grasping pose estimation task by using boundary distance weighted fusion and support vector regression pose correction mechanism, and output grasping pose data. The collaborative control module is used to allocate and control the grasping tasks of multiple robotic arms based on the grasping pose data and adopt a collaborative task cost optimization mechanism, and output collaborative control commands for multiple robotic arms.

[0018] The present invention also provides a multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line. The multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line includes: a memory, a processor, and a multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line stored in the memory and executable on the processor. When the multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line is executed by the processor, the above-mentioned method is implemented.

[0019] The present invention also provides a computer program product, which includes a multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line. When the multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line is executed by a processor, it implements the above-mentioned method.

[0020] The beneficial effects of this invention are as follows: 1. This invention utilizes lightweight instance segmentation of fresh milk soft packaging image data, combined with point cloud spatial mapping, curvature feature extraction, and collapse type classification, to ensure that the instance boundaries, spatial morphology, and local collapse features of the soft packaging correspond within the same data chain. This reduces interference from adjacent soft packaging occlusion, packaging surface wrinkles, and conveyor line background on collapse judgment, providing well-defined and type-distinguished collapse judgment data for subsequent morphological completion and grasping pose calculation.

[0021] 2. This invention sets film tension conditions based on the collapse type and performs Poisson compensation. Then, it connects the compensated area with the original non-collapsed area through boundary distance weighted fusion, ensuring the reconstructed flexible packaging surface remains continuous at the collapse boundary, avoiding abrupt contour changes caused by direct filling. Furthermore, combining integrity confidence and support vector regression for pose correction reduces the deviations caused by collapse deformation in calculating the packaging length direction, surface normal, and gripping position, thereby reducing situations where the robotic arm misaligns, grips instably, or crushes the packaging.

[0022] 3. This invention incorporates the working distance between the robotic arm and the soft packaging to be grasped, joint adjustment amount, morphological integrity confidence level, and the current load of the robotic arm into the collaborative task cost, and completes the task allocation for multiple robotic arms accordingly. This allows the grasping task to be dynamically adjusted according to the packaging position, packaging shape, and robotic arm operating status. By combining end-effector velocity feedforward, position closed-loop correction, and inverse velocity kinematic control, the task allocation result can be further converted into continuous joint motion commands. This reduces the problems of multiple robotic arms competing for the same target, mutual interference of motion paths, and continuous overload of local robotic arms, thereby improving the continuity, balance, and control reliability of multi-robotic arm collaborative operation on the fresh milk cold chain conveyor line. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the first embodiment of a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line according to the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line according to the present invention.

[0027] In the first embodiment, the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line includes: Step S10: Obtain image data of fresh milk soft packages on the fresh milk cold chain conveyor line, and perform soft package instance perception task based on the image data using the YOLO lightweight instance segmentation mechanism to output soft package instance perception data. The image data is a combination of data collected simultaneously by multiple RGB-D cameras from the fresh milk pouches passing through the gripping area on the fresh milk cold chain conveyor line. It includes RGB images and depth maps at the same acquisition time. Each camera undergoes pre-calibration of its intrinsic parameters and pixel registration of the RGB and depth channels. During acquisition, camera identification, frame time, and image resolution are preserved, ensuring that a pouch instance in an RGB image corresponds to the depth value of the same camera and the same frame. Multiple cameras cover different lateral positions or overlapping observation areas on the conveyor line, and image acquisition maintains the same control cycle to prevent contour and depth edge shifts caused by time misalignment between RGB and depth frames during the conveying process.

[0028] YOLO's lightweight instance segmentation mechanism first inputs an RGB image into a pre-trained lightweight detection network. The backbone network extracts image features at different spatial scales. These multi-scale features preserve not only the overall package outline and orientation but also local variations such as folds and concave edges. The detection part outputs bounding boxes for each milk soft pack based on the multi-scale image features. The lightweight mask decoder simultaneously receives the multi-scale image features and the corresponding bounding boxes, recovering the pixel-level main region within the instance defined by each bounding box, forming a two-dimensional instance segmentation mask for each milk soft pack. When the same frame contains multiple adjacent soft packs, each bounding box and mask retains an independent instance number, preventing the merging of contacting soft packs into a single foreground region.

[0029] The lightweight mask decoder is pre-trained using an edge-aware contrast loss. The calculation order of this loss is as follows: first, the predicted mask boundary neighborhood and the real mask boundary neighborhood are determined separately; then, the spatial gradients of the predicted mask and the real mask are calculated within the corresponding neighborhoods; finally, the gradient differences at the same pixel location are squared and accumulated. During training, this accumulated result is used to constrain the mask decoder, ensuring that the position and direction of changes in the predicted contour closely resemble manually annotated contours, rather than solely focusing on whether the mask covers the soft packaging. The edge loss is concentrated near the packaging contour, reducing edge expansion and contraction that can occur due to low-temperature frost, surface reflection, or conveyor belt color closely resembling the packaging.

[0030] After each 2D instance segmentation mask is generated, it is pixel-aligned with the corresponding depth map. If the RGB image and the depth map have different resolutions, the foreground pixels of the mask are transformed to depth map coordinates according to the calibration mapping of the RGB-D camera; pixels that are outside the effective depth range or for which no effective depth is returned only retain their mask positions, without generating false zero-depth points. The pixel-aligned 2D instance segmentation mask and depth map are encapsulated into soft-pack instance-aware data according to the camera identifier, frame time, and instance number. This output not only gives the region where the soft-pack is located in the RGB image, but also provides the depth pixels and instance attribution that can be directly back-projected for step S20.

[0031] The soft-pack instance perception data pre-classifies the entire field of vision data into individual soft-pack layers. If collapses are analyzed directly in the complete depth map, gaps in the conveyor belt, guide structures, and the end effector of the robotic arm may form areas similar to holes in the point cloud; the rectangular detection box will also mix in adjacent soft packs and background points. The two-dimensional instance segmentation mask defines the foreground according to the actual outer contour of the packaging material. Step S20 can extract the point cloud instance by instance and continuously associate the collapse type, compensation point set, and grasping pose along the instance number. This processing reduces the interference of the entire field of vision background on point cloud classification and also ensures that when multiple soft packs enter the field of vision in parallel, their respective depth map regions will not be used interchangeably.

[0032] When multiple cameras have overlapping views, this step avoids forcibly merging the masks from different cameras in the 2D stage. Instead, it preserves the soft-pack instance perception data and calibration information generated by each camera. Subsequently, more complete data from the observation angle can be selected based on the transport line coordinates and instance location, or multi-view point clouds of the same soft pack can be processed in a unified coordinate system. This avoids back-projecting the depth map of one camera onto the intrinsic parameters of another. For instances where motion blur leads to insufficient detection confidence, they can be excluded from the current control cycle and re-perceived in the next frame, instead of continuing to generate erroneous 3D boundaries with incomplete masks.

[0033] For example, two bags of fresh milk are moving side by side. The top of the right bag is dented due to pressure, and frost forms on its surface, reducing its contrast with the light-colored conveyor belt. The YOLO lightweight detection network still distinguishes the two bags from multi-scale features and outputs bounding boxes for each. Under edge-aware training constraints, the mask decoder forms a closed mask along the outer perimeter of the right bag and retains the dented projection area inside the mask. After pixel alignment of this mask with the depth map of the same frame, it is encapsulated along with the instance number into bag instance-aware data. When reading this data in step S20, only the depth values ​​of the foreground pixels of the right bag are processed; the contact area of ​​adjacent bags and the depth points of the conveyor belt are not mistakenly classified into the same bag point cloud.

[0034] Step S20: Based on the soft package instance perception data, a point cloud spatial mapping and curvature feature classification mechanism is used to perform the soft package collapse discrimination task and output the soft package collapse discrimination data; Point cloud spatial mapping extracts a 2D instance segmentation mask and a depth map aligned with the pixels of each fresh milk soft-pack from the soft-pack instance perception data, while simultaneously reading the pre-calibrated intrinsic parameter matrix of the RGB-D camera that generated the data. For each foreground pixel in the mask, its horizontal coordinates, vertical coordinates, and corresponding depth value are obtained. After expanding the pixel coordinates to homogeneous coordinates, the normalized imaging ray is recovered through the inverse transformation of the intrinsic parameter matrix, and then scale recovery is performed using the depth value to obtain the 3D point in the camera coordinate system. After mapping, the effective foreground pixels of the same instance form a subset of the fresh milk soft-pack point cloud; pixels with invalid depth do not generate 3D points.

[0035] A subset of the point cloud of fresh milk soft-pack packaging was subjected to voxel downsampling and statistical outlier filtering. Voxel downsampling retained representative points within a fixed spatial grid, controlling the difference in point density at different distances. Statistical outlier filtering removed scattered points that significantly deviated from the packaging surface based on the average distance of each point to its neighbors. Subsequently, a random sampling consensus plane fitting method was used to repeatedly select sample planes from the filtered point cloud and statistically analyze the interior points to obtain the best-fit plane with sufficient interior point support. This plane serves as a unified reference for the projected area of ​​the soft-pack, the geometry of the collapse boundary, and the normal deflection angle, avoiding direct comparison of soft-packs with different tilt postures using the camera imaging plane.

[0036] After projecting the filtered subset of the milk pouch point cloud onto the best-fit plane, the Alpha Shapes algorithm is used to extract the outer contour of the pouch and the collapsed boundaries located within the outer contour. The algorithm scale should be adapted to the camera's spatial resolution and the minimum collapsed width to be identified, avoiding both misinterpreting normal sampling gaps as collapsed areas and crossing true concave boundaries. The collapsed rate is obtained by dividing the projected area of ​​the region enclosed by the collapsed boundary by the projected area of ​​the region enclosed by the outer contour of the pouch. This ratio eliminates the influence of the pouch's distance in the image and the projection scale, allowing for comparison of the collapsed proportion between pouches of the same size at different transport locations.

[0037] The geometric features of the collapse include the mean of the maximum principal curvature, the variance of the Gaussian curvature, and the normal scattering angle within the neighborhood of the collapse boundary. The normal scattering angle is obtained according to the process defined in the claims: For each collapse boundary point, select neighborhood points within a predetermined radius; fit a local tangent plane to these neighborhood points to obtain the local normal vector of that boundary point; then calculate the angle between the local normal vector and the reference normal vector of the best-fit plane to form the normal deflection angle of each boundary point; finally, calculate the standard deviation of these normal deflection angles and use it as the normal scattering angle. A larger standard deviation indicates a more dispersed change in orientation at different boundary locations, reflecting the degree of distortion at the compressed corners or in continuous collapses across multiple regions.

[0038] After normalizing the collapse rate, mean maximum principal curvature, Gaussian curvature variance, and normal scattering angle according to the dimensions of the training phase, the data are input into a pre-trained fully connected neural network classifier to obtain data on top center concavity, corner compression collapse, or multi-region continuous collapse. Subsequently, based on the collapse boundaries, the milk carton point cloud subset is divided into collapsed region point clouds and the original non-collapsed point cloud. The milk carton point cloud subset, collapse boundaries, collapse types, and the original non-collapsed point cloud are collectively encapsulated into soft-carton collapse discrimination data. This output allows step S30 to both read the collapse type and boundaries required by the physical model and retain the original non-collapsed point cloud required for fusion in step S40.

[0039] Point cloud integrity or collapse area alone cannot reliably distinguish the causes of collapse. Top-center depressions typically exhibit symmetrical projections and concentrated boundary normal deflections, while corner compression collapses often show high curvature on one side and significant normal dispersion. Multi-regional continuous collapses display inconsistent curvature across multiple adjacent regions. General interpolation methods, without prior classification, may incorrectly construct symmetrical surfaces for compressed corners or apply the same tension parameter to multiple continuous depressions. This step converts projected area, curvature, and normal variations into distinct collapse types, while preserving the best-fit plane and the original non-collapsed point cloud, providing a data foundation for applying differentiated initial tension values ​​and continuous constraints at common boundaries in the next step.

[0040] For example, a soft package's lower right corner is subjected to lateral compression, resulting in an irregular elliptical gap in the filtered point cloud at that location. Random sampling consistency fitting yields the best-fit plane for the package's surface, and the Alpha Shapes algorithm extracts the outer contour and closed collapse boundary at a 2.0 mm scale. The projected area of ​​the collapsed region accounts for approximately 21% of the projected area of ​​the outer contour, with a mean maximum principal curvature of approximately 0.33 per millimeter, a Gaussian curvature variance of approximately 0.038, and a standard deviation of the normal deflection angle of approximately 26 degrees. Based on this, the classifier outputs the corner compression collapse and saves the effective surface outside the collapse boundary as the original non-collapsed point cloud, ultimately forming the soft package collapse discrimination data for this instance.

[0041] Step S30: Based on the soft package collapse discrimination data, the soft package morphology completion task is performed using the thin film tension Poisson compensation mechanism, and the soft package morphology completion data is output. The film tension Poisson compensation mechanism extracts the collapse boundary, collapse type, and original non-collapse point cloud from the soft-pack collapse discrimination data. First, the original non-collapse point cloud is projected onto the best-fit plane determined in step S20. Then, the area enclosed by the collapse boundary on this plane is defined as the collapse projection region. The surface of the fresh milk soft-pack is equivalent to a liquid-filled film tension shell. The height of the compensation surface in the planar coordinate system is used as the unknown quantity, and the measured surface height at the collapse boundary is used as a fixed boundary constraint, ensuring that the restored surface continuously enters the interior of the collapse from the edge of the true point cloud.

[0042] The Poisson equation can be expressed in words according to the physical quantity relationships: At each coordinate position in the collapsed projection region, the product of the film tension coefficient and the two-dimensional Laplace quantity of the compensation surface is added to the product of the equivalent surface density of the liquid inside the soft-pack milk and the acceleration due to gravity, and this result is equal to the distributed pressure exerted by the internal liquid on the surface of the soft-pack at that position. The boundary condition requires that the height of the compensation surface on the boundary of the collapsed projection region is equal to the measured surface height corresponding to the collapsed boundary. Among them, the equivalent surface density describes the internal liquid effect per unit area of ​​the projected region and is not used interchangeably with ordinary volume density; the distributed pressure with coordinate position represents the effect of the liquid on the surface of the packaging material.

[0043] The initial value of the film tension coefficient is set based on the pre-determined low-temperature tensile stiffness per unit width of the packaging material and the collapse type. For top center depression, 80% of the low-temperature tensile stiffness is used; for corner compression collapse, 60% of the low-temperature tensile stiffness is used, plus a tangential frictional tension correction determined based on the tangential frictional force at the collapse boundary. For multi-region continuous collapse, corresponding initial tension values ​​are set according to the preset angle range where the normal dispersion angle of each continuous collapse region is located, and a continuous curvature height constraint is applied to the common boundary of adjacent continuous collapse regions. The low-temperature tensile stiffness is derived from the unit width tensile test of packaging materials of the same specification at cold chain operating temperatures, ensuring that the parameters correspond to the actual packaging condition.

[0044] In the numerical solution, the collapsed projection region is divided into a regular grid. A five-point difference scheme is used, and the height relationship between the current grid point and its upper, lower, left, and right adjacent points is used to approximate the two-dimensional Laplace quantity, resulting in a sparse linear equation system. The conjugate gradient method iteratively solves for the internal grid height starting from the initial tension value and boundary height, stopping when a preset residual condition is reached. If the local boundary depth is missing, it can be filled by interpolation using the effective heights of adjacent closed boundaries, instead of setting the missing height to zero. For continuous collapses in multiple regions, the same surface height is used at the common boundary to prevent steps from forming between adjacent compensation surfaces.

[0045] After obtaining the discrete grid points, they are mapped to 3D space according to the coordinate transformation relationship between the best-fit plane and the corresponding RGB-D camera coordinate system to obtain the compensation point set. The compensation point set, the original non-collapsed point cloud, and the collapsed boundary are collectively encapsulated into soft-pack morphological completion data. The original non-collapsed point cloud is not replaced by the compensation points, and the collapsed boundary is also retained. Therefore, step S40 can distinguish between the camera's measured points and the points generated by the physical model, and establish fusion weights based on the boundary distance. This data encapsulation makes it easier to track the compensation range and its impact on the final capture pose than simply outputting a single hybrid point cloud.

[0046] Moving least squares or minimum curvature completion tends to produce geometrically smooth surfaces, but fails to account for cryogenic packaging tension, internal liquid effects, and corner friction. Corner compression, if compensated into symmetrical concave surfaces, alters the length direction and surface normal of the soft packaging; multiple consecutive collapses, if solved independently without constraining a common boundary, may also result in discontinuous stitching. Thin-film tension Poisson compensation incorporates measured boundaries, cryogenic stiffness, and collapse type into the solution conditions, ensuring the compensated point set conforms to the original point cloud at the boundaries and recovers inward according to the corresponding stress patterns, providing a more complete surface for subsequent weighted principal component analysis.

[0047] In the aforementioned corner-compression soft packaging, the low-temperature tensile stiffness per unit width of the packaging material was obtained from a tensile test of this specification of packaging at 4 degrees Celsius. The initial tension value was taken as 60% of this value, and a correction amount determined based on the boundary tangential friction was added. Approximately one hundred effective boundary heights were used as fixed constraints. The collapsed projection area was divided into a grid of 200 rows and 200 columns. After the conjugate gradient iteration reached the preset residual condition, approximately two thousand discrete grid points were obtained. After the grid points were mapped back to the camera coordinate system, a compensation point set was formed. This set, along with the original non-collapsed point cloud and the collapsed boundary, was written into the soft packaging morphology completion data. The curved surface continuously rose from the compressed boundary inward without excessive corner bulging.

[0048] Step S40: Based on the soft package morphology completion data, the grasping pose estimation task is performed using a boundary distance weighted fusion and support vector regression pose correction mechanism, and the grasping pose data is output. Boundary distance-weighted fusion extracts compensation point sets, original non-collapsed point clouds, and collapse boundaries from the soft-pack morphological completion data. Only within a preset neighborhood of the collapse boundary, a compensation point is selected for each original point through nearest neighbor search, forming a pair; original points outside the neighborhood are not paired. The shortest distance to the collapse boundary is calculated for each pair of points. The weight of the original point is determined by this distance and a smoothing factor, resulting in an exponentially decaying weight, while the weight of the compensation point is its complementary weight. The coordinates of the two points are multiplied by their respective weights, summed, and normalized according to the total weight to obtain the fused point.

[0049] At the boundary, the shortest distance is close to zero, resulting in a larger weight for the original measured points and a smaller weight for the compensated points. As the point moves inward into the collapse, the weight of the original points gradually decreases, while the contribution of the compensated points increases accordingly. A preset smoothing factor controls the weight transition speed and can be calibrated based on the curvature scale of the soft-pack surface and the camera point spacing. After pairing and fusion, the fused points, the original points outside the preset neighborhood, and the compensated points that did not participate in pairing are combined into a complete point cloud. This combination rule neither duplicates already fused original points nor discards compensated points inside the collapse that do not have corresponding original points.

[0050] The initial pose for grasping is determined using weighted principal component analysis. For each point in the complete point cloud, a predetermined number of nearest neighbors are selected. The local curvature is estimated based on this point and its neighbors, and the reciprocal of the local curvature is used as the weight for principal component analysis. Regions with smaller curvature and more stable surfaces receive larger weights, while the influence of local creases or abrupt changes is relatively reduced. Based on these weights, the weighted covariance relationship of the complete point cloud is calculated and eigenvalue decomposition is performed. The eigenvector corresponding to the largest eigenvalue is used as the length direction of the milk carton, and the eigenvector corresponding to the smallest eigenvalue is used as the surface normal direction of the carton. These two eigenvectors together establish the grasping reference coordinate system and form the initial six-degree-of-freedom grasping pose.

[0051] The reliability of point cloud completion is determined by both the point cloud completion ratio and the surface fitting residual. First, the proportion of compensated points in the complete point cloud is calculated, and the distance deviation between the complete point cloud and the preset soft-package surface model is calculated as the surface fitting residual. Both are then normalized. The normalized completion ratio is used as the first reliability component, and the value obtained by subtracting the normalized fitting residual from one is used as the second reliability component. Finally, these components are weighted and summed according to preset reliability weights. This definition allows the reliability to simultaneously reflect the dependence of the current point cloud on the compensated data and the degree of fit between the completed surface and the normal soft-package model. The resulting values ​​can be compared with different soft-package instances.

[0052] The initial six-DOF grasping pose value corresponds to the local point cloud normal histogram of the grasping region. This histogram, along with the point cloud completion confidence score and the initial pose value, is input into a support vector regression model pre-trained based on historical grasping data. The regression model outputs continuous translation and rotation corrections, which correct the position and orientation of the initial pose value, respectively, to obtain the final grasping pose. The final grasping pose and the point cloud completion confidence score are encapsulated as grasping pose data. Unlike simply classifying pose quality, the support vector regression outputs continuous correction values, providing controlled pose adjustments based on the current normal distribution and completion state.

[0053] Equal-weighted principal component analysis allows high-curvature creases and model compensation regions to influence the principal axis direction by a point advantage. Furthermore, when only the pose is output without retaining confidence, task allocation cannot distinguish between complete soft packages and large-area completed soft packages. This step smoothly merges measured and compensated points near the boundary, highlights stable surfaces using the inverse of curvature, and then uses support vector regression to correct residual biases. The completion confidence is output along with the final pose. Therefore, step S50 can both calculate the robot arm distance and target joint angle using the final grasping pose and incorporate the completion confidence of that pose into the load confidence cost.

[0054] For example, the compensation point of the top center concave soft package forms a restored surface on the upper surface. At the collapse boundary, the original points dominate, transitioning towards the fusion result where the compensation points contribute more significantly towards the interior 8 mm position; the complete point cloud does not show any stitching steps. Weighted principal component analysis, weighted by the inverse of local curvature, yields the length direction along the long side of the soft package and the surface normal towards the suction cup, forming the initial value of the grasping pose. Currently, the compensation points account for approximately 24%, and the normalized fitting residual is small, resulting in a point cloud completion confidence score of approximately 0.82. Based on this, the support vector regression model outputs a 0.8 mm translation adjustment towards the center of gravity of the soft package and a small-angle pose correction. Finally, the grasping pose and the confidence score of 0.82 are jointly written into the grasping pose data.

[0055] Step S50: Based on the grasping pose data, a collaborative task cost optimization mechanism is used to allocate grasping tasks and control motion of multiple robotic arms, and output collaborative control commands for multiple robotic arms.

[0056] The collaborative task cost optimization extracts the final grasping pose and point cloud completion reliability for each milk carton to be grasped from the grasping pose data, while simultaneously collecting the current end-effector position, current joint angle vector, and current load of each idle robotic arm. The dimensionless load coefficient of each robotic arm is determined in the order specified in the claims: first, the current load is divided by the rated load, and then one is added to the resulting ratio. The load coefficient of an idle robotic arm is close to one, and the closer the current load is to the rated load, the larger the load coefficient. This normalization method allows robotic arms with different rated loads to be compared in the same cost matrix.

[0057] For the i-th soft package to be grasped and the j-th idle robotic arm, the distance cost is calculated by dividing the actual Cartesian distance between the end effector of the robotic arm and the final grasping position by the preset maximum working distance; the joint motion cost is calculated by taking the square of the L2 norm of the difference between the normalized current joint angle vector and the normalized target joint angle vector; the load confidence cost is calculated by subtracting the confidence level of the soft package's point cloud completion from one, and then multiplying it by the dimensionless load coefficient of the robotic arm. These three terms are multiplied by preset distance cost weights, joint motion cost weights, and load confidence cost weights, respectively, and then summed to form the matching cost between the soft package and the robotic arm; a binary decision variable determines whether this term is included in the total cost of the collaborative task.

[0058] The matching costs of all soft packets to be grasped and idle robotic arms form a cost matrix. The optimization objective is to minimize the total cost of the collaborative task, and the Hungarian algorithm is used to solve the binary decision variables. The solution satisfies two constraints: each soft packet to be grasped is assigned to at most one idle robotic arm, and each idle robotic arm is assigned at most one soft packet in the current grasping cycle. When there are more soft packets to be grasped than idle robotic arms, only the matching portion is output in this cycle, and the remaining soft packets enter the next cycle after the delivery position is updated; when there are more idle robotic arms than soft packets, the unassigned robotic arms remain idle, and no instruction to repeatedly grasp the same soft packet is generated.

[0059] After the optimal grasping task allocation is determined, the conveying speed output by the encoder on the fresh milk cold chain conveyor line is read. For each assigned robotic arm, the end-effector velocity feedforward is calculated based on the conveying speed and direction, and the position closed-loop correction is calculated based on the position error between the current position of the robotic arm's end-effector and the corresponding final grasping pose. The two are superimposed to form the desired end-effector velocity, which is converted into joint speed commands through the inverse velocity kinematics of the robotic arm. The multi-robotic arm cooperative control command consists of the optimal grasping task allocation result and the corresponding joint speed commands, and retains the target soft package instance identifier, enabling each robotic arm to execute its grasping task along the same conveying cycle.

[0060] The three cost categories—distance, joint motion, and load confidence—all use dimensionless or normalized data to avoid the numerical range of millimeter-level distances directly overriding joint angles and confidence levels. The distance term favors robotic arms with shorter strokes, the joint motion term reduces large-scale reconstructions, and the load confidence term prevents targets with lower completion confidence from continuously concentrating on robotic arms with higher loads. Compared to fixed-region allocation or simple proximity principles, this cost can simultaneously utilize the pose and confidence level output from step S40, and global matching avoids multiple robotic arms competing for the same target, coordinating perceptual uncertainty and robotic arm states at the allocation level.

[0061] The target on the conveyor line moves continuously, and the static allocation result must be transformed into dynamic control through velocity feedforward and position closed-loop. When the encoder speed changes, the end-effector velocity feedforward is updated accordingly to compensate for the displacement of the soft package along the conveying direction in advance; the position closed-loop correction is adjusted periodically according to the remaining position error to compensate for deviations caused by camera calibration, robotic arm model, and conveyor speed fluctuations. Inverse velocity kinematics solves for joint velocities based on the current robotic arm configuration, ensuring that the grasping command is not a one-time static pose, but a continuous control quantity synchronized with the current allocation result and conveying motion.

[0062] For example, three robotic arms face four soft packages to be grasped. The point cloud completion confidence levels for packages A to D are 0.96, 0.70, 0.89, and 0.92, respectively. The end effector of robotic arm B is approximately 420 mm from package B, but its current load-to-rated load ratio is high, resulting in a higher dimensionless load coefficient. Robotic arm B is slightly further away but has a lower load. After normalized distance, joint angle difference, and load confidence cost are incorporated into the matrix, the Hungarian algorithm can assign package B to robotic arm B and assign the higher confidence target to robotic arm B. When robotic arm B executes, the encoder provides a conveying speed of 260 mm / s. The controller generates a unidirectional end effector velocity feedforward and superimposes a closed-loop correction corresponding to a 3 mm position error. The joint velocity command is obtained through inverse velocity kinematics, ultimately forming the multi-robotic arm cooperative control command for the current cycle.

[0063] Example 2: Furthermore, the present invention provides a multi-robotic arm collaborative operation control system for a fresh milk cold chain conveyor line, which employs a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in the above embodiments, and can solve the technical problem of multi-robotic arm collaborative operation control for a fresh milk cold chain conveyor line. The beneficial effects of the multi-robotic arm collaborative operation control system for a fresh milk cold chain conveyor line provided by the present invention are the same as those of the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line provided in the above embodiments, and other technical features in the multi-robotic arm collaborative operation control system for a fresh milk cold chain conveyor line are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0064] Example 3: This invention provides a multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line. The device includes at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the processor, which are then executed to enable the processor to perform the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line described in Example 1. The multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line in this invention can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line is merely an example and should not limit the functionality or scope of the invention. A multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can execute various appropriate actions and processes based on a program stored in a read-only memory or a program loaded from a storage device into a random access memory. The random access memory also stores various programs and data required for the operation of the multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line. The processing unit, read-only memory, and random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line to communicate wirelessly or wiredly with other devices to exchange data. Although a multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line with various systems has been described, it should be understood that implementation or possession of all the described systems is not required. It can be implemented alternatively or with more or fewer systems.

[0065] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described above. The computer program product provided by this invention can solve the technical problem of multi-robotic arm collaborative operation control in a fresh milk cold chain conveyor line. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as the beneficial effects of the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line provided in the above embodiments, and will not be repeated here.

[0066] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0067] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0068] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for controlling the collaborative operation of multiple robotic arms in a fresh milk cold chain conveyor line, characterized in that, The method includes: Step S10: Obtain image data of fresh milk soft packages on the fresh milk cold chain conveyor line, and perform soft package instance perception task based on the image data using the YOLO lightweight instance segmentation mechanism to output soft package instance perception data. Step S20: Based on the soft package instance perception data, a point cloud spatial mapping and curvature feature classification mechanism is used to perform the soft package collapse discrimination task and output the soft package collapse discrimination data; Step S30: Based on the soft package collapse discrimination data, the soft package morphology completion task is performed using the thin film tension Poisson compensation mechanism, and the soft package morphology completion data is output. Step S40: Based on the soft package morphology completion data, the grasping pose estimation task is performed using a boundary distance weighted fusion and support vector regression pose correction mechanism, and the grasping pose data is output. Step S50: Based on the grasping pose data, a collaborative task cost optimization mechanism is used to allocate grasping tasks and control motion of multiple robotic arms, and output collaborative control commands for multiple robotic arms.

2. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 1, characterized in that, Step S10 involves acquiring image data of fresh milk pouches from the fresh milk cold chain conveyor line, performing a pouch instance perception task using the YOLO lightweight instance segmentation mechanism based on the image data, and outputting pouch instance perception data. This step specifically includes: Step S101: Using multiple RGB-D cameras arranged above the fresh milk cold chain conveyor line, synchronous image acquisition is performed on the soft bags of fresh milk passing through the fresh milk cold chain conveyor line to obtain the image data including RGB images and depth maps. Step S102: Input the RGB image into the pre-trained YOLO lightweight detection network, extract multi-scale image features through the backbone network of the YOLO lightweight detection network, detect the bounding boxes of each fresh milk soft pack according to the multi-scale image features, and input the multi-scale image features and each bounding box into the lightweight mask decoder to generate a two-dimensional instance segmentation mask for each fresh milk soft pack. Step S103: The lightweight mask decoder is pre-trained using edge-aware contrastive loss, which is determined by accumulating the squared differences between the spatial gradients in the neighborhood of the predicted mask boundary and the spatial gradients in the neighborhood of the real mask boundary; each of the two-dimensional instance segmentation masks and the corresponding depth maps are pixel-aligned, and the pixel-aligned two-dimensional instance segmentation masks and depth maps are encapsulated into the soft-pack instance-aware data.

3. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 2, characterized in that, Step S20, which involves performing a soft-packet collapse discrimination task based on the soft-packet instance perception data using a point cloud spatial mapping and curvature feature classification mechanism, and outputting soft-packet collapse discrimination data, specifically includes: Step S201: Extract the two-dimensional instance segmentation mask of each fresh milk soft package and the depth map aligned with the pixels of the two-dimensional instance segmentation mask from the soft package instance perception data, and obtain the pre-calibrated intrinsic parameter matrix of each RGB-D camera; based on the pre-calibrated intrinsic parameter matrix and the depth value of each foreground pixel in the depth map, map the foreground pixels in the two-dimensional instance segmentation mask to three-dimensional space to obtain the fresh milk soft package point cloud subset corresponding to each fresh milk soft package; Step S202: The fresh milk soft bun point cloud subset is sequentially subjected to voxel downsampling and statistical outlier filtering. The best fitting plane of the fresh milk soft bun point cloud subset is determined by the random sampling consistency plane fitting method. The filtered fresh milk soft bun point cloud subset is projected onto the best fitting plane, and the outer contour of the soft bun and the collapsed boundary located within the outer contour of the soft bun are extracted by the Alpha Shapes algorithm. The ratio between the projected area of ​​the region surrounded by the collapsed boundary and the projected area of ​​the region surrounded by the outer contour of the soft bun is determined as the collapse rate. The mean of the maximum principal curvature, the variance of the Gaussian curvature and the normal scattering angle in the neighborhood of the collapsed boundary are extracted to obtain the collapsed geometric features. Step S203: Input the collapse rate and the collapse geometric features into a pre-trained fully connected neural network classifier to obtain the collapse type; divide the soft milk bun point cloud subset into collapsed region point clouds and original non-collapse point clouds according to the collapse boundary, and encapsulate the soft milk bun point cloud subset, the collapse boundary, the collapse type and the original non-collapse point cloud into the soft bun collapse discrimination data, wherein the collapse type includes top center depression, corner compression collapse and multi-region continuous collapse.

4. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 3, characterized in that, Step S202, the step of extracting the normal scattering angle within the neighborhood of the collapse boundary, specifically includes: Taking each boundary point on the collapsed boundary as the center, select neighboring points within a preset neighborhood radius from the cloud subset of fresh milk soft buns; Local tangent plane fitting is performed on the neighborhood points corresponding to each boundary point to obtain the local normal vector corresponding to each boundary point; Calculate the angle between each local normal vector and the reference normal vector of the best-fit plane to obtain the normal deflection angle corresponding to each boundary point; Calculate the standard deviation of each of the said normal deflection angles, and determine the standard deviation as the normal scattering angle.

5. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 3, characterized in that, Step S30, based on the soft package collapse discrimination data, employs a thin film tension Poisson compensation mechanism to perform the soft package morphology completion task and outputs the soft package morphology completion data, specifically includes: Step S301: Extract the collapse boundary, the collapse type, and the original non-collapse point cloud from the soft-pack collapse discrimination data; project the original non-collapse point cloud onto the best-fit plane; and determine the area enclosed by the collapse boundary on the best-fit plane as the collapse projection region; equip the surface of the fresh milk soft-pack with a liquid-filled film tension shell, and use the measured surface height at the collapse boundary as a boundary constraint to establish a Poisson equation for recovering the soft-pack surface within the collapse projection region: ； The boundary conditions for the Poisson equation are: ； in, The coordinates of the collapsed projection area The height of the compensated curved surface at that location, The film tension coefficient, The equivalent areal density of the liquid inside the soft milk package within the collapsed projection area. It is the acceleration due to gravity. The coordinate position of the liquid inside the fresh milk soft pack The distributed pressure applied to the surface of the fresh milk soft package, The collapsed projection area, The boundary of the collapsed projection area. The coordinates of the collapsed boundary are... The corresponding measured surface height; Step S302: Obtain the pre-measured low-temperature tensile stiffness per unit width of the fresh milk soft packaging material, and set the initial value of the film tension coefficient according to the collapse type; when the collapse type is a top center depression, set the initial value of the film tension coefficient to 0.8 times the low-temperature tensile stiffness per unit width; when the collapse type is corner compression collapse, set the initial value of the film tension coefficient to the sum of 0.6 times the low-temperature tensile stiffness per unit width and the tangential friction tension correction amount, the tangential friction tension correction amount being determined based on the tangential friction force at the collapse boundary; when the collapse type is multi-region continuous collapse, set the initial value of the film tension coefficient corresponding to each continuous collapse region according to the preset angle interval where the normal dispersion angle corresponding to each continuous collapse region is located, and apply a continuous surface height constraint at the common boundary of adjacent continuous collapse regions; Step S303: Discretize the Poisson equation using a five-point difference scheme, and solve the sparse linear equation system formed after discretization using the conjugate gradient method to obtain discrete grid points of the compensation surface; according to the coordinate transformation relationship between the best-fit plane and the corresponding RGB-D camera coordinate system, map the discrete grid points to the corresponding RGB-D camera coordinate system to obtain the compensation point set, and encapsulate the compensation point set, the original non-collapsed point cloud, and the collapsed boundary into the soft-pack morphological completion data.

6. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 5, characterized in that, In step S40, based on the soft-package morphology completion data, the step of performing the grasping pose estimation task using boundary distance weighted fusion and support vector regression pose correction mechanism, and outputting the grasping pose data, specifically includes: Step S401: Extract the compensation point set, the original non-collapsed point cloud, and the collapse boundary from the soft-pack morphological completion data. Within a preset neighborhood of the collapse boundary, pair the original points in the original non-collapsed point cloud with the nearest neighbor compensation points in the compensation point set point by point. Determine the shortest distance from each paired point to the collapse boundary. Based on the shortest distance and a preset smoothing factor, determine the exponential decay weight corresponding to the original point and the complementary weight corresponding to the nearest neighbor compensation point, respectively. Then, perform weighted fusion on each paired point based on the exponential decay weight and the complementary weight to obtain a fused point. Combine the fused point, the original points located outside the preset neighborhood, and the compensation points that did not participate in the pairing to obtain a complete point cloud. Step S402: Select a preset number of nearest neighbor points corresponding to each point from the complete point cloud. Estimate the local curvature of each point based on each point and its corresponding preset number of nearest neighbor points. Determine the reciprocal of the local curvature as the principal component analysis weight of the corresponding point. Perform weighted principal component analysis on the complete point cloud based on the principal component analysis weight. Determine the eigenvector corresponding to the largest eigenvalue as the length direction of the milk soft pack. Determine the eigenvector corresponding to the smallest eigenvalue as the normal direction of the surface of the milk soft pack. Establish a grasping reference coordinate system based on the length direction and the normal direction to obtain the initial value of the six-degree-of-freedom grasping pose. Step S403: Extract the local point cloud normal histogram of the grasping region corresponding to the initial six-DOF grasping pose value; determine the point cloud completion ratio based on the proportion of compensated points in the complete point cloud; and determine the surface fitting residual based on the distance deviation between the complete point cloud and the preset soft-packaged surface model; normalize the point cloud completion ratio and the surface fitting residual respectively; determine the normalized point cloud completion ratio as the first confidence component; and determine the value obtained by subtracting the normalized surface fitting residual from the first confidence component as the second confidence component, and then proceed according to the preset confidence... The first confidence component and the second confidence component are weighted and summed to obtain the point cloud completion confidence. The initial value of the six-degree-of-freedom grasping pose, the local point cloud normal histogram, and the point cloud completion confidence are input into a support vector regression model pre-trained based on historical grasping data to obtain translation adjustment and rotation correction. The initial value of the six-degree-of-freedom grasping pose is corrected according to the translation adjustment and rotation correction to obtain the final grasping pose. The final grasping pose and the point cloud completion confidence are encapsulated as the grasping pose data.

7. The multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in claim 6, characterized in that, Step S50, which involves allocating and controlling the grasping tasks of multiple robotic arms based on the grasping pose data using a collaborative task cost optimization mechanism, and outputting collaborative control commands for the multiple robotic arms, specifically includes: Step S501: Extract the final grasping pose and point cloud completion confidence level of each fresh milk soft bag to be grasped from the grasping pose data; collect the current end position, current joint angle vector, and current load of each idle robotic arm; determine the sum of the ratios between the current load and the rated load of the corresponding idle robotic arm as the dimensionless load coefficient; and construct the cooperative task cost function: ； in, The total cost of the collaborative task. The set of indices for the fresh milk soft packs to be grabbed. The set of indices for the idle robotic arms. The index of the soft milk package to be grabbed. For the index of the idle robotic arm, To indicate the first Have the fresh milk pouches to be grabbed been assigned to the first...? Binary decision variables for an idle robotic arm. For the first The current position of the end effector of the idle robotic arm is compared with that of the first idle robotic arm. The ratio between the actual Cartesian distance between the final gripping positions of each fresh milk pouch to be gripped and the preset maximum working distance. For the first The normalized current joint angle vector of an idle robotic arm. For the first The idle robotic arm reaches the first Each soft bag of fresh milk to be grasped corresponds to the normalized target joint angle vector at the final grasping pose. Let L be the square of the L2 norm of the vector. For the first The reliability of the point cloud completion corresponding to the fresh milk soft package to be captured, and , For the first The dimensionless load coefficient of each idle robotic arm, and , Assuming a preset distance cost weight, To preset the joint movement cost weights, Preset load confidence cost weights; Step S502: With the goal of minimizing the total cost of the collaborative task, the Hungarian algorithm is used to solve the binary decision variables, and the binary decision variables are subject to the constraint that each fresh milk soft bag to be grabbed can be assigned to at most one idle robotic arm and each idle robotic arm can be assigned to at most one fresh milk soft bag to be grabbed in the current grabbing cycle, so as to obtain the optimal grabbing task allocation result. Step S503: Collect the conveying speed output by the encoder pre-set on the fresh milk cold chain conveyor line; for each idle robotic arm in the optimal grasping task allocation result, calculate the end-effector velocity feedforward based on the conveying speed, and calculate the position closed-loop correction based on the position error between the current position of the end of the corresponding idle robotic arm and the corresponding final grasping pose; superimpose the end-effector velocity feedforward and the position closed-loop correction, calculate the corresponding joint speed command through the inverse velocity kinematics of the robotic arm, and generate the multi-robotic arm cooperative control command based on the joint speed command and the optimal grasping task allocation result.

8. A multi-robotic arm collaborative operation control system for a fresh milk cold chain conveyor line, applied to the multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line as described in any one of claims 1 to 7, characterized in that, The multi-robotic arm collaborative operation control system for the fresh milk cold chain conveyor line includes: The visual perception module is used to acquire image data of fresh milk soft packages on the fresh milk cold chain conveyor line, and to perform soft package instance perception tasks based on the image data using the YOLO lightweight instance segmentation mechanism, and output soft package instance perception data. The collapse discrimination module is used to perform the soft package collapse discrimination task based on the soft package instance perception data, using a point cloud spatial mapping and curvature feature classification mechanism, and output soft package collapse discrimination data; The morphological completion module is used to perform the soft package morphological completion task based on the soft package collapse discrimination data and adopt the film tension Poisson compensation mechanism to output the soft package morphological completion data. The pose estimation module is used to complete the data based on the soft package shape, and to perform the grasping pose estimation task by using boundary distance weighted fusion and support vector regression pose correction mechanism, and output grasping pose data. The collaborative control module is used to allocate and control the grasping tasks of multiple robotic arms based on the grasping pose data and adopt a collaborative task cost optimization mechanism, and output collaborative control commands for multiple robotic arms.

9. A multi-robotic arm collaborative operation control device for a fresh milk cold chain conveyor line, characterized in that, The multi-robotic arm collaborative operation control device for the fresh milk cold chain conveyor line includes: a memory, a processor, and a multi-robotic arm collaborative operation control program for the fresh milk cold chain conveyor line stored in the memory and executable on the processor. When the multi-robotic arm collaborative operation control program for the fresh milk cold chain conveyor line is executed by the processor, it implements a multi-robotic arm collaborative operation control method for the fresh milk cold chain conveyor line according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line. When the processor executes the multi-robotic arm collaborative operation control program for a fresh milk cold chain conveyor line, it implements a multi-robotic arm collaborative operation control method for a fresh milk cold chain conveyor line according to any one of claims 1 to 7.

Images