A fresh milk cold chain micro-damage grabbing and shelf life priority boxing method and system based on a multi-modal large model

Abstract

The application discloses a fresh milk cold chain micro-damage grabbing and shelf life priority boxing method and system based on a multi-modal large model, and belongs to the field of logistics automation and artificial intelligence. The method comprises the following steps: acquiring a multi-modal original data stream of a cold chain environment and generating synchronous data; generating a denoising transparent object contour feature map through active thermal excitation time difference analysis; fusing a three-dimensional topological model and a characteristic tensor extracted from a depth point cloud; generating a force control mask through a thermal-force coupling reasoning model; converting a shelf life value into a weighted factor to embed a cost function, planning a shelf life priority grabbing sequence and a placement track; and executing actions according to the force control mask and performing real-time monitoring and re-planning of abnormalities. The application adopts multi-modal perception and thermal-force coupling control, can solve the problems of condensate water reflection interference and material brittleness grabbing, realizes priority boxing of near-expiration products, improves sorting efficiency and reduces the damage rate.

Description

Technical Field

[0001] This invention relates to the fields of logistics automation and artificial intelligence, and in particular to a method and system for grasping and prioritizing the packaging of fresh milk in the cold chain based on a multimodal large model. Background Technology

[0002] Automated material handling and packing technology is a core component of modern logistics and manufacturing, especially in areas involving cargo transfer, sorting, stacking, and loading / unloading, where industrial robots play a crucial role. In the food industry, particularly in sectors requiring a complete cold chain for fresh milk, using robots for automated packing operations not only significantly improves production efficiency but also effectively ensures hygiene standards and product quality stability throughout the production process.

[0003] In existing technologies, robotic systems used for automated box packing typically rely on vision systems for target localization and recognition. These systems often employ two-dimensional color images or three-dimensional vision sensors incorporating depth information to acquire the position and orientation of the object to be grasped. The robotic arm's end effector, such as a vacuum suction cup or parallel gripper, performs grasping and placement actions on the target object based on the information provided by the vision system. Its path planning algorithms typically optimize for the shortest path or fastest cycle time to achieve material transfer from the grasping point to the placement point.

[0004] However, existing technologies have several technical shortcomings when applied to the specific scenario of cold chain packaging for fresh milk. First, the high humidity of the cold chain environment causes condensation on the packaging surface, which can lead to strong specular reflection and contour occlusion in the vision system, significantly reducing positioning and recognition accuracy. Second, traditional gripping methods typically apply uniform gripping force, failing to consider the brittleness of fresh milk packaging materials at low temperatures, especially in stress-concentrated areas such as corners and creases. A uniform gripping force can easily cause minor damage or even tearing of the packaging. Furthermore, existing robotic packing systems operate blindly, unable to identify and differentiate the remaining shelf life of different products, thus failing to automatically implement the storage principle of prioritizing products with earlier expiration dates during the packing process. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and system for grasping and prioritizing fresh milk cold chain products with minimal damage based on a multimodal large model. Employing multimodal fusion sensing, thermo-mechanical coupling force control, and expiration date-driven path planning, it can achieve accurate identification, minimal damage grasping, and intelligent priority packing of fragile goods in the cold chain environment.

[0006] The above objectives can be achieved through the following approach:

[0007] A method and system for detecting minimal damage in the cold chain of fresh milk and prioritizing packaging based on a multimodal large model, comprising:

[0008] The system acquires multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, performs timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generates synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images, and optical character recognition information.

[0009] The synchronous multimodal data is subjected to time difference analysis based on active thermal excitation. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated.

[0010] The transparent object contour feature map is fused with the depth point cloud in three dimensions to extract the three-dimensional topological model of each object and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topological model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface.

[0011] The three-dimensional feature tensor is input into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and a force control mask with spatial gradient is generated. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object.

[0012] The remaining shelf life value in the optical character recognition information is obtained, the remaining shelf life value is converted into a time-space weighting factor, the time-space weighting factor is embedded into the path cost function of the robotic arm joint space, and the grasping order and placement height are planned by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes the expiration date priority.

[0013] The force control mask and the grasping sequence control the end effector of the robotic arm to perform grasping and placing actions in sequence, and collect force and temperature signals in real time during the execution. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

[0014] Optionally, performing time-difference analysis on the synchronous multimodal data based on active thermal excitation includes:

[0015] The surface of the target packaging is heated briefly by a miniature controllable infrared heat source integrated at the end of the robotic arm, and the temperature decay sequence of multiple consecutive frames of infrared thermal images after heating is obtained.

[0016] Calculate the temperature gradient decay rate of each pixel in two adjacent frames of infrared thermal images in the temperature decay sequence to generate a time-difference thermal map.

[0017] Pixel regions in the time-difference thermal image whose temperature decay rate is lower than the decay threshold used to distinguish the difference in heat capacity between water and packaging materials are marked as condensation water regions. The condensation water regions are used to remove reflective noise from the color image and to interpolate and complete the contour, generating a denoised transparent object contour feature map.

[0018] Optionally, the three-dimensional fusion of the transparent object contour feature map with the depth point cloud includes:

[0019] Edge detection is performed on the denoised transparent object contour feature map to extract the two-dimensional contour boundary of each object;

[0020] The two-dimensional contour boundary is spatially registered with the depth point cloud, and a subset of the three-dimensional point cloud within the range defined by the two-dimensional contour boundary is extracted.

[0021] Outlier filtering and surface reconstruction are performed on the subset of the three-dimensional point cloud. The centroid coordinates, principal axis direction, and cross-sectional profile along the principal axis direction of each object are calculated to generate a three-dimensional topological model.

[0022] Optionally, the force control mask is obtained by multiplying the brittleness index by a reference normal force used to characterize the standard gripping force, and the force control mask allows each contact point of the robotic arm end effector to independently apply a dynamic normal force inversely proportional to the local brittleness index.

[0023] Optionally, converting the remaining shelf life value into a time-space weighted factor includes:

[0024] The time-space weighting factor is embedded as a penalty term coefficient in the path cost function, which also includes a joint displacement sum of squares term representing motion smoothness and a total path time term representing execution efficiency.

[0025] The robotic arm joint angle sequence that minimizes the path cost function is obtained by solving a global optimization algorithm, and the grasping sequence and placement trajectory are generated.

[0026] Optionally, controlling the robotic arm end effector to sequentially perform grasping and placement actions according to the force control mask and the grasping sequence further includes:

[0027] Obtain a 3D topological model of the ice pack to be placed and identify tiny ice ridges on the surface of the ice pack;

[0028] The breakage energy budget of the ice pack at its current placement position is calculated based on the force control mask. The breakage energy budget is determined by the difference between the maximum safe shear torque that the end effector of the robotic arm can apply and the average breaking torque of the tiny ice crystals.

[0029] The robotic arm end effector is guided to press the ice pack into the remaining gaps in the box in a preset posture, so that the tiny ice crystals break in a controllable manner under conditions below the breakage energy budget, generating the final posture of the ice pack that matches the shape of the gap.

[0030] Optionally, the real-time acquisition of force and temperature signals during execution includes:

[0031] High-frequency vibration signals at the contact point between the robotic arm's end effector and the packaging are collected using a miniature piezoelectric sensor, and local temperature signals in the gripping area are collected using an infrared thermopile.

[0032] The high-frequency vibration signal, the local temperature signal, and the torque signal output by the six-dimensional force sensor are input into a lightweight time series prediction model, and the lightweight time series prediction model outputs the probability of anomaly occurrence.

[0033] The probability of an anomaly occurring is compared with a warning threshold that characterizes the sensitivity of the warning. When the probability of an anomaly occurring exceeds the warning threshold, an interruption command is generated.

[0034] Optionally, after completing a predetermined number of fetch and place cycles, online self-optimization is also included:

[0035] Record the actual output value of the force control mask, the actual execution result of the grasping sequence, the packaging surface temperature change data, and the space utilization rate after packing in each loop to generate a historical running dataset.

[0036] The model parameters of the thermal-mechanical coupling inference model and the weight coefficients in the path cost function are updated using the historical running dataset through an online learning algorithm.

[0037] The updated model parameters are distilled into a computationally efficient lightweight actuator model, which is then deployed to the robotic arm controller for real-time inference in subsequent loops.

[0038] Optionally, the method further includes:

[0039] The geometry of the flexible bag packaging is monitored using the color image, and the maximum normal distance deviation between its outline and the standard geometric model representing a normal product is calculated to generate the local bulge amount.

[0040] The amount of local bulges is compared with the bulging threshold of the quality control standard. When the amount of local bulges exceeds the bulging threshold, the soft bag is suspected of being fermented and spoiled inside.

[0041] The rejection command is triggered, causing the robotic arm's end effector to skip the soft bag, mark the abnormal item information in the logistics system, and update the grab sequence.

[0042] Based on the same inventive concept, this invention also provides a method and system for detecting and prioritizing the packaging of fresh milk in the cold chain based on a multimodal large model, the system comprising:

[0043] The multimodal data acquisition and calibration module is used to acquire multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, perform timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generate synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images and optical character recognition information.

[0044] The thermal excitation differential denoising module is used to perform time difference analysis based on active thermal excitation on synchronous multimodal data. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated.

[0045] The three-dimensional topology feature extraction module is used to perform three-dimensional fusion of the transparent object contour feature map and the depth point cloud, extract the three-dimensional topology model of each object, and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topology model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface.

[0046] The thermo-mechanical coupling force control mask generation module is used to input a three-dimensional feature tensor into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and generate a force control mask with spatial gradient. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object.

[0047] The shelf-life driven path planning module is used to obtain the remaining shelf-life value in the optical character recognition information, convert the remaining shelf-life value into a time-space weighting factor, embed the time-space weighting factor into the path cost function of the robotic arm joint space, and plan the grasping order and placement height by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes shelf-life priority.

[0048] The closed-loop execution and abnormal replanning module is used to control the end effector of the robotic arm to perform grasping and placement actions sequentially according to the force control mask and the grasping sequence. During the execution, force and temperature signals are collected in real time. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] This invention effectively overcomes visual interference such as condensation and reflection commonly found in cold chain environments by fusing data from multiple sensors and utilizing active thermal excitation technology. It achieves high-precision recognition and contour extraction of semi-transparent objects such as fresh milk packaging, improving the perception robustness and grasping success rate of automated systems in complex visual environments.

[0051] This invention proposes a thermo-mechanical coupling force control gripping strategy. By evaluating the brittleness differences on the packaging surface caused by low temperature and curvature changes in real time, a force control mask with spatial gradient is generated, enabling the robotic arm to apply adaptive, non-uniform gripping force to different parts of the packaging. This reduces the risk of minor damage or breakage to the low-temperature brittle packaging while ensuring gripping stability, thus achieving minimal damage operation.

[0052] This invention transforms the remaining shelf-life information of products into dynamic weighting factors in path planning, enabling the robotic arm's grasping sequence and placement strategy to automatically follow the business logic of prioritizing expiration dates. This innovative method, which deeply embeds logistics management needs into the robot's underlying motion planning, achieves intelligent inventory management during material handling, effectively reducing the loss of near-expiration products.

[0053] This invention constructs a complete closed-loop control system that includes real-time monitoring, anomaly early warning, and online self-optimization. By analyzing multi-source signals in real time to predict failure risks and proactively intervene, and by continuously learning and optimizing the decision-making model during operation, the system not only ensures the safety of individual operations but also possesses the ability to adapt and self-evolve in long-term operation, thereby improving the overall reliability, intelligence, and production flexibility of automated packing production lines.

[0054] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims, and drawings. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a method for capturing minimal damage in the cold chain of fresh milk and prioritizing its shelf life based on a multimodal large model, according to an embodiment of the present invention.

[0057] Figure 2 This is a spatial gradient heatmap of the force control mask according to an embodiment of the present invention.

[0058] Figure 3 This is a performance weight curve of the path cost function in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of a fresh milk cold chain low-damage grasping and shelf-date priority packing system based on a multimodal large model according to an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Reference Figure 1 One embodiment of the present invention proposes a method and system for grasping and prioritizing fresh milk cold chain products with minimal damage based on a multimodal large model. It adopts multimodal fusion perception, thermo-mechanical coupling force control and expiration date driven path planning and other technical means to achieve accurate identification, grasping of minimal damage and intelligent priority packing of fragile products in cold chain environment.

[0062] The method described in this embodiment specifically includes:

[0063] The system acquires multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, performs timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generates synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images, and optical character recognition information.

[0064] The synchronous multimodal data is subjected to time difference analysis based on active thermal excitation. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated.

[0065] The transparent object contour feature map is fused with the depth point cloud in three dimensions to extract the three-dimensional topological model of each object and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topological model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface.

[0066] The three-dimensional feature tensor is input into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and a force control mask with spatial gradient is generated. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object.

[0067] The remaining shelf life value in the optical character recognition information is obtained, the remaining shelf life value is converted into a time-space weighting factor, the time-space weighting factor is embedded into the path cost function of the robotic arm joint space, and the grasping order and placement height are planned by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes the expiration date priority.

[0068] The force control mask and the grasping sequence control the end effector of the robotic arm to perform grasping and placing actions in sequence, and collect force and temperature signals in real time during the execution. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

[0069] Specifically, the technical principle of this invention lies in constructing an intelligent robot control framework based on multimodal perception, model prediction, and closed-loop feedback. First, by fusing infrared thermal images, depth point clouds, color images, and optical character recognition information, comprehensive digital perception of the work objects in the cold chain environment is achieved. This method creatively employs active thermal excitation and temporal difference analysis technology to solve the interference of condensation water on visual recognition, thereby accurately separating the true contour of the object. Based on this accurate contour, a three-dimensional topological model of each object to be grasped is constructed using depth data. Subsequently, the prediction and planning stage begins. On one hand, a pre-trained thermo-mechanical coupling inference model predicts the brittleness of various regions on the object's surface based on its temperature and local curvature, generating a force control mask with spatial gradients to guide refined force-controlled grasping. On the other hand, the expiration date information obtained from optical character recognition is quantified as a weight factor in the path planning cost function. Through global optimization, a grasping sequence and placement trajectory that takes into account expiration date priority are generated. Finally, during the execution phase, closed-loop monitoring is performed by collecting force and temperature signals in real time. Once a risk of capture failure is detected, the system can be immediately interrupted and replanned, forming a complete adaptive control closed loop of "perception-modeling-prediction-planning-execution-feedback".

[0070] Optionally, performing time-difference analysis on the synchronous multimodal data based on active thermal excitation includes:

[0071] The surface of the target packaging is heated briefly by a miniature controllable infrared heat source integrated at the end of the robotic arm, and the temperature decay sequence of multiple consecutive frames of infrared thermal images after heating is obtained.

[0072] Calculate the temperature gradient decay rate of each pixel in two adjacent frames of infrared thermal images in the temperature decay sequence to generate a time-difference thermal map.

[0073] Pixel regions in the time-difference thermal image whose temperature decay rate is lower than the decay threshold used to distinguish the difference in heat capacity between water and packaging materials are marked as condensation water regions. The condensation water regions are used to remove reflective noise from the color image and to interpolate and complete the contour, generating a denoised transparent object contour feature map.

[0074] Specifically, a miniature, controllable infrared heat source integrated into the end effector of a robotic arm is used to pulse-heat the surface of the target packaging for short periods. The thermal response of the surface after heating is stopped is captured in real time by an infrared thermal imager. The core of this processing logic lies in utilizing the significant physical differences in heat capacity and thermal diffusivity between water droplets and packaging materials. Because the specific heat capacity of water molecules is much higher than that of solid packaging materials, the temperature decrease rate of the condensate area is significantly slower than that of the packaging surface after the external thermal excitation is removed. This is achieved by calculating the temperature gradient decay rate of each pixel in adjacent frames of infrared images. A time-difference thermal model is established. The decay rate is quantified using the following formula:

[0075] ;

[0076] in, For pixels Temperature decay rate, in units of Its value directly reflects the thermal inertia characteristics of the material at that point; The instant heating stops The initial temperature value of this pixel at any given time comes from the first frame of data from the infrared thermal imaging sensor; For time step back The temperature value at any given moment is derived from sampling data of subsequent consecutive frames. The preset time sampling interval is determined by the frame rate of the infrared sensor. A preset attenuation threshold is used. Binarizing the image will satisfy the following conditions: The pixel region was identified as a high-heat-capacity condensate area. Subsequently, the coordinates of the identified noise region were mapped onto the synchronously acquired color image. A pixel reconstruction algorithm was used to remove the specular reflection bright spots caused by the condensate, and the occluded contour edges were interpolated and completed based on gradient operators, thereby obtaining a pure and transparent object contour feature map that is not affected by ambient humidity.

[0077] For example, when the robotic arm's end effector moves above the milk package to be grasped, a 10W miniature infrared heater is activated and continuously irradiates the package for 0.5 seconds. An infrared sensor captures the cooling process at a frame rate of 60fps, with a set sampling step size. The interval is 0.1 seconds, or 6 frames. At a specific pixel... At that point, the sensor measures the initial temperature at the instant heating ends. for Temperature after 0.1s for Substituting into the formula, the decay rate at that point can be calculated. The characteristic attenuation threshold of the packaging material under this cold chain environment is known. Set as Due to the calculated decay rate Much smaller than the threshold The pixel was automatically marked as condensation noise. However, in the dry area at the edge of the packaging, the measured... for , for Its decay rate is greater than the threshold Therefore, it was determined to be the edge of a real object. By performing this logical judgment on all pixels in the image, three condensation patches covering the packaging surface were successfully identified. The outline of the patches was then repaired in the color image using the mean interpolation algorithm of the surrounding pixels, ultimately generating an object feature map with an edge closure rate of up to 98%.

[0078] Optionally, the three-dimensional fusion of the transparent object contour feature map with the depth point cloud includes:

[0079] Edge detection is performed on the denoised transparent object contour feature map to extract the two-dimensional contour boundary of each object;

[0080] The two-dimensional contour boundary is spatially registered with the depth point cloud, and a subset of the three-dimensional point cloud within the range defined by the two-dimensional contour boundary is extracted.

[0081] Outlier filtering and surface reconstruction are performed on the subset of the three-dimensional point cloud. The centroid coordinates, principal axis direction, and cross-sectional profile along the principal axis direction of each object are calculated to generate a three-dimensional topological model.

[0082] Specifically, after obtaining the denoised transparent object contour feature map... Then, edge detection operators, such as the Canny operator, are invoked to extract the 2D contour boundaries of each object. To achieve deep coupling between the 2D features and the 3D physical space, a pre-calibrated extrinsic parameter matrix is ​​used. Map the pixel coordinates of the 2D contour boundary to the depth camera coordinate system and compare them with the original depth point cloud. Spatial coordinate registration is performed to extract a subset of the 3D point cloud within the area defined by the 2D contour boundary. Since the point cloud of ice packs or glass bottles in a cold chain environment may contain local missing points or outliers, statistical filtering is applied to the 3D point cloud subset to eliminate noise, and surface reconstruction is performed based on the Poisson reconstruction algorithm. Subsequently, principal component analysis (PCA) is performed on the reconstructed geometry to calculate the geometric centroid coordinates, the three mutually perpendicular principal axis direction vectors, and the cross-sectional features along the principal axis directions. At this point, the geometric center of the object is extracted. Main spindle direction and the local curvature distribution of each surface This forms a complete three-dimensional topological model. Based on this, global average pooling is used to map the structural information of the point cloud into a high-dimensional vector, which is then concatenated with semantic features from the visual feature map to generate a three-dimensional feature tensor of the object to be grasped. To ensure the stability of the packing plan, it is also necessary to calculate the space between the object and the insulated box. The geometric fit. Space occupancy rate is introduced as part of the topological features, and its calculation formula is:

[0083] ;

[0084] in, The space occupancy coefficient for a single object; The volume of the object is calculated by integration using a 3D topological model, in units of 1. The source is the volume calculation of a subset of three-dimensional point clouds after surface reconstruction, which is then meshed into a closed manifold triangular mesh. This is the preset total usable volume constant inside the insulation box, in units of... This coefficient With the local curvature of the object They are all encapsulated in a 3D feature tensor, providing comprehensive input data about the physical shape of the object and environmental constraints for subsequent thermo-mechanical coupling models.

[0085] For example, an irregularly shaped, frozen ice pack was observed. First, its edges were extracted from a two-dimensional feature map. After spatial coordinate registration, 5000 point cloud samples belonging to the ice pack were extracted from the depth point cloud. After Poisson surface reconstruction, the centroid of the ice pack was calculated to be located at the coordinate... The location was identified, and its longest principal axis was determined to be distributed along the X-axis. The volume of the ice pack was calculated using grid integration. for If the currently used insulated box has a preset total volume... for The space occupancy coefficient of the ice pack can then be calculated using the formula. Simultaneously, local curvature at the top of the ice pack was detected. Dramatic changes occur at the edges, indicating the presence of tiny icicles. These geometric features, such as the center of mass, principal axes, , Encapsulated into The three-dimensional feature tensor is input into the subsequent model to determine the optimal gripping plane. When gripping the refrigerant, which accounts for 2% of the volume, the robotic arm can automatically align its longest dimension with the direction of the largest gap inside the box.

[0086] Optionally, the force control mask is obtained by multiplying the brittleness index by a reference normal force used to characterize the standard gripping force, and the force control mask allows each contact point of the robotic arm end effector to independently apply a dynamic normal force inversely proportional to the local brittleness index.

[0087] Specifically, the thermo-mechanical coupling inference model extracts the surface temperature distribution and local curvature distribution from the three-dimensional feature tensor, and uses a nonlinear regression algorithm to map the brittleness index of each sampling point on the object's surface. To achieve minimally damaged grasping, the normal force needs to be dynamically adjusted according to the degree of embrittlement in different areas. Force control mask. The generation logic couples the basic grasping load with the brittleness influence factor in the calculation, and the calculation formula is as follows:

[0088] ;

[0089] in, For the end effector of the robotic arm in the first The actual dynamic normal force applied at each contact point, in units of ; The standard reference normal force is preset based on the object's mass and coefficient of friction, with units of . The value is derived from the experimental determination of the critical force of packaging slippage. This is the brittleness adjustment weighting coefficient, with a value range of [value range missing]. It is used to balance gripping stability and breakage risk, and its value comes from the calibration test of the low temperature impact toughness of different packaging materials; For the first The brittleness index at each contact point is output by a thermo-mechanical coupling inference model; a higher value indicates that the area is more prone to failure under the current temperature and curvature. This formula allows the end effector to spontaneously reduce pressure in areas with high brittleness index, such as corners with low-temperature stress concentration, while maintaining sufficient pressure in flat and more resilient areas, thus forming a force-controlled mask with a spatial gradient. .like Figure 2 As shown, a force control mask is generated based on the fragility index of different parts of the fresh milk packaging surface. The horizontal and vertical axes in the figure represent the packaging contact surface, and the gray values ​​represent the pressure distribution, clearly demonstrating the spatial gradient characteristics of automatic pressure reduction in highly brittle areas such as packaging corners.

[0090] For example, a surface temperature of The Tetra Pak cartons for fresh milk have highly curved creases along their edges. A baseline positive pressure is set based on a 500g weight of this package. for During the thermo-mechanical coupling analysis, the model identified the brittleness index of the flat area on the side of the packaging. The value is 0.2, while the brittleness index at the top corner is... Due to extremely low temperatures and drastic changes in curvature, the value reaches as high as 0.8. A brittleness adjustment weighting coefficient is set. The value is 0.5. Substituting this into the formula, the dynamic normal force at the flat side surface is calculated. The dynamic positive pressure at the top corner Based on this force control mask, the end effector of the robotic arm automatically reduces the pressure by 40% when it comes into contact with the corner, thereby preventing the packaging material from cracking due to low-temperature embrittlement.

[0091] Optionally, converting the remaining shelf life value into a time-space weighted factor includes:

[0092] The time-space weighting factor is embedded as a penalty term coefficient in the path cost function, which also includes a joint displacement sum of squares term representing motion smoothness and a total path time term representing execution efficiency.

[0093] The robotic arm joint angle sequence that minimizes the path cost function is obtained by solving a global optimization algorithm, and the grasping sequence and placement trajectory are generated.

[0094] Specifically, to automatically prioritize the handling of products expiring earlier in the packing process, the non-geometric constraint of shelf life needs to be translated into the motion cost of the robotic arm. First, optical character recognition is used to obtain the remaining shelf life days for each package. It also calculates the normalized reciprocal of the shelf life relative to the longest shelf life in the current batch, thereby generating a time-weighted factor. Simultaneously, spatial height weights are generated based on the current stacking height determined by the 3D topology model. Couple these two factors into a time-space weighting factor. :

[0095] ;

[0096] It is also embedded as a penalty term in the path cost function of the robotic arm. The specific formula for calculating this function is:

[0097] ;

[0098] in, The total path cost is derived from a comprehensive performance evaluation of the robotic arm's grasping sequence. For the robotic arm in The joint angular velocity vector at time t, in units of The values ​​are fed back in real time by the encoders of each joint. The term represents the sum of squares of joint displacements that characterize the smoothness of motion; The total path time to complete the current crawling task, This is the time efficiency coefficient; As a time-space weighting factor, This is the expiration date penalty coefficient, the value of which is derived from the inventory turnover priority intensity preset by the logistics system. The term forces the guided algorithm to prioritize solving the problem by minimizing the cost function. Objects with longer paths, meaning shorter shelf lives and lower placement levels, have higher action priority. This function is solved using global optimization algorithms such as particle swarm optimization to obtain... Minimized joint angle sequence This generates a fetch sequence and placement trajectory that balances execution efficiency with expiration date priority. For example... Figure 3 As shown, the curve depicts the change of the path cost function with the remaining shelf life. As the shelf life decreases, the penalty term weight increases non-linearly, intuitively demonstrating how the algorithm guides the robotic arm to execute "first-in, first-out" logic by maximizing the action priority of short-shelf-life products.

[0099] For example, suppose there are two fresh milk packages in the current processing area, and package A has a remaining shelf life of... The remaining shelf life of package B is 2 days. The time is 7 days. The calculated time weighting factor for packaging A is 0.8, and for packaging B it is 0.2. An expiration date penalty coefficient is set. The time efficiency coefficient is 100. The value is 10. For path one: first capture A and then capture B, the calculated motion smoothness term is 50, and the total time is... The duration is 5s. Since A has priority and is located at the bottom level, its corresponding duration cost is... Total cost For path two: apprehending B before A, the cost of expiration dates skyrockets because it violates the principle of processing items based on their expiration dates first. Total cost The global optimization algorithm found through comparison that... Therefore, the system automatically generates an action command to grab package A, which has a shelf life of only 2 days, and place it at the bottom of the box. The "first-in, first-out" inventory management logic is automatically executed in the underlying motion control.

[0100] Optionally, controlling the robotic arm end effector to sequentially perform grasping and placement actions according to the force control mask and the grasping sequence further includes:

[0101] Obtain a 3D topological model of the ice pack to be placed and identify tiny ice ridges on the surface of the ice pack;

[0102] The breakage energy budget of the ice pack at its current placement position is calculated based on the force control mask. The breakage energy budget is determined by the difference between the maximum safe shear torque that the end effector of the robotic arm can apply and the average breaking torque of the tiny ice crystals.

[0103] The robotic arm end effector is guided to press the ice pack into the remaining gaps in the box in a preset posture, so that the tiny ice crystals break in a controllable manner under conditions below the breakage energy budget, generating the final posture of the ice pack that matches the shape of the gap.

[0104] Specifically, during automated box packing, ice packs often develop irregular micro-ice ridges on their surface due to freezing. If these are directly and forcefully pressed into the gaps inside the box, stress concentration may cause packaging damage or damage to the box structure. Therefore, a three-dimensional topological model is first used to identify the geometric singularities on the ice pack surface, i.e., the micro-ice ridges, and their average breaking torque is calculated. To ensure a safe and controlled pressing process, a breakage energy budget model needs to be established. The formula used to limit the maximum energy that a robotic arm is allowed to dissipate in ice breakage when squeezing an ice pack is as follows:

[0105] ;

[0106] in, For the energy budget of the fragmentation, the unit is joules (J). The numerical value represents the energy reserve that the system allows for changing the physical form of the ice pack without damaging the main packaging. The maximum safe shear torque that the robotic arm end effector is allowed to apply, in units of Its value comes from the force control mask generated by the thermo-mechanical coupling inference model, and represents the critical torque at which the packaging material will not tear at the current temperature; The average breaking torque of the tiny icicles, in units of The value is estimated by the three-dimensional topological feature extraction module based on the root diameter and material strength of the icicle. The preset indentation angular displacement, in units of This represents the range of posture adjustment the robotic arm makes when performing the pressing motion. Using this formula, the robotic arm can quantify the physical constraints of its current position, guiding the end effector to push the ice pack into the gap with a specific compensated posture. During the pressing motion, the joint torque is monitored in real time to ensure that the actual energy consumption is lower than [the specified value]. This allows the ice shards to break in a controlled manner, enabling the ice pack to adapt its final shape to fill the remaining irregular gaps in the box.

[0107] For example, suppose a robotic arm is preparing to press an ice pack with protruding ice ridges into the corner of a cold chain container. The 3D topology feature extraction module identifies an average diameter of 5mm at the root of the ice ridges and estimates its average breaking torque. for At this point, the thermo-mechanical coupling inference model is based on the current... Based on the ambient temperature and the brittleness of the packaging material, the maximum safe shear torque of the packaging was calculated using a force control mask. for If the attitude adjustment angular displacement during pressing is set... for Then, calculate the crushing energy budget according to the formula. During actual operation, the robotic arm's end effector moves at an angle. The posture cuts into the gap, and when the real-time torque feedback shows the energy consumed to break the icicle... If the resistance is too high, causing the torque to approach its limit, the action continues; if the resistance is too high, the torque will approach its limit. The robotic arm will immediately stop pressing down and fine-tune its posture. In this way, the originally irregularly shaped ice pack... The controlled breakage of the icicle was successfully completed and inserted into the originally unacceptable gap, while the packaging remained intact.

[0108] Optionally, the real-time acquisition of force and temperature signals during execution includes:

[0109] High-frequency vibration signals at the contact point between the robotic arm's end effector and the packaging are collected using a miniature piezoelectric sensor, and local temperature signals in the gripping area are collected using an infrared thermopile.

[0110] The high-frequency vibration signal, local temperature signal, and torque signal output by the six-dimensional force sensor are input into a lightweight time series prediction model, which outputs the probability of anomaly occurrence.

[0111] The probability of an anomaly occurring is compared with a warning threshold that characterizes the sensitivity of the warning. When the probability of an anomaly occurring exceeds the warning threshold, an interruption command is generated.

[0112] Specifically, during the robotic arm's grasping action, the system establishes an anomaly prediction model using multi-source high-frequency sensors to identify potential failure precursors such as slippage, damage, or poor contact in real time. First, a miniature piezoelectric sensor integrated into the fingertip of the robotic arm's end effector captures high-frequency vibration signals from the packaging surface, reflecting the microscopic shear deformation of the contact surface. Simultaneously, an infrared thermopile sensor acquires the local temperature signal at the grasping point; this signal is used to correct the elastic modulus of the packaging material at the current instantaneous temperature, improving the accuracy of torque analysis. The high-frequency vibration signal, local temperature signal, and torque signal fed back in real time by the six-dimensional force sensor at the robotic arm's base are then fused and input into a lightweight time-series prediction model. The core of this prediction model lies in calculating the deviation of the current state from the safe grasping envelope and outputting the probability of anomaly occurrence. The calculation formula is as follows:

[0113] ;

[0114] in, The probability of an anomaly occurring is represented by the following values: The range, the higher the value, the closer the current capture state is to the edge of failure; The actual contact torque fed back by the six-dimensional force sensor, in units of This is derived from the real-time force feedback when the actuator comes into contact with the object; The desired torque value of the force control mask generated by the thermal-mechanical coupling inference model at the current sampling point, in units of This serves as a benchmark for secure data capture. This is a local temperature signal of the grab point area acquired by an infrared thermopile, in units of... As a nonlinear correction factor, it reflects the sensitivity contribution of low-temperature embrittlement to torque deviation; This is the preset sensitivity attenuation coefficient, in units of... The value is derived from experimental calibration of the friction coefficient drift under different humidity levels in cold chain environments. The calculated value... The system compares the data with a preset warning threshold in real time. If the threshold is exceeded, it is determined that there is a risk of capture failure and an interrupt command is immediately triggered.

[0115] For example, when the end effector of a robotic arm grasps a Tetra Pak carton of fresh milk with an extremely low surface temperature, the thermo-mechanical coupling inference model pre-determines the desired torque. for In the initial stage of the lifting action, the actual torque fed back by the six-dimensional force sensor... Due to fluctuations caused by minute slippage, the measured value is At this point, the infrared thermopile measures the local temperature of the contact area. for Preset sensitivity attenuation coefficient The value is 0.05. Substitute this value into the formula to calculate the probability of the current anomaly occurring: If the miniature piezoelectric sensor captures high-frequency slip vibration at this time, leading to If the dynamic compensation is increased to 5.0, then... Rapidly soaring: Assuming the system's warning threshold is set to 0.10, since the calculated value of 0.1175 exceeds this threshold, the robotic arm controller will immediately interrupt the current lifting action to prevent the packaging from slipping or being crushed due to abnormal gripping torque, and trigger the replanning module to find a more stable gripping point.

[0116] Optionally, after completing a predetermined number of fetch and place cycles, online self-optimization is also included:

[0117] Record the actual output value of the force control mask, the actual execution result of the grasping sequence, the packaging surface temperature change data, and the space utilization rate after packing in each loop to generate a historical running dataset.

[0118] The model parameters of the thermal-mechanical coupling inference model and the weight coefficients in the path cost function are updated using the historical running dataset through an online learning algorithm.

[0119] The updated model parameters are distilled into a computationally efficient lightweight actuator model, which is then deployed to the robotic arm controller for real-time inference in subsequent loops.

[0120] Specifically, during the operation of the automated production line, the deviations of the offline-trained model are corrected by continuously accumulating operational feedback from the real environment. After each round of grasping and placing, the actual output pressure of the force control mask, the actual slippage feedback from the robotic arm's end effector, the instantaneous temperature change of the packaging surface recorded by the infrared sensor, and the space utilization rate after packing are structured and encapsulated to construct a historical operation dataset. The core of online self-optimization lies in dynamically updating the weights of the inference model by calculating the deviation between the current execution result and the global optimal goal. To quantify the improvement in model performance, a self-optimization evaluation function is introduced. :

[0121] ;

[0122] in, This is a self-optimizing evaluation coefficient, with a value range of [value range missing]. Between these values, a higher value indicates a better fit between the current model parameters and the environment. This value directly determines the update magnitude of the hidden layer weight matrix of the neural network by the online learning algorithm. This represents the cumulative force deviation during this cycle, in units of... Its source is the integral of the absolute value of the difference between the pressure value fed back in real time by the six-dimensional force sensor and the preset pressure value of the force control mask; To optimize space utilization after packing, the calculation is based on the volume of remaining empty spaces inside the box, performed by the 3D topology feature extraction module, and the volume of the packaged goods. Calculation of the ratio; The historical forgetting factor, whose value is derived from the statistical calibration of the frequency of environmental humidity fluctuations over the past 100 cycles, is used to balance the influence weights of old and new data on model parameters. The online learning algorithm is based on... The numerical values ​​are used to fine-tune the nonlinear mapping weights in the thermo-mechanical coupling inference model using stochastic gradient descent. Then, through knowledge distillation, the updated complex model parameters are migrated to a lightweight actuator model with only three fully connected layers, thereby achieving adaptation to the physical properties of different batches of packaging materials while ensuring computational efficiency.

[0123] For example, suppose that in a single packing task, the robotic arm continuously handles 50 bags of fresh milk with a thick layer of frost on their surfaces. Because the frost alters the coefficient of friction, it reduces the actual gripping force. A significant deviation occurred between the force control mask preset value and the actual force deviation. The cumulative force deviation for this cycle was statistically analyzed. for Due to frost causing unstable stacking, the space utilization after packing is reduced. It decreased to 0.75. At this point, the historical forgetting factor was set based on environmental stability. The value is 0.1. Substitute this value into the formula to calculate the current self-optimization evaluation coefficient: ;because The value is low, far below the ideal value of 1.0, indicating that the current thermo-mechanical coupling inference model failed to accurately predict the brittleness and frictional properties of the frost-covered packaging material. An online update was then triggered, using the temperature signal and pressure feedback data collected in this cycle as training samples, and setting the update step size of the weight matrix to... The model parameters are then adjusted by a significant margin. The updated parameters are then distilled and redeployed to the actuator model. In the next iteration, if... Reduce to and If increased to 0.85, then This indicates that the model has successfully completed online self-optimization for frost environments, and the capture accuracy has been significantly compensated.

[0124] Optionally, the method further includes:

[0125] The geometry of the flexible bag packaging is monitored using the color image, and the maximum normal distance deviation between its outline and the standard geometric model representing a normal product is calculated to generate the local bulge amount.

[0126] The amount of local bulges is compared with the bulging threshold of the quality control standard. When the amount of local bulges exceeds the bulging threshold, the soft bag is suspected of being fermented and spoiled inside.

[0127] The rejection command is triggered, causing the robotic arm's end effector to skip the soft bag, mark the abnormal item information in the logistics system, and update the grab sequence.

[0128] Specifically, during fresh milk packaging, internal fermentation in soft-bag packaging can generate gas, causing abnormal bulging of the bag. A real-time color image of the target object is acquired using a color image acquisition module, and a deep learning edge detection algorithm is used to extract the real-time surface contour curve of the soft bag. To quantify the degree of bulging, a standard geometric model of this type of soft bag under normal vacuum or nitrogen-filled packaging conditions is pre-stored; this model is defined as a set of three-dimensional reference coordinate points. The real-time contour is projected onto the coordinate system of the standard geometric model through spatial transformation, and the normal distance between each sampling point on the real-time contour and the corresponding point on the surface of the standard model is calculated. To accurately characterize local distortion, a local bulge evaluation index is introduced. :

[0129] ;

[0130] in, This represents the amount of localized protrusion, in units of... Its value represents the degree to which the surface of the soft bag deviates from its normal shape, and is determined by the statistical fluctuation of the contour deviation and the maximum deviation. The total number of contour sampling points is derived from the number of points sampled at equal intervals on the edges of the real-time color image. For the first The normal distance of each sampling point relative to the standard geometric model, in units of It originates from the scalar difference between the coordinates of the real-time contour points and the coordinates of the corresponding points on the surface of the standard model in the direction of the normal vector; This is the arithmetic mean of the normal distances to all sampling points, in units of... A bulging threshold, representing a quality control standard, was preset. This threshold is derived from experimental calibration of the relationship between the volume expansion rate of the soft bag and the headspace pressure at different fermentation stages. When the calculated... Exceed If the system detects an abnormal physical pressure inside the soft bag, indicating a risk of fermentation and spoilage, the controller immediately blocks the grabbing commands for that object and removes its coordinates from the global grabbing sequence.

[0131] For example, when processing a batch of pillow-shaped soft-bag fresh milk, the vision system detected a distinct spherical bulge in the central area of ​​the soft bag. This bulge was extracted from the color image along the long axis of the soft bag. 100 sampling points were used. By comparing with a standard flat bag model, the normal distance of the edge region was measured at these 100 points. near And the central area point Gradually increases, with the maximum normal distance being... for The arithmetic mean of this set of distance data was calculated. for The statistical standard deviation is calculated as follows: Substitute the values ​​into the formula to calculate the local bulge amount of the soft bag. The known swelling threshold for quality control of this type of fresh milk is... Set as .because Much larger The system automatically determines that the product is an abnormal item suspected of fermentation and spoilage based on the threshold. The robotic arm then skips the location and moves to grab the next normal target in the sequence, while simultaneously generating an alarm with the abnormal item coordinates marked in red on the logistics dashboard, preventing the spoiled product from entering the subsequent cold chain packaging process.

[0132] Based on the same inventive concept, such as Figure 4As shown, the present invention also provides a fresh milk cold chain minimal-damage capture and shelf-date priority packing system based on a multimodal large model, the system comprising:

[0133] The multimodal data acquisition and calibration module is used to acquire multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, perform timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generate synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images and optical character recognition information.

[0134] The thermal excitation differential denoising module is used to perform time difference analysis based on active thermal excitation on synchronous multimodal data. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated.

[0135] The three-dimensional topology feature extraction module is used to perform three-dimensional fusion of the transparent object contour feature map and the depth point cloud, extract the three-dimensional topology model of each object, and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topology model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface.

[0136] The thermo-mechanical coupling force control mask generation module is used to input a three-dimensional feature tensor into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and generate a force control mask with spatial gradient. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object.

[0137] The shelf-life driven path planning module is used to obtain the remaining shelf-life value in the optical character recognition information, convert the remaining shelf-life value into a time-space weighting factor, embed the time-space weighting factor into the path cost function of the robotic arm joint space, and plan the grasping order and placement height by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes shelf-life priority.

[0138] The closed-loop execution and abnormal replanning module is used to control the end effector of the robotic arm to perform grasping and placement actions sequentially according to the force control mask and the grasping sequence. During the execution, force and temperature signals are collected in real time. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

[0139] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0140] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A method for detecting minimal damage in the cold chain of fresh milk and prioritizing packaging based on a multimodal large model, characterized in that, The method includes: The system acquires multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, performs timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generates synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images, and optical character recognition information. The synchronous multimodal data is subjected to time difference analysis based on active thermal excitation. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated. The transparent object contour feature map is fused with the depth point cloud in three dimensions to extract the three-dimensional topological model of each object and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topological model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface. The three-dimensional feature tensor is input into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and a force control mask with spatial gradient is generated. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object. The remaining shelf life value in the optical character recognition information is obtained, the remaining shelf life value is converted into a time-space weighting factor, the time-space weighting factor is embedded into the path cost function of the robotic arm joint space, and the grasping order and placement height are planned by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes the expiration date priority. The force control mask and the grasping sequence control the end effector of the robotic arm to perform grasping and placing actions in sequence, and collect force and temperature signals in real time during the execution. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

2. The method for capturing minimal damage and prioritizing shelf-date packing of fresh milk in the cold chain based on a multimodal large model according to claim 1, characterized in that, The time-difference analysis of the synchronous multimodal data based on active thermal excitation includes: The surface of the target packaging is heated briefly by a miniature controllable infrared heat source integrated at the end of the robotic arm, and the temperature decay sequence of multiple consecutive frames of infrared thermal images after heating is obtained. Calculate the temperature gradient decay rate of each pixel in two adjacent frames of infrared thermal images in the temperature decay sequence to generate a time-difference thermal map. Pixel regions in the time-difference thermal image whose temperature decay rate is lower than the decay threshold used to distinguish the difference in heat capacity between water and packaging materials are marked as condensation water regions. The condensation water regions are used to remove reflective noise from the color image and to interpolate and complete the contour, generating a denoised transparent object contour feature map.

3. The method for capturing and prioritizing fresh milk cold chain products based on a multimodal large model according to claim 1, characterized in that, The three-dimensional fusion of the transparent object contour feature map and the depth point cloud includes: Edge detection is performed on the denoised transparent object contour feature map to extract the two-dimensional contour boundary of each object; The two-dimensional contour boundary is spatially registered with the depth point cloud, and a subset of the three-dimensional point cloud within the range defined by the two-dimensional contour boundary is extracted. Outlier filtering and surface reconstruction are performed on the subset of the three-dimensional point cloud. The centroid coordinates, principal axis direction, and cross-sectional profile along the principal axis direction of each object are calculated to generate a three-dimensional topological model.

4. The method for capturing and prioritizing fresh milk cold chain packaging with minimal damage based on a multimodal large model according to claim 1, characterized in that, The force control mask is obtained by multiplying the brittleness index by a reference normal force used to characterize the standard gripping force. The force control mask allows each contact point of the robotic arm end effector to independently apply a dynamic normal force inversely proportional to the local brittleness index.

5. The method for capturing and prioritizing fresh milk cold chain packaging with minimal damage based on a multimodal large model according to claim 1, characterized in that, Converting the remaining shelf life value into a time-space weighted factor includes: The time-space weighting factor is embedded as a penalty term coefficient in the path cost function, which also includes a joint displacement sum of squares term representing motion smoothness and a total path time term representing execution efficiency. The robotic arm joint angle sequence that minimizes the path cost function is obtained by solving a global optimization algorithm, and the grasping sequence and placement trajectory are generated.

6. The method for capturing and prioritizing fresh milk cold chain packaging with minimal damage based on a multimodal large model according to claim 1, characterized in that, The process of sequentially executing grasping and placing actions by the robotic arm end effector based on the force control mask and the grasping sequence also includes: Obtain a 3D topological model of the ice pack to be placed and identify tiny ice ridges on the surface of the ice pack; The breakage energy budget of the ice pack at its current placement position is calculated based on the force control mask. The breakage energy budget is determined by the difference between the maximum safe shear torque that the end effector of the robotic arm can apply and the average breaking torque of the tiny ice crystals. The robotic arm end effector is guided to press the ice pack into the remaining gaps in the box in a preset posture, so that the tiny ice crystals break in a controllable manner under conditions below the breakage energy budget, generating the final posture of the ice pack that matches the shape of the gap.

7. The method for capturing and prioritizing fresh milk cold chain products based on a multimodal large model according to claim 1, characterized in that, The real-time acquisition of force and temperature signals during execution includes: High-frequency vibration signals at the contact point between the robotic arm's end effector and the packaging are collected using a miniature piezoelectric sensor, and local temperature signals in the gripping area are collected using an infrared thermopile. The high-frequency vibration signal, the local temperature signal, and the torque signal output by the six-dimensional force sensor are input into a lightweight time series prediction model, and the lightweight time series prediction model outputs the probability of anomaly occurrence. The probability of an anomaly occurring is compared with a warning threshold that characterizes the sensitivity of the warning. When the probability of an anomaly occurring exceeds the warning threshold, an interruption command is generated.

8. The method for capturing and prioritizing fresh milk cold chain products based on a multimodal large model according to claim 1, characterized in that, After completing a predetermined number of fetch and place cycles, online self-optimization is also included: Record the actual output value of the force control mask, the actual execution result of the grasping sequence, the packaging surface temperature change data, and the space utilization rate after packing in each loop to generate a historical running dataset. The model parameters of the thermal-mechanical coupling inference model and the weight coefficients in the path cost function are updated using the historical running dataset through an online learning algorithm. The updated model parameters are distilled into a computationally efficient lightweight actuator model, which is then deployed to the robotic arm controller for real-time inference in subsequent loops.

9. A method for detecting minimal damage and prioritizing shelf-date packaging of fresh milk in the cold chain based on a multimodal large model, as described in claim 1, is characterized in that... The method further includes: The geometry of the flexible bag packaging is monitored using the color image, and the maximum normal distance deviation between its outline and the standard geometric model representing a normal product is calculated to generate the local bulge amount. The amount of local bulges is compared with the bulging threshold of the quality control standard. When the amount of local bulges exceeds the bulging threshold, the soft bag is suspected of being fermented and spoiled inside. The rejection command is triggered, causing the robotic arm's end effector to skip the soft bag, mark the abnormal item information in the logistics system, and update the grab sequence.

10. A fresh milk cold chain low-damage handling and shelf-date priority packing system based on a multimodal large model, characterized in that, The system is used for the fresh milk cold chain micro-damage capture and shelf-date priority packing method based on a multimodal large model as described in any one of claims 1-9, the system comprising: The multimodal data acquisition and calibration module is used to acquire multimodal raw data streams of fresh milk packaging and ice packs to be grabbed in a cold chain environment, perform timestamp alignment and spatial coordinate system calibration on the multimodal raw data streams, and generate synchronous multimodal data. The multimodal raw data streams include infrared thermal images, depth point clouds, color images and optical character recognition information. The thermal excitation differential denoising module is used to perform time difference analysis based on active thermal excitation on the synchronous multimodal data. By comparing the temperature decay rate difference between adjacent frames of infrared thermal images after heating, the condensation water area and the edge of the real object are separated, and a denoised transparent object contour feature map is generated. The three-dimensional topology feature extraction module is used to perform three-dimensional fusion of the transparent object contour feature map and the depth point cloud, extract the three-dimensional topology model of each object, and generate the three-dimensional feature tensor of the object to be grasped. The three-dimensional topology model includes the geometric center of the object, the principal axis direction, and the local curvature distribution of each surface. The thermo-mechanical coupling force control mask generation module is used to input the three-dimensional feature tensor into a thermo-mechanical coupling inference model obtained by nonlinear fitting of the packaging rupture threshold under different temperatures and curvatures in historical capture experimental data, and generate a force control mask with spatial gradient. The thermo-mechanical coupling inference model maps the brittleness index of each region according to the surface temperature distribution and local curvature distribution of the object. The shelf-life driven path planning module is used to obtain the remaining shelf-life value in the optical character recognition information, convert the remaining shelf-life value into a time-space weighting factor, embed the time-space weighting factor into the path cost function of the robotic arm joint space, and plan the grasping order and placement height by minimizing the path cost function, thereby generating a grasping sequence and placement trajectory that includes shelf-life priority. The closed-loop execution and abnormal replanning module is used to control the end effector of the robotic arm to perform grasping and placement actions sequentially according to the force control mask and the grasping sequence. During the execution, force and temperature signals are collected in real time. When the force or temperature signal is detected to have a precursory feature that indicates the risk of grasping failure, the current action is interrupted and the grasping sequence is replanned.

Images