Self-adaptive stacking and fallen material re-grabbing method for AGV binocular vision mechanical arm

Abstract

The invention discloses an AGV binocular vision mechanical arm self-adaptive stacking and fallen material re-grabbing method, and relates to the technical field of intelligent robots and automatic logistics. According to the method, after cargo compartment space measurement is completed through an AGV carrying a binocular camera, material characteristics and cargo compartment vibration data are collected firstly, then dynamic grabbing force is calculated based on material density, volume, a flexibility coefficient and vibration frequency, anomaly detection confidence is calculated in combination with binocular vision depth-of-field data, and a reset path is corrected according to the confidence and a material falling distance; meanwhile, a grading processing mechanism of automatic resetting in the view and remote linkage outside the view is constructed; the method solves the problems that in the prior art, the material flexibility and vibration coupling influence cannot be adapted, the anomaly detection precision is low, and falling material treatment is passive, the stacking stability and the special material safety are improved, and the method is suitable for the automatic stacking scene of multiple types of materials.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots and automated logistics technology, specifically to an adaptive palletizing and re-grabbing method for AGV binocular vision robotic arms. Background Technology

[0002] In automated logistics palletizing scenarios, the collaborative operation of AGVs and robotic arms has become the mainstream solution. However, existing technologies still face significant technical bottlenecks when dealing with special materials and dynamic operating environments. Current technologies typically employ fixed gripping force and a single vision detection mode, failing to consider the flexibility of materials—for example, flexible materials are prone to irreversible deformation under fixed gripping force, leading to high breakage rates—and also failing to adapt to the dynamic impact of cargo box vibration. Cargo box vibration causes fluctuations in the contact pressure of stacked materials, resulting in tilting or material falling, which traditional low-frequency vision detection cannot capture in time. Furthermore, existing technologies rely on manual intervention to handle abnormal materials; even if an anomaly is detected, on-site adjustments by the operator are required, severely impacting operational continuity. In summary, existing technologies cannot dynamically balance the coupled effects of material characteristics and environmental vibration, resulting in poor palletizing stability, high breakage rates for special materials, and low efficiency in handling anomalies.

[0003] Based on the above problems, there is an urgent need for an adaptive palletizing solution that can achieve dynamic force control adaptation, multi-source anomaly detection, and hierarchical processing. Summary of the Invention

[0004] This invention provides an adaptive palletizing and material re-grabbing method for an AGV binocular vision robotic arm, comprising the steps of AGV equipped with a binocular camera measuring the space of a van's cargo compartment, intelligent stacking pattern planning, and the robotic arm performing palletizing operations. The method further includes: collecting physical characteristic parameters of the material to be stacked and real-time vibration parameters of the cargo compartment; calculating the dynamic gripping force of the robotic arm based on the physical characteristic parameters and vibration parameters; acquiring depth data of the stacked material through the binocular camera and calculating anomaly detection confidence based on the dynamic gripping force; determining whether the stacked material has tilting or material drop anomalies based on the anomaly detection confidence; if anomalies are found, calculating a reset path correction coefficient; performing corresponding anomaly handling operations based on the reset path correction coefficient, including automatic reset of anomalies within the field of view and remote linkage of anomalies outside the field of view; after anomaly handling is completed, confirming the material status through the binocular camera, restarting the robotic arm palletizing operation, and recording the anomaly handling data.

[0005] Preferably, the physical characteristic parameters include material density, material volume, material flexibility coefficient, and material deformation threshold; the real-time vibration parameters of the cargo box include cargo box vibration frequency and vibration influence coefficient. When collecting the physical characteristic parameters, the material density and material volume are obtained through the material information input module, and the material flexibility coefficient and material deformation threshold are obtained by pressing the material surface through a pressure sensor. When collecting the real-time vibration parameters of the cargo box, the cargo box vibration frequency is obtained through a vibration sensor installed at the bottom of the AGV, and the vibration influence coefficient is preset according to the cargo box material.

[0006] Preferably, before calculating the dynamic gripping force, the six-dimensional force sensor at the end of the robotic arm needs to be zero-point calibrated to eliminate the interference of the robotic arm's own weight and the weight of the gripper on the force value detection. After calibration, the robotic arm gripper is switched according to the material classification results output by the intelligent stacking plan. Vacuum suction cup grippers are used for conventional materials, and flexible mechanical grippers are used for special materials. The gripper switching is completed by a pneumatic switching device. After switching, the coaxiality of the gripper installation is detected by a laser displacement sensor.

[0007] Preferably, during the anomaly detection process, the shooting angle of the binocular camera is at a 30° angle to the horizontal direction, the shooting frame rate is set to 15 FPS, and the image resolution is 1920×1080 pixels; after the binocular camera acquires the three-dimensional point cloud data of the stacked material, it extracts the vertical height difference between two diagonal points on the top edge of the material, and combines it with the real-time feedback value of the dynamic gripping force to input the anomaly detection confidence calculation model.

[0008] Preferably, the AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 1 is characterized in that the dynamic gripping force is calculated using the following formula:

[0009] ;

[0010] In the formula, This is a dynamic grasping force, measured in Newtons (N). This refers to the density of the material, expressed in kilograms per cubic meter (kg / m³). The volume of the material is expressed in cubic meters (m³). The acceleration due to gravity is taken as 9.8 meters per second squared (m / s²). This is the material flexibility coefficient, dimensionless, and its value range is determined by the material composition. For flexible materials, the value is 0.3-0.8, and for rigid materials, the value is 0.05-0.2. This is the material deformation threshold, expressed in meters (m), representing the maximum allowable deformation of the material. The vibration influence coefficient is dimensionless and determined by the material of the cargo box. The value is 0.1-0.3 for metal cargo boxes and 0.4-0.6 for non-metal cargo boxes. The vibration frequency of the cargo compartment is expressed in Hertz (Hz).

[0011] Preferably, the following formula is used to calculate the anomaly detection confidence level:

[0012] ;

[0013] In the formula, This represents the anomaly detection confidence level, which is dimensionless and ranges from 0 to 1. The closer the value is to 1, the more stable the material state. The depth of field value of the material top surface measured by the binocular camera is in meters (m). The reference depth value of the top surface of the material is preset for intelligent stacking planning, in meters (m). This is a dynamic grasping force, measured in Newtons (N). The reference gripping force for the corresponding material is expressed in Newtons (N) and is preset based on the material weight and coefficient of friction. This is the camera calibration coefficient, dimensionless, determined by the calibration accuracy of the binocular camera, with a value range of 0.9-1.0.

[0014] Preferably, the following formula is used to calculate the reset path correction coefficient:

[0015] ;

[0016] In the formula, This is the reset path correction coefficient, dimensionless. A larger value indicates a greater path correction magnitude; The confidence level for anomaly detection is dimensionless. This is the material drop distance coefficient, dimensionless, determined by the area of ​​the cargo compartment where the material drop point is located. The value is 0.6-0.9 in the edge area of ​​the cargo compartment and 0.2-0.5 in the middle area of ​​the cargo compartment. The straight-line distance between the material dropping point and the preset stacking position is expressed in meters (m).

[0017] Preferably, when performing the automatic reset of anomalies within the field of view, the binocular camera first continuously acquires 10 frames of images of the abnormal area, and reconstructs a three-dimensional model of the abnormal material using point cloud stitching technology. The center coordinates and attitude angle of the material's top surface are extracted, and the displacement and angle correction required for reset are calculated. Then, a reset path is planned based on the reset path correction coefficient, and the minimum distance between the path and the surrounding normal materials must meet a preset safety threshold. Finally, the robotic arm grabs the abnormal material according to the planned path, places it at the preset coordinates, and the binocular camera continuously captures 2 frames of images to confirm the material's status.

[0018] Preferably, when performing the remote linkage for out-of-view anomalies, the AGV's onboard industrial control computer generates standardized alarm information, including the anomaly type, the abnormal material ID, the material's basic parameters, the AGV's real-time location, and a photo of the abnormal area. The alarm information is transmitted to the remote control tablet via a 5G network. The customized APP on the remote control tablet displays an audible and visual alarm and marks the AGV's location and the abnormal area on the map interface. The operator selects to skip the anomaly or intervene manually through the APP. After receiving the instruction, the system executes the corresponding operation and records the processing result.

[0019] Preferably, before restarting the robotic arm's palletizing operation, the entire palletized area needs to be scanned by the binocular camera to confirm that there are no other potential anomalies. After restarting, the robotic arm continues to perform palletizing operations according to the original pallet pattern plan. At the same time, the on-board industrial control computer records the dynamic gripping force, anomaly detection confidence level, and reset path correction coefficient in real time during the palletizing process, forming an operation data log. The operation data log can be exported for subsequent parameter optimization and fault tracing.

[0020] Technical effects:

[0021] This invention addresses the core issues of existing technologies, such as their inability to adapt to the coupling effects of material flexibility and cargo box vibration, low anomaly detection accuracy, and passive handling of material spillage, through a three-level collaborative mechanism of dynamic gripping force calculation, multi-source anomaly confidence assessment, and graded path correction. The dynamic gripping force formula integrates material physical properties and environmental vibration parameters to prevent damage to special materials due to improper force control; the anomaly detection confidence score combines visual and force control data to improve anomaly identification accuracy; and the reset path correction coefficient enables personalized path planning, ensuring safe reset. The overall solution improves palletizing stability and the safety of special materials, reduces manual intervention, and meets the high-efficiency operation requirements of automated logistics. Attached Figure Description

[0022] Figure 1 This is a flowchart of the adaptive palletizing and material re-grabbing method for the AGV binocular vision robotic arm in this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Traditional technical solutions have the following technical problems: existing AGV robotic arm palletizing methods use fixed gripping force and single vision detection, which cannot dynamically adapt to the coupled effects of material flexibility deformation and cargo box vibration, resulting in a high breakage rate for special materials; anomaly detection relies solely on visual data, ignoring force control feedback, and is prone to missing transient anomalies; there is no graded mechanism for handling dropped materials, relying entirely on manual labor, resulting in poor operational continuity.

[0025] Based on this, please refer to Figure 1 This embodiment provides an adaptive palletizing and material re-grabbing method for an AGV binocular vision robotic arm, including:

[0026] S1: The AGV is equipped with a binocular camera to measure the space of the van cargo box, intelligently plan the stacking pattern, and have the robotic arm perform the palletizing operation.

[0027] S2: Collect the physical property parameters of the material to be stacked and the real-time vibration parameters of the cargo box; calculate the dynamic gripping force of the robotic arm based on the physical property parameters and the vibration parameters;

[0028] S3: Obtain depth data of the stacked materials through the binocular camera, and calculate the anomaly detection confidence level by combining the dynamic gripping force;

[0029] S4: Determine whether the stacked materials are tilted or falling off based on the anomaly detection confidence level. If an anomaly is found, calculate the reset path correction coefficient.

[0030] S5: Execute the corresponding exception handling operation based on the reset path correction coefficient. The exception handling operation includes automatic reset of exceptions within the field of view and remote linkage of exceptions outside the field of view.

[0031] S6: After the anomaly is handled, the material status is confirmed through the binocular camera, the robotic arm palletizing operation is restarted, and the anomaly handling data is recorded.

[0032] The core implementation process of this technical solution requires the collaboration of multiple modules: First, after the AGV stops at the cargo compartment, the binocular camera completes the 3D modeling of the cargo compartment space and outputs the stacking plan results; then, the material characteristic acquisition module obtains the material density, volume, flexibility coefficient, and deformation threshold through pressure sensors and information input interface, and the vibration sensor at the bottom of the AGV collects the vibration frequency of the cargo compartment in real time, transmitting these parameters to the on-board industrial control computer; the industrial control computer calls the dynamic force control algorithm to calculate the dynamic gripping force, and at the same time, the robotic arm switches the corresponding fixture, vacuum suction cup or flexible gripper according to the stacking plan, and completes the zero-point calibration of the force sensor; during the stacking process, the binocular camera continuously acquires the depth data of the stacked materials, and inputs it into the anomaly confidence model in sync with the dynamic gripping force data. If the confidence is lower than the preset threshold, an anomaly is determined, and the reset path correction coefficient is further calculated; for anomalies within the field of view, the robotic arm executes the reset according to the corrected path, and for anomalies outside the field of view, a 5G remote alarm is triggered; after the reset or manual handling, the binocular camera confirms the material status again. If there are no anomalies, the stacking is restarted, and at the same time, the industrial control computer records all operation data. The entire process achieves closed-loop control, encompassing parameter acquisition, dynamic adaptation, anomaly detection, tiered processing, and job recovery. Real-time data exchange between modules ensures that each step is dynamically adjusted based on the actual material and environmental conditions. This solution addresses the rigid adaptation and passive processing issues of traditional methods through multi-source data fusion and tiered processing, thereby improving palletizing stability and operational efficiency.

[0033] Traditional technical solutions have the following technical problems: When collecting material and environmental parameters, existing methods only obtain basic parameters such as material weight, ignoring key characteristics such as flexibility coefficient and deformation threshold, and do not quantify the influence coefficient of cargo box vibration, resulting in incomplete parameter dimensions and inability to support dynamic force control calculation; the parameter collection methods are scattered, requiring manual input of most parameters, which is inefficient and prone to errors.

[0034] Based on this, the physical characteristic parameters include material density, material volume, material flexibility coefficient, and material deformation threshold; the real-time vibration parameters of the cargo box include cargo box vibration frequency and vibration influence coefficient. When collecting the physical characteristic parameters, the material density and material volume are obtained through the material information input module, and the material flexibility coefficient and material deformation threshold are obtained by pressing the material surface through a pressure sensor. When collecting the real-time vibration parameters of the cargo box, the cargo box vibration frequency is obtained through a vibration sensor installed at the bottom of the AGV, and the vibration influence coefficient is preset according to the cargo box material.

[0035] The parameter acquisition process of this technical solution needs to be implemented in two steps: The first step is to collect the physical properties of the material. The material information input module integrates a touch screen and barcode scanning function. The operator can scan the material packaging barcode to automatically import the density and volume. If there is no barcode, it can be manually entered. The pressure sensor is installed on the side of the gripper at the end of the robotic arm. Before gripping the material, the sensor will gradually press the surface of the material with a pressure gradient of 0.5N and record the curve of the material deformation as a function of pressure. When the deformation reaches the preset upper limit, the corresponding pressure value and deformation are the basis for calculating the flexibility coefficient and deformation threshold. The data is automatically transmitted to the industrial control computer. The second step is the acquisition of vibration parameters for the cargo compartment. A three-axis vibration sensor installed at the bottom of the AGV has a sampling frequency set to 100Hz, continuously collecting vibration data from the cargo compartment in the X, Y, and Z directions. The dominant vibration frequency is extracted using Fourier transform and used as the cargo compartment's vibration frequency. The vibration influence coefficient is preset based on the cargo compartment material. For metal cargo compartments, due to their high rigidity and rapid vibration decay, the coefficient is set to 0.1-0.3. For non-metallic cargo compartments, due to their lower rigidity and larger vibration influence range, the coefficient is set to 0.4-0.6. These preset values ​​are stored in the system configuration library, and operators can select the corresponding coefficient based on the actual cargo compartment material. The entire parameter acquisition process combines automation and semi-automation. Key material characteristics are automatically detected by sensors, and basic information is quickly imported via barcodes. Vibration parameters require no manual intervention, ensuring both parameter integrity and accuracy while improving acquisition efficiency. This provides comprehensive and reliable data support for subsequent dynamic grasping force calculations, avoiding force control adaptation deviations caused by missing parameters.

[0036] Traditional technical solutions have the following problems: existing robotic arm force sensors have not undergone targeted zero-point calibration, and the weight of the robotic arm itself and the weight of the gripper will cause a fixed deviation in force detection, affecting the accuracy of gripping force control; gripper switching relies on manual operation, and the coaxiality of the installation is not checked after switching, which can easily lead to gripping posture deviation, thereby causing material tilting or falling; gripper type selection is based only on material weight and does not take into account the classification results of stacking planning, which cannot adapt to the flexibility requirements of special materials.

[0037] Based on this, before calculating the dynamic gripping force, the six-dimensional force sensor at the end of the robotic arm needs to be zero-point calibrated to eliminate the interference of the robotic arm's own weight and the weight of the gripper on the force value detection. After calibration, the robotic arm gripper is switched according to the material classification results output by the intelligent stacking plan. Vacuum suction cup grippers are used for conventional materials, and flexible mechanical grippers are used for special materials. The gripper switching is completed by a pneumatic switching device. After switching, the coaxiality of the gripper installation is detected by a laser displacement sensor.

[0038] The fixture and force sensor preprocessing process of this technical solution must strictly execute three steps: The first step is the six-axis force sensor zero calibration. The robotic arm first moves the fixture to the preset calibration position to ensure that there is no external force acting on the fixture. At this time, the industrial control computer sends a calibration instruction to the force sensor, and the sensor records the current force value and torque data as the zero point reference. All subsequent detected force values need to subtract this reference value to eliminate the deviation caused by the weight of the robotic arm forearm, wrist, and the fixture itself. The calibration process takes about 5 seconds, and the system prompts that force control is ready after calibration. The second step is fixture switching. The intelligent stacking pattern planning will classify the materials into regular materials and special materials according to the material characteristics and send a switching instruction to the pneumatic switching device; the pneumatic device controls the cylinder action through the solenoid valve to achieve the rapid switching between the vacuum suction cup and the flexible gripper, and the switching time ≤ 3 seconds; among them, the vacuum suction cup uses a sponge suction cup, which is suitable for flat rigid materials, and the flexible gripper uses a silicone material, and there are anti-slip patterns on the inner side of the gripper, which is suitable for special materials that are prone to deformation or have a fragile surface. The third step is fixture coaxiality detection. The laser displacement sensor is installed on the side of the AGV vehicle body, facing the central axis of the fixture. After the switching is completed, the sensor emits a laser beam to detect the deviation value between the center of the fixture and the rotation center of the end of the robotic arm. If the deviation ≤ 0.02 mm, it is judged to be qualified. If the deviation exceeds the standard, the system automatically fine-tunes the joint angle of the robotic arm until the coaxiality meets the requirements. The entire preprocessing process ensures accurate detection of the force sensor, proper adaptation of the fixture type, and accurate installation position, laying a foundation for subsequent dynamic grasping force control and precise stacking, and avoiding material damage and abnormalities caused by force value deviation or pose offset.

[0039] The traditional technical solution has the following technical problems: The installation angle and shooting parameters of the existing binocular cameras are fixed, and they cannot cover the materials in the middle and upper layers of the cargo compartment at the same time, easily forming a blind spot in the field of view; The abnormal detection only relies on the height difference judgment of the three-dimensional point cloud and does not combine the force control data, resulting in low sensitivity and high misjudgment rate for the identification of slight inclination or potential material dropping; The selection of image resolution and frame rate is unreasonable. High resolution leads to data processing delay, and low frame rate cannot capture transient abnormalities.

[0040] Based on this, during the abnormal detection process, the shooting angle of the binocular camera forms a 30° angle with the horizontal direction, the shooting frame rate is set to 15 FPS, and the image resolution is 1920×1080 pixels; After the binocular camera obtains the three-dimensional point cloud data of the stacked materials, it extracts the vertical height difference between the two diagonal points at the edge of the top surface of the material, and combines the real-time feedback value of the dynamic grasping force, and inputs them into the abnormal detection confidence calculation model together.

[0041] The implementation of this technical solution for anomaly detection needs to be carried out from two aspects: hardware parameter settings and data fusion. On the hardware level, the binocular camera adopts an industrial-grade camera with a resolution of 1920×1080 pixels. This resolution can ensure clear image details without causing excessive data volume. The shooting angle is at a 30° angle to the horizontal direction and is fixed to the top of the AGV by a camera bracket. This angle can cover an area of ​​1.5-3m from the bottom of the cargo box upwards, avoiding blind spots in the field of vision of the middle and upper layers of materials. The camera calibration cycle is once a week to ensure the accuracy of depth measurement. At the data processing level, each frame of the binocular camera generates 3D point cloud data of the corresponding material. The system extracts the Z-axis coordinates of two diagonal points on the top edge of the material and calculates the vertical height difference between the two points. If the height difference exceeds 10% of the material's height, a preliminary assessment indicates a tilting trend. Simultaneously, a six-dimensional force sensor at the end of the robotic arm provides real-time feedback on the fluctuation value of the dynamic grasping force. If the fluctuation exceeds 10%, it indicates potential contact instability. The height difference data and force fluctuation data are simultaneously input into the anomaly detection confidence calculation model. The model combines both parameters to output a confidence value, avoiding misjudgments caused by a single parameter. For example, when the height difference slightly exceeds the threshold but the force value is stable, the confidence value may still be higher than the threshold, and it is judged as normal; when the height difference is normal but the force value fluctuates drastically, the confidence value decreases, and it is judged as a potential anomaly. This solution improves the coverage and recognition accuracy of anomaly detection and reduces false positives and false negatives by optimizing camera parameters and fusing multi-source data.

[0042] Traditional technical solutions have the following problems: Existing robotic arm gripping force calculations are based solely on material weight, using fixed values, without considering the material's flexible deformation characteristics—flexible materials are easily damaged by excessive force, while rigid materials are easily detached by insufficient force; the impact of cargo box vibration is also not quantified, as vibration can lead to an increase in the actual required gripping force, and fixed force values ​​cannot adapt to dynamic environments, resulting in poor palletizing stability; gripping force calculations lack a clear mathematical model, relying only on empirical values, resulting in poor adaptability.

[0043] Based on this, the following formula is used to calculate the dynamic gripping force:

[0044] ;

[0045] In the formula, This is a dynamic grasping force, measured in Newtons (N). This refers to the density of the material, expressed in kilograms per cubic meter (kg / m³). The volume of the material is expressed in cubic meters (m³). The acceleration due to gravity is taken as 9.8 meters per second squared (m / s²). This is the material flexibility coefficient, dimensionless, and its value range is determined by the material composition. For flexible materials, the value is 0.3-0.8, and for rigid materials, the value is 0.05-0.2. This is the material deformation threshold, expressed in meters (m), representing the maximum allowable deformation of the material. The vibration influence coefficient is dimensionless and determined by the material of the cargo box. The value is 0.1-0.3 for metal cargo boxes and 0.4-0.6 for non-metal cargo boxes. The vibration frequency of the cargo compartment is expressed in Hertz (Hz).

[0046] The dynamic gripping force calculation of this technical solution needs to be implemented in three steps: the first step is to input the basic parameters, which are first obtained through the material information acquisition module. and Multiply the two together to get the material mass, then multiply by... Obtain the weight of materials, such as the weight of a cardboard box. , Therefore, the basic gripping force is 500 × 0.02 × 9.8 = 98 N. The second step is material flexibility correction, which is determined based on the material composition. and Flexible materials , The correction factor is then 1 + 0.6 × 0.005 = 1.003. Multiplying the base gripping force by this factor, it becomes 98 × 1.003 ≈ 98.29 N. This ensures the gripping force is slightly greater than the base value to avoid excessive deformation. (For rigid materials...) , The correction factor is 1 + 0.1 × 0.001 = 1.0001, ensuring the gripping force is close to the base value to avoid excessive force damaging the material. The third step is vibration impact correction, determined based on the material of the cargo box. Metal cargo box ,like Then the exponent term is ,final If the cargo box is made of non-metallic material, , The exponent term is , This formula achieves dynamic adaptation of the gripping force through three levels of calculation: basic gravity, flexibility correction, and vibration correction. This ensures that the material is not damaged while also meeting the gripping requirements in vibrating environments. The calculation process is completed by the real-time calculation engine of the onboard industrial control computer, with parameters updated every 0.1 seconds to ensure that the gripping force is dynamically adjusted according to the environment and material conditions, solving the adaptation problem of traditional fixed force values.

[0047] Traditional technical solutions have the following technical problems: existing anomaly detection relies solely on visual data and does not incorporate force control data, resulting in the inability to identify potential problems with normal visual appearance but abnormal force values; anomaly judgment lacks quantitative standards and relies solely on threshold comparisons, which cannot reflect the severity of the anomaly and is not conducive to subsequent graded processing; the impact of camera calibration accuracy is not considered, resulting in poor consistency of detection results under different camera accuracies and a high false positive rate.

[0048] Based on this, the following formula is used to calculate the anomaly detection confidence level:

[0049] ;

[0050] In the formula, This represents the anomaly detection confidence level, which is dimensionless and ranges from 0 to 1. The closer the value is to 1, the more stable the material state. The depth of field value of the material top surface measured by the binocular camera is in meters (m). The reference depth value of the top surface of the material is preset for intelligent stacking planning, in meters (m). This is a dynamic grasping force, measured in Newtons (N). The reference gripping force for the corresponding material is expressed in Newtons (N) and is preset based on the material weight and coefficient of friction. This is the camera calibration coefficient, dimensionless, determined by the calibration accuracy of the binocular camera, with a value range of 0.9-1.0.

[0051] The anomaly detection confidence calculation of this technical solution requires the integration of visual and force control data. The implementation process is as follows: The first step is visual data calculation, where a binocular camera measures the weight of a stacked material. The stack type planning is preset Then the visual term is 1 - |1.205 - 1.2| / 1.2 = 1 - 0.005 / 1.2 ≈ 0.9958. The visual value is 1 - 0.02 / 1.2 ≈ 0.9833. A smaller visual value indicates a more severe tilting or displacement of the material. The second step is force control data calculation. Preset Based on the material weight and friction coefficient, the minimum force value to ensure stable gripping is calculated. Therefore, the force control term is 1 - |65.9 - 68| / 68 = 1 - 2.1 / 68 ≈ 0.9691. The force control term is approximately 1-8 / 68≈0.8824. A smaller force control term indicates a larger deviation in gripping force, making it easier for material to fall off. The third step is camera calibration coefficient correction. If the binocular camera has just been calibrated and has high accuracy, If the calibration time exceeds one week, the accuracy will decrease slightly. .final Under normal circumstances, if and , Potential anomalies, if and , Identifying anomalies. This formula achieves multi-source data fusion through the product of visual and force control operations, avoiding the limitations of single-source data; The introduction of this feature ensures comparability of confidence levels across different camera accuracies, reducing misjudgments caused by equipment differences. The confidence level quantifies the severity of the anomaly. This is normal. This is a potential anomaly. To clearly identify anomalies and provide quantitative basis for subsequent graded processing, this addresses the problems of lack of standards and inaccuracy in traditional anomaly judgment.

[0052] Traditional technical solutions have the following problems: Existing material drop reset path planning uses a fixed algorithm, which does not consider the severity of the anomaly. It uses the same path for minor and severe anomalies, resulting in overcorrection for minor anomalies and undercorrection for severe anomalies; it does not take into account the differences in the cargo compartment area where the material drop point is located. The space at the edge of the cargo compartment is narrow, and different path correction strategies are required for the central area. Fixed paths are prone to collisions with surrounding materials; the path planning does not have a clear correction coefficient and relies on experience for adjustment, resulting in poor adaptability.

[0053] Based on this, the following formula is used to calculate the reset path correction coefficient:

[0054] ;

[0055] In the formula, This is the reset path correction coefficient, dimensionless. A larger value indicates a greater path correction magnitude; The confidence level for anomaly detection is dimensionless. This is the material drop distance coefficient, dimensionless, determined by the area of ​​the cargo compartment where the material drop point is located. The value is 0.6-0.9 in the edge area of ​​the cargo compartment and 0.2-0.5 in the middle area of ​​the cargo compartment. The straight-line distance between the material dropping point and the preset stacking position is expressed in meters (m).

[0056] The calculation of the reset path correction coefficient for this technical solution needs to consider both the severity of the anomaly and the characteristics of the material dropping area. The implementation process is as follows: The first step is to correct the severity of the anomaly. (If the anomaly is clearly identified), then 1 / , (Potential anomaly), 1 / , The smaller the value (the more severe the anomaly), the larger the base value for correction, ensuring that severe anomalies are fully corrected. The second step is to correct the material drop area and distance. If the material drop point is in the edge area of ​​the cargo compartment, , Then the area distance term is 1 + 0.8 × 0.05 = 1.04; if the material drop point is in the central area, , The regional distance term is 1 + 0.3 × 0.08 = 1.024. The third step is to calculate the final... , edge area and hour, This indicates that the path needs significant correction, such as increasing the elevation or decreasing the horizontal movement speed; in the central region and hour, The path correction range is relatively small. The path planning module is based on... Adjust path parameters: The larger the value, the higher the robotic arm can lift, the slower its movement speed, and the smaller the step size of its joint rotation angle. For example, At the same time, the lifting height increased from the usual 0.02m to 0.03m, and the moving speed decreased from 0.2m / s to 0.15m / s; At this time, the lifting height is maintained at 0.02m, and the speed is reduced to 0.18m / s. This formula achieves personalized correction of the reset path through the collaborative calculation of the anomaly confidence level and the distance of the material drop area, ensuring that the path is safe and accurate under different abnormal scenarios, solving the adaptation problem of traditional fixed paths, and reducing the risk of collision during the reset process.

[0057] Traditional technical solutions have the following technical problems: existing abnormal reset methods only involve simple grabbing and putting back operations, without reconstructing a 3D model of the abnormal material, making it impossible to accurately obtain the material's posture, which may result in the material still tilting after reset; the reset path does not take into account the surrounding normal materials, making collisions easy; and there is no secondary confirmation mechanism after reset, which cannot ensure the reset effect and may leave abnormalities.

[0058] Based on this, when performing the automatic reset of abnormalities within the field of view, the binocular camera first continuously acquires 10 frames of images of the abnormal area, and reconstructs a three-dimensional model of the abnormal material using point cloud stitching technology, extracting the center coordinates and attitude angle of the material's top surface, and calculating the displacement and angle correction required for reset; then, a reset path is planned based on the reset path correction coefficient, and the minimum distance between the path and the surrounding normal materials must meet a preset safety threshold; finally, the robotic arm grabs the abnormal material according to the planned path, places it at the preset coordinates, and the binocular camera continuously captures 2 frames of images to confirm the material's status.

[0059] The implementation of automatic reset of anomalies within the field of view in this technical solution requires three stages: The first stage is the modeling and parameter calculation of the abnormal material. The binocular camera continuously acquires 10 frames of images of the abnormal area, with an interval of 0.067 seconds between each frame. The system fuses the 10 frames of point cloud data through point cloud stitching technology to reconstruct a complete three-dimensional model of the abnormal material with a model accuracy of up to 0.001m. The center coordinates of the top surface of the material (e.g., X=2.5m, Y=1.8m, Z=1.2m) and the attitude angle (e.g., tilt angle 3°) are extracted from the model and compared with the preset coordinates (X=2.502m, Y=1.8m, Z=1.2m) to calculate the displacement ΔX=-0.002m, ΔY=0m, ΔZ=0m, and the angle correction Δθ=-3°.

[0060] The second stage is reset path planning, based on the reset path correction coefficient. The path planning module treats surrounding normal materials as obstacles, planning a path that first lifts, then adjusts the angle, and finally translates: the robotic arm first lifts the abnormal material by 0.03m to avoid friction with the material below; then it adjusts the wrist angle to correct the material's tilt angle to 0°; finally, it translates to the preset coordinates, with the minimum distance between the entire path and surrounding normal materials set at 0.05m to ensure no collision risk. The third stage is reset execution and confirmation. The robotic arm grabs the abnormal material according to the planned path, places it at the preset coordinates, and holds the placement posture for 1 second to ensure the material smoothly contacts the material below; then, the binocular camera continuously captures two frames of images, detecting that the deviation of the center coordinates of the material's top surface is ≤0.001m and the tilt angle is ≤0.5°, indicating a successful reset; if the deviation or tilt angle exceeds the threshold, the reset process is re-executed. If two consecutive resets fail, it is upgraded to a difficult abnormality, triggering a remote alarm. This solution, through precise positioning via 3D modeling, personalized path planning, and secondary confirmation, ensures that abnormalities within the field of view can be reset efficiently and accurately, avoiding the coarseness and uncertainty of traditional reset operations.

[0061] Traditional technical solutions have the following technical problems: Existing out-of-view anomaly alarms only send simple material drop prompts without detailed information on the abnormal material and the AGV location. Operators cannot quickly determine the severity and location of the anomaly, resulting in low response efficiency; there is no remote interaction function, and operators need to go to the site to handle it, increasing labor costs and work interruption time; the results of anomaly handling are not recorded, making it impossible to trace and optimize subsequent operations.

[0062] Based on this, when performing remote linkage for out-of-view anomalies, the AGV's onboard industrial control computer generates standardized alarm information, including the anomaly type, the abnormal material ID, the material's basic parameters, the AGV's real-time location, and a photo of the abnormal area. The alarm information is transmitted to the remote control tablet via a 5G network. The customized APP on the remote control tablet displays an audible and visual alarm and marks the AGV's location and the abnormal area on the map interface. The operator can choose to skip the anomaly or intervene manually through the APP. After receiving the instruction, the system executes the corresponding operation and records the processing result.

[0063] The implementation of remote linkage for out-of-view anomalies in this technical solution requires the realization of a complete process of alarm generation, transmission, interaction, and recording: The first stage is alarm information generation. When the on-board industrial control computer detects that a certain material has not appeared in the camera's field of view for 5 consecutive frames and its coordinates are outside the field of view, it is determined to be an out-of-view anomaly. The system automatically generates alarm information, including: anomaly type is material dropping, the abnormal material ID is E005, the basic parameters of the material are: type: electronic product, size: 0.15m×0.1m×0.05m, weight: 0.75kg, the AGV's real-time position is latitude and longitude N23°06′, E113°17′, error ±0.1m, and the on-site photo of the abnormal area is the last frame containing the material. The second stage is alarm information transmission and notification. The alarm information is transmitted to the remote control tablet via the 5G network with a transmission delay of ≤1 second. The customized APP on the tablet immediately pops up an audible and visual alarm notification, and the APP map interface automatically locates the AGV's position, marking the abnormal area with a red circle. The operator can click on the mark to view the complete alarm information and on-site photos. The third stage involves remote interaction and recording. The operator determines the issue based on material parameters: if the material is a common electronic product with low value, the operator chooses to skip the anomaly, and the app sends a command to the AGV. The system then marks the material as pending processing and restarts palletizing. If the material is a high-value precision instrument, the operator chooses manual intervention. After the command is sent, the AGV remains stationary, waiting for the operator to arrive. After the operator completes the on-site handling, they click "Anomaly Resolved" through the app. The system records the result, such as manually picking up and re-stacking the material, which takes 3 minutes and is then stored in the work data log. This solution addresses the problems of insufficient information, slow response, and lack of traceability in traditional off-site anomaly handling by standardizing alarm information, enabling real-time remote interaction, and recording results. It improves anomaly handling efficiency and reduces the frequency of manual on-site intervention.

[0064] Traditional technical solutions have the following technical problems: the existing palletizing operation does not fully scan the already stacked area before restarting, which may miss other potential anomalies and cause the problem to recur after restarting; there are no complete data records during the operation, making it impossible to trace the cause of the anomaly, which is not conducive to subsequent parameter optimization; the data logs cannot be exported and are only stored locally, making it impossible to realize the summary analysis of data from multiple devices.

[0065] Based on this, before restarting the robotic arm's palletizing operation, the entire palletized area needs to be scanned by the binocular camera to confirm that there are no other potential anomalies. After restarting, the robotic arm continues to perform palletizing operations according to the original pallet pattern plan. At the same time, the on-board industrial control computer records the dynamic gripping force, anomaly detection confidence level, and reset path correction coefficient in real time during the palletizing process, forming an operation data log. The operation data log can be exported for subsequent parameter optimization and fault tracing.

[0066] The implementation of this technical solution for operation recovery and data recording requires two stages: The first stage is a comprehensive inspection and confirmation before restarting. After the anomaly handling is completed, whether it is a reset within the field of view or manual intervention outside the field of view, the binocular camera no longer only photographs the abnormal area, but also scans the entire stacked area through the movement of the AGV. The scanning path is S-shaped, covering all stacking stations in the cargo compartment. During the scanning process, the camera takes one frame of image every 0.5 seconds, generating three-dimensional point cloud data of the entire area. The system compares the point cloud data with the preset stacking model to detect whether there is any tilting, displacement or dropping of other materials. If the anomaly detection confidence of all materials is ≥0.9, it is determined that there is no potential anomaly, and stacking is allowed to restart. If there are materials with a confidence of <0.9, the anomaly handling process is repeated for those materials until there are no anomalies in the entire area. The second stage is operation data recording and export. After restarting stacking, the on-board industrial control computer records the dynamic gripping force of each batch of materials in real time at 0.1-second intervals. ), Anomaly detection confidence ( ), reset path correction factor ( Simultaneously, it records information such as the time of the anomaly, the handling method, and the processing time, forming a structured operation data log (in CSV format); the log content includes "timestamp, material ID, ... , , Fields such as "Abnormality Type, Processing Result, and Time Consumed" are included. After the operation is completed, the operator can export the log to a computer via USB interface or wireless transmission, and use data analysis software to analyze the rationality of the parameters, such as if a certain material appears multiple times. Too low, can be adjusted. or The solution optimizes subsequent stacking parameters and backs up logs locally for at least 3 months to meet fault tracing requirements. This approach ensures safe job recovery through comprehensive testing and supports subsequent optimization through complete data recording and export. It addresses the high restart risk and lack of data support for optimization issues inherent in traditional solutions, thereby improving job reliability and continuous improvement capabilities.

[0067] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for adaptive palletizing and re-grabbing of dropped materials using an AGV binocular vision robotic arm, comprising the steps of AGV equipped with a binocular camera to perform spatial measurement of the cargo compartment of a van, intelligent palletization planning, and the robotic arm performing palletizing operations, characterized in that, Also includes: Collect the physical property parameters of the materials to be stacked and the real-time vibration parameters of the cargo box; The dynamic gripping force of the robotic arm is calculated based on the physical characteristic parameters and the vibration parameters. The binocular camera acquires depth data of the stacked materials, and the anomaly detection confidence level is calculated based on the dynamic gripping force. The anomaly detection confidence level is used to determine if the stacked materials are tilted or falling. If an anomaly is found, a reset path correction coefficient is calculated. Based on the reset path correction coefficient, corresponding anomaly handling operations are performed, including automatic reset of anomalies within the field of view and remote linkage for anomalies outside the field of view. After anomaly handling is completed, the binocular camera is used to confirm the material status, the robotic arm palletizing operation is restarted, and the anomaly handling data is recorded.

2. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 1, characterized in that, The physical characteristic parameters include material density, material volume, material flexibility coefficient, and material deformation threshold. The real-time vibration parameters of the cargo box include the cargo box vibration frequency and vibration influence coefficient. When collecting the physical characteristic parameters, the material density and material volume are obtained through the material information input module, and the material flexibility coefficient and material deformation threshold are obtained by pressing the material surface through a pressure sensor. When collecting the real-time vibration parameters of the cargo box, the cargo box vibration frequency is obtained through a vibration sensor installed at the bottom of the AGV, and the vibration influence coefficient is preset according to the cargo box material.

3. The adaptive palletizing and material re-grabbing method for AGV binocular vision robotic arm according to claim 1, characterized in that, Before calculating the dynamic gripping force, the six-dimensional force sensor at the end of the robotic arm needs to be zero-point calibrated to eliminate the interference of the robotic arm's own weight and the weight of the gripper on the force value detection. After calibration, the robotic arm gripper is switched according to the material classification results output by the intelligent stacking plan. Vacuum suction cup grippers are used for conventional materials, while flexible mechanical grippers are used for special materials. The gripper switching is completed by a pneumatic switching device. After switching, the coaxiality of the gripper installation is detected by a laser displacement sensor.

4. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 1, characterized in that, During the anomaly detection process, the binocular camera's shooting angle is at a 30° angle to the horizontal direction, the shooting frame rate is set to 15 FPS, and the image resolution is 1920×1080 pixels. After the binocular camera acquires the three-dimensional point cloud data of the stacked material, it extracts the vertical height difference between two diagonal points on the top edge of the material, and combines it with the real-time feedback value of the dynamic gripping force to input the anomaly detection confidence calculation model.

5. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 1, characterized in that, The following formula is used to calculate the dynamic gripping force: ； In the formula, This is a dynamic grasping force, measured in Newtons (N). This refers to the density of the material, expressed in kilograms per cubic meter (kg / m³). The volume of the material is expressed in cubic meters (m³). The acceleration due to gravity is taken as 9.8 meters per second squared (m / s²). This is the material flexibility coefficient, dimensionless, and its value range is determined by the material composition. For flexible materials, the value is 0.3-0.8, and for rigid materials, the value is 0.05-0.

2. This is the material deformation threshold, expressed in meters (m), representing the maximum allowable deformation of the material. The vibration influence coefficient is dimensionless and determined by the material of the cargo box. The value is 0.1-0.3 for metal cargo boxes and 0.4-0.6 for non-metal cargo boxes. The vibration frequency of the cargo compartment is expressed in Hertz (Hz).

6. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 5, characterized in that, The following formula is used to calculate the anomaly detection confidence level: ； In the formula, This represents the anomaly detection confidence level, which is dimensionless and ranges from 0 to 1. The closer the value is to 1, the more stable the material state. The depth of field value of the material top surface measured by the binocular camera is in meters (m). The reference depth value of the top surface of the material is preset for intelligent stacking planning, in meters (m). This is a dynamic grasping force, measured in Newtons (N). The reference gripping force for the corresponding material is expressed in Newtons (N) and is preset based on the material weight and coefficient of friction. This is the camera calibration coefficient, dimensionless, determined by the calibration accuracy of the binocular camera, with a value range of 0.9-1.

0.

7. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 6, characterized in that, The following formula is used to calculate the reset path correction coefficient: ； In the formula, This is the reset path correction coefficient, dimensionless. A larger value indicates a greater path correction magnitude; The confidence level for anomaly detection is dimensionless. This is the material drop distance coefficient, dimensionless, determined by the area of ​​the cargo compartment where the material drop point is located. The value is 0.6-0.9 in the edge area of ​​the cargo compartment and 0.2-0.5 in the middle area of ​​the cargo compartment. The straight-line distance between the material dropping point and the preset stacking position is expressed in meters (m).

8. The adaptive palletizing and re-grabbing method for AGV binocular vision robotic arms according to claim 1, characterized in that, When performing the automatic reset of abnormalities within the field of view, the binocular camera first continuously acquires 10 frames of images of the abnormal area, and reconstructs a three-dimensional model of the abnormal material using point cloud stitching technology. The center coordinates and attitude angle of the material's top surface are extracted, and the displacement and angle correction required for reset are calculated. Then, a reset path is planned based on the reset path correction coefficient, and the minimum distance between the path and the surrounding normal materials must meet a preset safety threshold. Finally, the robotic arm grabs the abnormal material according to the planned path, places it at the preset coordinates, and the binocular camera continuously captures 2 frames of images to confirm the material's status.

9. The adaptive palletizing and material re-grabbing method for AGV binocular vision robotic arm according to claim 1, characterized in that, When performing remote linkage for out-of-view anomalies, the AGV's onboard industrial control computer generates standardized alarm information, including the anomaly type, the abnormal material ID, the material's basic parameters, the AGV's real-time location, and a photo of the abnormal area. The alarm information is transmitted to the remote control tablet via a 5G network. The customized APP on the remote control tablet displays an audible and visual alarm and marks the AGV's location and the abnormal area on the map interface. The operator can choose to skip the anomaly or intervene manually through the APP. After receiving the instruction, the system executes the corresponding operation and records the processing result.

10. The AGV binocular vision robotic arm adaptive palletizing and material re-grabbing method according to claim 1, characterized in that, Before restarting the robotic arm's palletizing operation, the entire palletized area must be scanned using the binocular camera to confirm that there are no other potential anomalies. After restarting, the robotic arm continues to perform palletizing operations according to the original pallet pattern plan. At the same time, the on-board industrial control computer records the dynamic gripping force, anomaly detection confidence level, and reset path correction coefficient in real time during the palletizing process, forming an operation data log. The operation data log can be exported for subsequent parameter optimization and fault tracing.

Images