Ton bag hoisting unmanned control system based on binocular vision camera and laser radar

Abstract

The present application relates to the field of machine vision and perception technology, especially to a ton bag hoisting unmanned control system based on binocular vision camera and laser radar, comprising an intelligent control unit, a lifting tool actuator, a perception unit and a special ton bag; the perception unit contains a binocular vision camera and a laser radar, which collects depth vision and three-dimensional point cloud information in the operation area; the intelligent control unit fuses multi-source data, locates the lifting lug through an improved weighted multi-feature fusion algorithm, plans a safe path and generates phased instructions; the lifting tool actuator uses an electromagnetic adsorption module to pull down the collapsed lifting lug and a mechanical gripper module to clamp the lifting lug, and cooperates with a verification mechanism to realize reliable operation; the special ton bag is adapted to the perception and execution modules through high-contrast color and pre-buried metal parts. The present application realizes full-process unmanned operation, improves the robustness and safety of operation in complex environments, and adapts to the ton bag hoisting needs of multiple industries.

Description

Technical Field

[0001] This invention relates to the field of machine vision and perception technology, specifically to an unmanned control system for ton bag hoisting based on binocular vision cameras and lidar. Background Technology

[0002] In the fields of industrial production and warehousing logistics, ton bags serve as the primary packaging and transport carrier for bulk materials (such as chemical raw materials, building materials, and grains), and their hoisting operations are a crucial process connecting production, storage, and transportation. With the advancement of industrial automation and intelligence, traditional ton bag hoisting methods have gradually revealed problems such as low efficiency, poor safety, and reliance on manual labor, making it difficult to meet the demands of modern factories for efficient, safe, and unmanned operations.

[0003] Currently, the mainstream method for lifting ton bags still relies primarily on a combination of manual assistance and semi-automated equipment. In practice, this requires the coordinated efforts of multiple workers: some operate the crane to move the bag above it, while others manually attach the hooks to the bag's lifting lugs at close range. Once hooked, the crane operator is notified via walkie-talkie to begin the lifting operation. This method is highly dependent on manual labor and poses significant safety hazards. Workers operate in high-risk areas such as below the crane and between stacks of ton bags, increasing the risk of injury due to crane operator errors, accidental bag falls, or stack collapses. This is especially true in scenarios involving flammable, explosive, or toxic materials, such as those in the chemical or lithium battery industries, where the risk of manual intervention is even higher. Furthermore, the efficiency of manual hooking is affected by factors such as operator skill and physical condition, resulting in long lifting times per operation, making it unsuitable for the high-frequency, continuous operation demands of large-scale production.

[0004] Therefore, to address the above issues, an unmanned control system for ton bag hoisting based on binocular vision cameras and lidar is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide an unmanned control system for ton bag hoisting based on binocular vision cameras and lidar, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The unmanned control system for ton bag lifting based on binocular vision camera and lidar includes intelligent control unit, lifting device actuator, sensing unit and special ton bag;

[0008] The sensing unit includes a binocular vision camera and a lidar. The sensing unit is installed above the lifting device actuator and is used to collect depth visual information and three-dimensional point cloud information of the work area.

[0009] The intelligent control unit communicates with the sensing unit and the lifting device actuator to process and fuse multi-source sensing data and generate control commands.

[0010] The lifting device actuator includes an independently controllable electromagnetic adsorption module and a mechanical gripper module;

[0011] The lifting lugs of the special ton bag have embedded metal parts that can be magnetically attracted, and the color of the lifting lugs of the special ton bag has a high contrast with the main color of the bag body.

[0012] As a preferred embodiment, the intelligent control unit includes:

[0013] The data fusion module is configured to receive and fuse RGB-D images from a binocular vision camera and 3D point cloud data from a LiDAR, and generate a dense 3D environment reconstruction model with color information through calibration and registration.

[0014] The identification and positioning module is configured to accurately locate the three-dimensional coordinates and posture of the lug based on a dense three-dimensional environment reconstruction model and an improved weighted multi-feature fusion identification algorithm. The objective function of the improved weighted multi-feature fusion identification algorithm is expressed as: finding the point that minimizes the weighted sum of the matching degree between the Euclidean distance of the color feature and the geometric feature, minus the weighted value of the reflection intensity value. The weighting coefficients of the color feature, geometric feature, and reflection intensity feature are dynamically adjusted according to the sensor accuracy and environmental conditions.

[0015] The path planning module is configured to plan a safe movement path for the lifting device actuator based on the three-dimensional coordinates and attitude of the lifting lug and obstacle information in the three-dimensional environment reconstruction model.

[0016] The control decision module is configured to send phased collaborative control commands to the spreader actuator based on a safe movement path.

[0017] As a preferred option, the phased collaborative control instructions executed by the control decision module include, in sequence:

[0018] First-stage movement control: Control the movement of the lifting device actuator so that the center point of the electromagnetic adsorption module is aligned with the identified lifting lug target point in three-axis space;

[0019] Second stage adsorption control: After maintaining the position alignment, control the electromagnetic adsorption module to generate magnetic force, adsorb and lift the hanging lugs that have collapsed due to their own weight or stacking pressure to a natural hanging state.

[0020] The third stage of gripper control: After the lifting lug is attracted and lifted, the mechanical gripper module is controlled to perform the actions of opening, passing through the lifting lug, and closing and clamping.

[0021] The fourth stage of release and handling control: the electromagnetic adsorption module is de-energized to release the magnetic force, and at the same time the lifting device actuator is controlled to move the grabbed ton bag to the target location.

[0022] As a preferred approach, after the second-stage adsorption control, the sensing unit restarts to perform verification and identification. Only after confirming that the lug has been successfully adsorbed and lifted and is in a graspable state will the control decision module trigger the third-stage gripper control. Verification and identification are achieved by calculating the average height change rate of the point cloud in the lug area in the vertical direction before and after adsorption. When the height change rate is greater than the preset height change threshold, the adsorption and lifting are determined to be successful.

[0023] As a preferred option, the geometric feature matching degree of the improved weighted multi-feature fusion recognition algorithm in the identification and positioning module is obtained by calculating the weighted sum of the differences in normal vector features and curvature features, wherein the weighting factors of normal vector features and curvature features are used to adjust the contribution of the corresponding features.

[0024] As a preferred approach, the reflection intensity feature weight coefficient is dynamically correlated with the reflection intensity value. Specifically, it is expressed as the product of the basic reflection intensity weight value plus the gain coefficient and the portion of the average reflection intensity value that exceeds the reflection intensity threshold.

[0025] As a preferred option, the lidar data is also used for real-time scanning during the movement of the lifting device actuator, and the path planning module dynamically corrects the movement path based on the real-time point cloud data to achieve obstacle avoidance.

[0026] As a preferred option, the mechanical gripper module is a gripper or hook mechanism arranged symmetrically on both sides, and the opening and closing action is achieved by an electric drive device or a pneumatic drive device.

[0027] As a preferred option, the lifting lugs of the special ton bag are blue, the main body of the special ton bag is white, and the outline of the special ton bag is sewn with yellow edging.

[0028] As a preferred solution, the system performs the following steps during the lifting of ton bags:

[0029] S1: Collect multi-source perception data of the work area using a binocular vision camera and LiDAR;

[0030] S2: The multi-source sensing data is fused and processed to reconstruct the three-dimensional environment, and an improved weighted multi-feature fusion recognition algorithm is used to identify and locate the three-dimensional coordinates and attitude of the ton bag and its lifting lugs.

[0031] S3: Plan the safe movement path of the spreader actuator to the target lifting lug, and control the spreader actuator to move to the predetermined adsorption position;

[0032] S4: Control the electromagnetic adsorption module to be powered on, and use magnetic force to lift the lug with the embedded metal parts.

[0033] S5: After confirming that the lifting lug has been successfully lifted using a verification and recognition method, control the mechanical gripper module to complete the gripping of the lifting lug;

[0034] S6: Control the electromagnetic adsorption module to shut off power and control the lifting device actuator to move the ton bag to the target location.

[0035] As can be seen from the technical solution provided by the present invention above, the unmanned control system for ton bag hoisting based on binocular vision camera and lidar provided by the present invention has the following beneficial effects:

[0036] 1. By deeply integrating binocular vision and LiDAR, and combining them with a dynamic weighted multi-feature recognition algorithm, a dual-protection recognition system is constructed. The binocular vision relies on the high-contrast design of the special ton bag's "blue lifting lugs - white bag body" to quickly lock the visual area of ​​the lifting lugs. The LiDAR, on the other hand, relies on the high reflectivity of the metal parts embedded in the lifting lugs to accurately verify the spatial position of the lifting lugs. The two complement each other to effectively eliminate environmental interference such as dust and low light. At the same time, the algorithm can dynamically adjust the weights of features such as color, geometry, and reflectivity according to environmental changes, ensuring that the lifting lugs can still be stably identified and accurately located in scenarios with low visibility and unstable lighting. This avoids misjudgment or missed judgment due to the failure of a single feature, and greatly improves the robustness of the system in complex industrial scenarios.

[0037] 2. A closed-loop logic of "electromagnetic adsorption-verification-mechanical gripping-release" is constructed to ensure safety across the entire chain. The electromagnetic adsorption module first lifts the collapsed lifting lug to a natural hanging state, and then the sensing unit verifies the lifting effect to ensure that the lifting lug is stable before the mechanical gripper performs the gripping action, thus avoiding gripping failure caused by abnormal lug status from the source. At the same time, electromagnetic adsorption and mechanical gripping form a "cooperative relay". Even if the electromagnetic adsorption is unexpectedly powered off, the rigid clamping of the mechanical gripper can still ensure that the ton bag does not fall off. The real-time scanning and dynamic obstacle avoidance function of the lidar can also avoid sudden obstacles during the operation in time, preventing equipment collisions or ton bag falling accidents, and significantly improving the reliability and safety of the lifting operation.

[0038] 3. This invention eliminates the need for manual intervention throughout the entire process, from data acquisition, lug positioning, path planning to grabbing and handling, significantly reducing the time required for a single operation and greatly increasing the lifting capacity per unit time. At the same time, unmanned operation reduces labor costs, requiring only a small number of maintenance personnel to manage multiple devices. Especially in high-risk scenarios, personnel do not need to work on-site; scheduling can be completed through remote monitoring, which reduces labor costs and avoids personnel safety risks, providing strong support for enterprises to improve production efficiency and control operating costs.

[0039] 4. Through modular design and adjustable parameters, the system can quickly adapt to ton bags of different weights and materials without replacing hardware, resulting in short adaptation time and convenient operation. Simultaneously, the system possesses self-checking and fault-tolerant capabilities, automatically correcting sensor calibration deviations, providing real-time fault information and handling suggestions, reducing the workload of daily debugging and troubleshooting. The modular design of key components also shortens maintenance and replacement time, reduces maintenance costs, and is compatible with existing factory equipment such as cranes and conveyors, eliminating the need for overall production line modifications. This makes it particularly suitable for upgrading older factories, further expanding the system's application scope. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall structure of the unmanned control system for ton bag hoisting based on binocular vision camera and lidar of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, this embodiment of the invention provides an unmanned control system for ton bag lifting based on binocular vision camera and lidar, including an intelligent control unit, a lifting device actuator, a sensing unit, and a special ton bag;

[0044] The sensing unit includes a binocular vision camera and a lidar. The sensing unit is installed above the lifting device actuator and is used to collect depth visual information and three-dimensional point cloud information of the work area.

[0045] The intelligent control unit communicates with the sensing unit and the lifting device actuator to process and fuse multi-source sensing data and generate control commands.

[0046] The lifting device actuator includes an independently controllable electromagnetic adsorption module and a mechanical gripper module;

[0047] The lifting lugs of the special ton bag have embedded metal parts that can be magnetically attracted, and the color of the lifting lugs of the special ton bag has a high contrast with the main color of the bag body.

[0048] In this embodiment, the mechanical gripper module is a gripper or hook mechanism arranged symmetrically on both sides, and the opening and closing action is realized by an electric drive device or a pneumatic drive device.

[0049] Furthermore, the lifting device actuator is the "core" of the unmanned control system for ton bag lifting based on binocular vision cameras and LiDAR. Through the coordinated action of electromagnetic adsorption and mechanical gripping, it precisely completes the adsorption, lifting, gripping, and handling operations of the ton bag lifting lugs, making it a key actuator for achieving unmanned operation throughout the entire ton bag lifting process. The following section provides a detailed explanation of this mechanism, from its overall structure to its specific details:

[0050] I. Overall Function Overview:

[0051] The lifting device actuator is mainly responsible for receiving phased collaborative control commands from the intelligent control unit. Through the coordinated action of the independently controllable electromagnetic adsorption module and the mechanical gripper module, it completes operations such as positioning and alignment of the ton bag lifting lugs, adsorption and lifting, gripping and fixing, and ton bag transportation. It needs to adapt to the structural characteristics of the special ton bag lifting lugs (embedded metal parts, high contrast color) and achieve the precise function of the lifting lugs with the cooperation of the sensing unit. At the same time, it works with the path planning module to complete obstacle avoidance during movement, ensuring the stability and safety of the lifting process.

[0052] II. Submodule Composition and Functions:

[0053] (a) Electromagnetic adsorption module:

[0054] Structure and core components: Composed of an electromagnetic coil, an iron core, and a positioning calibration component, it is installed in the upper area of ​​the lifting device actuator. Its center point can be aligned with the target point of the lifting lug in three axes through the control of the intelligent control unit. The design of the coil and the iron core must meet the adsorption requirements of the metal parts embedded in the lifting lug to ensure that sufficient magnetic force can be generated when energized.

[0055] Adsorption and release function: By receiving the adsorption control command from the control decision module, a magnetic force is generated. The magnetic force is used to attract the magnetically adsorbable metal parts pre-embedded in the lifting lug, and the lifting lug that has collapsed due to its own weight or stacking is lifted to a natural hanging state. After the mechanical gripper module completes the gripping, it receives the release command and cuts off the power, and the magnetic force disappears, realizing the separation from the lifting lug. Its adsorption force must be adapted to the load-bearing requirements of the ton bag lifting lug to ensure that the lifting lug does not fall off during the lifting process.

[0056] Independent control features: It adopts an independent control loop from the mechanical gripper module, and can receive power-on, power-off and position adjustment commands from the intelligent control unit independently. It does not depend on the action status of the mechanical gripper module, ensuring that the posture correction of the lifting lug can be completed independently during the adsorption and lifting stage.

[0057] (ii) Mechanical gripper module:

[0058] Structure and layout: The gripper or hook mechanism is arranged symmetrically on both sides, and its opening and closing range can be adapted to the size of the lifting lugs of different ton bags; the end design of the gripper / hook must avoid damage to the lifting lugs, while ensuring the clamping stability after gripping.

[0059] Drive and motion functions: Powered by an electric or pneumatic drive unit, it receives gripper control commands from the control decision module and executes the action sequence of "opening-penetrating the lifting lug-closing and clamping": When opening, it ensures that the lifting lug can smoothly enter the gripping range; when penetrating the lifting lug, it coordinates with the overall position adjustment of the lifting device to achieve precise alignment; when closing, it generates a stable clamping force (the controllable range is adapted to the strength of the lifting lug) to fix the lifting lug to meet the handling requirements.

[0060] Synergistic characteristics: The action triggering requires the electromagnetic adsorption module to complete the adsorption and lifting and the sensing unit to verify the success. It forms a synergy with the electromagnetic adsorption module in terms of action sequence - the action is performed after the electromagnetic adsorption module lifts the lifting lug to a stable state, and the clamping state is maintained when the electromagnetic adsorption module is de-energized and released, so as to ensure that the ton bag does not fall off during the handling process.

[0061] III. Key Technology Principles:

[0062] (I) Electromagnetic adsorption lifting principle:

[0063] Based on the principle of electromagnetic induction, after the electromagnetic adsorption module is powered on, the coil generates an alternating magnetic field, and the iron core is magnetized to form a strong magnet, which generates a magnetic attraction force on the metal parts embedded in the lifting lug. The magnitude of this attraction force is positively correlated with the coil current intensity. By controlling the current through the intelligent control unit, the adsorption force can be dynamically adjusted, which can ensure that the collapsed lifting lug is lifted to a natural hanging state (verified by the height change rate monitored by the sensing unit), and also avoid damage to the lifting lug due to excessive adsorption force.

[0064] (II) Mechanical gripping force control principle:

[0065] The mechanical gripper module's drive unit (electric or pneumatic) transmits power to the grippers / hooks through a transmission mechanism (such as gears or linkages). The stroke and clamping force of its opening and closing action are fed back by displacement or pressure sensors. The intelligent control unit presets the clamping force threshold based on the material and size of the lifting lugs. The drive unit monitors the feedback value in real time during the closing process and stops the action when the threshold is reached, thus achieving adaptive gripping for different types of lifting lugs and avoiding loose gripping that leads to detachment or excessive gripping that leads to damage.

[0066] (III) Principle of Dual-Module Cooperative Action:

[0067] The electromagnetic adsorption module and the mechanical gripper module coordinate their actions through timing planning by the intelligent control unit: the electromagnetic adsorption module first completes the positioning, alignment, and lifting correction of the lugs, creating conditions for the mechanical gripper module to grasp; after successful adsorption verification, the mechanical gripper module moves to achieve rigid fixation of the lugs through mechanical force; in the final stage, the electromagnetic adsorption module releases, while the mechanical gripper module remains clamped, forming a dual guarantee mechanism of "adsorption-assisted correction - mechanical rigid fixation" to compensate for the stability defects of a single adsorption or gripping method;

[0068] IV. Organizational Workflow:

[0069] (a) Initialization phase:

[0070] After the lifting device actuator is started, the electromagnetic adsorption module is in a de-energized state (no magnetic force), and the mechanical gripper module is in an initial open state; it completes the communication connection with the intelligent control unit, receives the initial position calibration command, adjusts the overall posture to the ready-to-work state, and waits for the lifting lug positioning information from the sensing unit;

[0071] (II) Positioning and Alignment Stage:

[0072] Upon receiving the first-stage movement control command from the intelligent control unit, the spatial position of the lifting device actuator is adjusted according to the three-dimensional coordinates and attitude information of the lifting lug, so that the center point of the electromagnetic adsorption module is aligned with the identified target point of the lifting lug in the three-axis space, and the alignment accuracy is controlled within a preset range (such as ±2mm), laying the positional foundation for subsequent adsorption actions.

[0073] (III) Adsorption and Lifting Stage:

[0074] After the position is aligned, the second-stage adsorption control command is received. The electromagnetic adsorption module is powered on to generate magnetic force, which adsorbs the metal parts embedded in the lifting lug.

[0075] Under the influence of magnetic force, the collapsed lugs are gradually lifted to a natural hanging state. At this point, the sensing unit initiates verification and recognition by calculating the rate of change of the vertical dimension of the point cloud in the lug region. To determine the adsorption effect;

[0076] when Once the value exceeds the preset threshold (indicating successful adsorption and lifting), the electromagnetic adsorption module remains powered on to maintain the suspended posture of the lug and await the action command from the mechanical gripper module.

[0077] (iv) Mechanical gripping stage:

[0078] Upon receiving the third-stage gripper control command, the mechanical gripper module operates according to the following steps:

[0079] Maintain the initial open position and fine-tune the position with the lifting gear so that the grippers / hooks pass down from both sides of the lifting lugs to the preset gripping position;

[0080] The drive unit operates to control the gripper / hook to close until the preset clamping force threshold is reached, and then stops, thus completing the mechanical gripping and fixing of the lifting lug.

[0081] (v) Release and transportation phase:

[0082] Upon receiving the fourth-stage release and handling control command, the electromagnetic adsorption module is powered off, the magnetic force disappears, and the adsorption on the lifting lug is released.

[0083] The mechanical gripper module remains clamped, and the lifting device actuator, under the control of the intelligent control unit, moves the ton bag to the target location according to the planned safe movement path;

[0084] Upon reaching the target location, the mechanical gripper module receives the release command, executes the opening action, releases the lifting lugs, and completes one lifting cycle.

[0085] In this embodiment, the lifting lugs of the special ton bag are blue, the main body of the special ton bag is white, and the outline of the special ton bag is sewn with yellow edging.

[0086] Furthermore, the specialized ton bag serves as an "adapter" for the unmanned ton bag lifting control system based on binocular vision cameras and LiDAR. Its structural design must simultaneously meet the requirements of accurate identification by the sensing unit and stable operation of the lifting actuator, making it a key component connecting the system's sensing and execution links. The following section provides a detailed explanation of this specialized ton bag, from overall structure to specific details:

[0087] I. Overall Function Overview:

[0088] Specialized ton bags are mainly used for loading bulk materials in industries such as chemicals and building materials. Their core function is to achieve fully automatic identification, adsorption, and gripping of the ton bag's lifting lugs through customized structural design and system integration. On the one hand, the high-contrast color design between the lifting lugs and the bag body provides clear recognition features for the binocular vision camera. On the other hand, the magnetically adsorbable metal parts embedded in the lifting lugs are compatible with the electromagnetic adsorption module of the lifting device's actuator. At the same time, the yellow edging design of the bag's outline helps the lidar to locate the overall range of the ton bag, avoiding collisions with surrounding objects during lifting, and ultimately ensuring the smoothness and safety of the unmanned lifting process.

[0089] II. Sub-component composition and function:

[0090] (a) Main body of the bag:

[0091] Material and load-bearing design: The main body of the bag is made of high-strength polypropylene woven fabric or polyester canvas, processed by double-layer weaving process, with a warp and weft density of not less than 16×16 threads / inch. The rated load-bearing capacity of a single bag is 1-2 tons, which can withstand the tensile force and material impact during the hoisting process. It also has waterproof and dustproof properties (the surface can be coated with polyvinyl chloride coating) and is suitable for the harsh environment of industrial scenarios.

[0092] Color and recognition adaptation: The main color of the bag is pure white (RGB color value: 255,255,255). This color selection needs to meet the high contrast requirement with the color of the ear loops. In the CIELAB color space, the color difference value ΔE between the white bag body and the blue ear loops is greater than 80. Even in complex lighting environments such as strong light and dust, the binocular vision camera can still quickly separate the bag body and ear loop areas to avoid confusion in ear loop recognition.

[0093] Structure and Loading Adaptation: The main body of the bag adopts a square or cylindrical structure (common size: 1000mm×1000mm×1200mm, which can be adjusted according to the material density). The top is equipped with 4 symmetrically distributed lifting lug mounting positions (spacing 500-600mm, adapted to the spacing of the double-sided grippers of the lifting device actuator). The bottom is equipped with a sealable discharge port (using zipper or buckle sealing for convenient subsequent unloading operations). The edges of the discharge port are reinforced to prevent tearing when loading heavy objects.

[0094] (ii) Lifting lug assembly:

[0095] Material and strength design: The lifting lugs are made of high-strength polyester webbing (breaking strength ≥50kN), with a width of 50-80mm and a thickness of 5-8mm. They are fixedly connected to the reinforcing layer (multi-layer woven fabric) at the top of the bag body through double-line sewing. The stitch distance is ≤5mm to ensure the connection strength between the lifting lugs and the bag body during lifting and to prevent the lifting lugs from falling off.

[0096] Color and recognition function: The ear loops are pure blue (RGB color value: 0,0,255), which creates a sharp visual contrast with the white bag body. This contrast design allows the binocular vision camera to quickly locate the ear loops within a 10-meter range, with a recognition accuracy of ≥98%. Even if a small amount of dust adheres to the bag surface, the blue ear loops can still form a clear boundary with the white bag body, reducing the difficulty of recognition.

[0097] Pre-embedded metal component design: A magnetically adsorbable metal component (made of iron-nickel alloy, with a magnetic permeability μ>1000μH / m) is pre-embedded inside the lifting lug along its length. The metal component is in the form of a thin sheet (thickness 1-2mm, width 30-50mm, length consistent with the lifting lug length), embedded in the middle layer of the lifting lug webbing. The distance between the two ends of the metal component and the end of the lifting lug is ≥10mm to prevent the exposed metal component from scratching the equipment or operators. The position of the metal component must coincide with the central axis of the lifting lug to ensure that the lifting lug is subjected to uniform force when the electromagnetic adsorption module adsorbs, and to prevent the lifting lug from tilting during the lifting process.

[0098] (iii) Outline binding:

[0099] Material and Durability: The bag body outline (top edge, bottom edge and side center line) is sewn with yellow binding tape (material is wear-resistant nylon tape, width 20-30mm, thickness 2-3mm). The binding tape is fixed by a combination of high-frequency heat sealing and sewing. It can withstand ≥5000 abrasion cycles and can withstand slight collisions with lifting equipment and shelves during hoisting, thus extending the service life of the ton bag.

[0100] Color and positioning function: The edge banding is pure yellow (RGB color value: 255,255,0). This color has high visibility in industrial environments. When scanning with LiDAR, the reflection intensity of the yellow edge banding (about 80-100) is higher than that of the white bag body (about 40-60). This can help LiDAR quickly outline the overall shape of the ton bag. Especially in scenarios where multiple ton bags are densely stacked, it can effectively distinguish the boundaries of adjacent ton bags, provide accurate obstacle information for the path planning module, and avoid collisions between the lifting device and adjacent ton bags.

[0101] III. Key Technology Design Principles:

[0102] (a) High-contrast color recognition principle:

[0103] Based on the color segmentation theory of visual recognition, the special ton bag, through the color combination of "blue lifting lugs + white bag body", forms a significant grayscale difference in the RGB-D image captured by the binocular vision camera (the grayscale value of the blue lifting lugs is about 40-60, and the grayscale value of the white bag body is about 220-240). The grayscale difference is >160, and the lifting lug area can be quickly extracted by a simple threshold segmentation algorithm (such as the Otsu adaptive thresholding method). At the same time, blue has a stable hue range in the HSV color space (H: 200-240°, S: 80%-100%, V: 50%-70%). Even under changes in light intensity (such as cloudy days or strong light) or dust interference, background noise can still be eliminated through color space filtering to ensure the robustness of lifting lug recognition.

[0104] (II) Principle of magnetic adsorption adaptation of embedded metal parts:

[0105] Based on the magnetic coupling theory of electromagnetic adsorption, the iron-nickel alloy metal parts embedded in the lifting lug have high magnetic permeability. When the electromagnetic adsorption module of the lifting device's actuator is energized, the metal parts are rapidly magnetized, forming a magnetic coupling with the electromagnetic adsorption module, generating an adsorption force. Satisfying the formula: (in, The electromagnetic adsorption module exerts an adsorption force on the pre-embedded metal parts inside the lifting lug, measured in Newtons (N). The magnetic induction intensity generated after the electromagnetic adsorption module is energized is expressed in Tesla (T). The effective contact area between the embedded metal parts in the lifting lug and the electromagnetic adsorption module is measured in square meters (m²). ρ is the free permeability, a physical constant with a value of 4π × 10⁻ 7 Henry per meter (H / m); By designing the contact area of ​​the metal parts (matching the adsorption surface size of the electromagnetic adsorption module, such as a circular metal part with a diameter of 50mm), the adsorption force can be ensured to be ≥200N, which is sufficient to lift the lifting lugs that have collapsed due to their own weight or pressure to a natural hanging state (vertical height change rate). >15%), creating conditions for subsequent mechanical grasping;

[0106] (III) The three-dimensional positioning principle of contour edging:

[0107] LiDAR distinguishes different objects by measuring the reflectance intensity of their surfaces. Due to its material properties (reflectivity of approximately 60%), the yellow nylon edging tape of the specialized ton bag has a higher laser reflection intensity than the white bag body (reflectivity of approximately 30%) and the surrounding environment (such as the ground and shelves, with a reflectivity of approximately 20%), forming a "high-reflectance contour line" in the 3D point cloud data. The path planning module can extract this contour line to calculate the spatial volume and position coordinates of the ton bag and determine whether there is a collision risk in the movement path of the lifting device. At the same time, the continuous distribution of the yellow edging tape (encircling the bag body) can avoid contour loss caused by unevenness of the material on the bag surface, ensuring the integrity of the ton bag contour in the 3D environment reconstruction model.

[0108] IV. Coordination and Adaptation Process with Other System Modules:

[0109] (I) Perception Stage: Assisted Ear Localization and Contour Recognition

[0110] When the binocular vision camera acquires images of ton bags, it quickly segments the candidate area of ​​the hanging ears by comparing the colors of the "blue hanging ears - white bag body". Then, it filters out the real hanging ears by combining the geometric shape of the hanging ears (such as long strips) and outputs the two-dimensional coordinates of the hanging ears.

[0111] During LiDAR scanning, on the one hand, the high reflectivity (approximately 120-150) of the metal parts inside the lifting lugs verifies the position of the lifting lugs recognized by binocular vision, and outputs the three-dimensional coordinates and posture of the lifting lugs; on the other hand, the high reflectivity outline of the yellow edging tape is used to construct the overall three-dimensional model of the ton bag, mark the spatial occupancy range of the ton bag, and provide obstacle information for path planning.

[0112] (II) Adsorption Stage: Lifting up the adapted electromagnetic adsorption module:

[0113] The electromagnetic adsorption module of the lifting device actuator moves to directly above the lifting lug based on the three-dimensional coordinates of the lifting lug output by the sensing unit, so that the center point of the electromagnetic adsorption module is aligned with the center of the lifting lug (the center of the metal part);

[0114] After the electromagnetic adsorption module is powered on, it generates an adsorption force through magnetic coupling with the metal parts inside the lifting lug, which lifts the collapsed lifting lug to a natural hanging state. At this time, the centered design of the metal parts can ensure that the lifting lug is balanced under force and prevent the lifting lug from shifting during the lifting process.

[0115] (III) Grasping Phase: Clamping with the Adaptive Mechanical Gripper Module:

[0116] The sensing unit calculates the rate of change of vertical height of the point cloud in the lug region. After confirming that the lifting lug has been raised to a gripping position, the mechanical gripper module adjusts the opening and closing degree according to the size of the lifting lug (width 50-80mm);

[0117] The mechanical gripper module passes under the two sides of the lifting lug and closes to clamp the lifting lug. The high-strength polyester webbing material of the lifting lug can withstand the clamping force of the mechanical gripper (50-200N), avoiding damage to the lifting lug during the clamping process.

[0118] (iv) Handling stage: Ensuring material safety and contour recognition:

[0119] The high-strength material of the bag body can withstand the weight of 1-2 tons of materials, preventing the bag from tearing and causing material leakage during handling;

[0120] The highly reflective outline of the yellow edging tape is continuously scanned by LiDAR during the handling process. The path planning module dynamically corrects the movement path of the lifting device based on the real-time scanning results to ensure that the ton bag does not collide with surrounding equipment or other ton bags during handling.

[0121] In this embodiment, after the second stage of adsorption control, the sensing unit restarts to perform verification and identification. Only after confirming that the lifting lug has been successfully adsorbed and lifted and is in a graspable state will the control decision module trigger the third stage of gripper control. The verification and identification is achieved by calculating the average height change rate of the point cloud in the lifting lug area in the vertical direction before and after adsorption. When the height change rate is greater than the preset height change threshold, the adsorption and lifting are determined to be successful.

[0122] The lidar data is also used for real-time scanning during the movement of the spreader actuator, and the path planning module dynamically corrects the movement path based on the real-time point cloud data to achieve obstacle avoidance;

[0123] Furthermore, the perception unit is the "perception center" of the unmanned control system for ton bag lifting based on binocular vision cameras and lidar. It provides the system with accurate environmental information of the working area through the collaborative work of multimodal sensors, and is the foundation for realizing fully automatic identification and positioning of ton bag lifting lugs. The following section elaborates on this unit from the overall structure to the details.

[0124] I. Overall Function Overview:

[0125] The perception unit is primarily responsible for real-time acquisition of visual and 3D spatial information of the work area. Its core functions include: acquiring color, texture, and depth information of the ton bags and lifting lugs through a binocular vision camera; acquiring 3D point cloud data of the work environment (including position, distance, and reflection intensity information) through a lidar; performing spatiotemporal synchronization and calibration registration of the two types of raw data to provide fused multi-source perception data for the intelligent control unit, supporting core tasks such as 3D environment reconstruction, lifting lug identification and positioning, and obstacle detection; and continuously and dynamically scanning during the lifting process to provide real-time updated environmental feedback for path planning and control decisions, ensuring the system's adaptability to complex scenarios.

[0126] II. Submodule Composition and Functions:

[0127] (a) Binocular vision camera module:

[0128] Hardware configuration and parameters: Employs an industrial-grade binocular vision camera (baseline distance 120-200mm), with both left and right cameras having a resolution of 2560×1440 pixels, a frame rate of 30fps, a lens focal length of 8-16mm (field of view 60°-90°), and supports global shutter (exposure time 10-1000μs) to avoid motion blur; the camera integrates an infrared fill light (wavelength 850nm), which automatically turns on in low-light environments to ensure stable image brightness; the protection rating reaches IP65, adapting to dusty, humid, and other industrial environments;

[0129] Data acquisition function: Simultaneously output RGB color images and disparity maps from the left and right cameras, and generate RGB-D images (depth range 0.5-15m, depth accuracy ≤2%@5m) through a stereo matching algorithm; Among them, the RGB images are used to extract the color features of the lugs and the bag body (such as the high contrast area between the blue lugs and the white bag body), and the depth information is used to calculate the preliminary three-dimensional coordinates of the lugs, providing a basis for subsequent fusion with LiDAR data;

[0130] Timing synchronization mechanism: The clock is kept synchronized with the LiDAR through a hardware trigger interface (such as GPIO) to ensure that the timestamp error between the visual image and the point cloud data is ≤1ms, avoiding fusion deviation caused by data asynchrony. Especially during the movement of the lifting device, it can ensure the consistency of spatial position.

[0131] (ii) LiDAR module:

[0132] Hardware configuration and parameters: Employs a multi-line lidar (16 or 32 lines), with a horizontal field of view of 360°, a vertical field of view of -15° to +15°, an angular resolution of 0.1° × 0.2° (horizontal × vertical), a scanning frequency of 10-20Hz, a ranging range of 0.5-100m, and a ranging accuracy of ±2cm@10m; the radar has anti-interference capabilities (such as signal differentiation when multiple radars are working simultaneously), an IP67 protection rating, and can operate stably in rain, snow, and dust environments;

[0133] Data acquisition function: Outputs 3D point cloud data, each point containing (x, y, z) spatial coordinates and reflection intensity value (0-255); among which, the spatial coordinates are used to construct the 3D structure of the working environment, and the reflection intensity value is used to distinguish objects of different materials (such as the high reflection intensity of the metal parts embedded in the lifting lugs and the low reflection intensity of the bag body); through continuous scanning, the location information of surrounding obstacles can be updated in real time, providing dynamic obstacle avoidance basis for path planning;

[0134] Point cloud preprocessing function: The built-in FPGA chip realizes real-time point cloud denoising (removing outliers and duplicates) and downsampling (preserving key feature points), compressing the original point cloud data volume by 30%-50%, reducing the bandwidth pressure of subsequent data transmission and processing, while ensuring the integrity of point clouds of key targets such as the lugs;

[0135] (III) Sensor Integration Bracket and Calibration Module:

[0136] Mechanical integration design: An L-shaped bracket made of aluminum alloy is used to rigidly fix the binocular vision camera and the LiDAR, which is installed above the lifting device's actuator (500-800mm from the end of the lifting device). The angle between the camera's optical axis and the central axis of the radar scanning plane is ≤5°, ensuring that the overlap area of ​​their fields of view is ≥70% (focusing on covering the working area 1-5m below the lifting device). The bracket has a shock-absorbing and buffering structure (such as rubber pads) to reduce the impact of vibration during the movement of the lifting device on the sensor.

[0137] Calibration and registration functions: Equipped with both offline calibration before shipment and online calibration on-site; during offline calibration, the camera intrinsic parameters (focal length, distortion coefficient), binocular baseline and extrinsic parameters (rotation matrix, translation vector), as well as the extrinsic parameter matrix between the camera and radar (converting the radar point cloud to the camera coordinate system) are calculated using a checkerboard calibration board and a ball target; during online calibration, the system can automatically correct calibration parameter drift caused by long-term use by recognizing known features of the special ton bag (such as the geometric dimensions of the yellow edging), ensuring coordinate transformation accuracy ≤3mm;

[0138] III. Key Technology Principles:

[0139] (I) Principle of Spatiotemporal Synchronization of Multi-Source Data:

[0140] Based on hardware triggering and timestamp alignment technology, the sensing unit generates a unified trigger signal through a synchronization controller, enabling the binocular camera and LiDAR to start data acquisition at the same time, ensuring data time synchronization; spatially, through the external parameter matrix... (in, These are the point coordinates in the lidar coordinate system. These are the coordinates of the corresponding point in the camera coordinate system. It is a 3×3 rotation matrix. The point cloud data is transformed into the camera coordinate system (using a 3×1 translation vector) to achieve spatial registration between the visual data and the point cloud data, laying the foundation for subsequent fusion processing.

[0141] (II) The principle of 3D reconstruction by fusing RGB-D and point clouds:

[0142] A voxel-based fusion algorithm is employed to fuse RGB-D images from binocular vision with LiDAR point cloud data. First, color information from the RGB-D images is used to assign color attributes to the point cloud (addressing the lack of color information in LiDAR). Second, high-precision distance information from the point cloud is used to correct noise in the visual depth map (such as depth jumps in edge regions). Finally, a dense 3D environment model with a resolution of 5mm×5mm×5mm is constructed. The point cloud in the hanging ear region forms a significant cluster due to its simultaneous presence of blue features (visual) and high reflectivity (radar), providing clear target features for the identification and positioning module.

[0143] (III) Principles of Ear Feature Enhancement and Extraction:

[0144] For the multimodal features (color, geometry, reflection intensity) of the lugs, the perceptual unit enhances and extracts features through the following steps:

[0145] Color feature enhancement: In the RGB image, the candidate region of the hanging ear is initially located by threshold segmentation of the blue channel (B>200, R<50, G<50), and then noise is removed by combining morphological filtering (erosion-dilation) to output the two-dimensional mask of the hanging ear;

[0146] Enhanced reflection intensity features: In point cloud data, high reflection points (corresponding to metal parts inside the lugs) are filtered out by reflection intensity threshold (I>100), and spatial intersection operation is performed with visual masks to accurately lock the three-dimensional spatial range of the lugs;

[0147] Geometric feature extraction: Plane fitting and edge detection are performed on the point cloud of the lug region, and its normal vector and curvature features are calculated to provide the original geometric parameters for the weighted multi-feature fusion recognition algorithm of the intelligent control unit;

[0148] IV. Collaboration process with other modules of the system:

[0149] (a) Collaboration with the intelligent control unit:

[0150] Data transmission: The sensing unit transmits the synchronized RGB-D image (compressed format, such as H.265) and the preprocessed point cloud (PCAP format) to the intelligent control unit via gigabit Ethernet. The transmission delay is ≤100ms, which meets the real-time requirements.

[0151] Parameter interaction: Receives dynamic configuration instructions from the intelligent control unit (such as adjusting camera exposure time and radar scanning frequency), and adaptively optimizes the acquisition parameters according to changes in the working environment (such as light intensity and ton bag stacking density). For example, it reduces the camera exposure time to 50μs under strong light to avoid overexposure.

[0152] (ii) Coordination with the spreader actuator:

[0153] Position association: The coordinate system of the sensing unit is associated with the motion coordinate system of the lifting device actuator through calibration, enabling the intelligent control unit to directly convert the three-dimensional coordinates of the lifting lug into the motion target coordinates of the lifting device;

[0154] Dynamic tracking: During the movement of the spreader, the sensing unit continuously outputs the real-time position deviation of the lifting lug (relative to the center point of the spreader) at a frequency of 20Hz, providing closed-loop control feedback to the control decision module to ensure that the spreader is accurately aligned with the lifting lug (positioning error ≤ ±2mm).

[0155] (iii) Compatibility with the features of specialized ton bags:

[0156] Color feature adaptation: For the "blue lifting lugs + white bag body" design of the special ton bag, the white balance parameters of the binocular vision camera have been preset and optimized (blue channel gain increased by 15%) to ensure that color contrast can still be maintained under different lighting conditions.

[0157] Reflection feature adaptation: The reflection intensity threshold of the lidar has been preset according to the characteristics of the metal parts inside the lug (reflection intensity 120-150) and the differences between the bag body (40-60) and the environment (20-40), which can quickly distinguish the lug from the background.

[0158] In this embodiment, the intelligent control unit includes:

[0159] The data fusion module is configured to receive and fuse RGB-D images from a binocular vision camera and 3D point cloud data from a LiDAR, and generate a dense 3D environment reconstruction model with color information through calibration and registration.

[0160] The identification and positioning module is configured to accurately locate the three-dimensional coordinates and posture of the lug based on a dense three-dimensional environment reconstruction model and an improved weighted multi-feature fusion identification algorithm. The objective function of the improved weighted multi-feature fusion identification algorithm is expressed as: finding the point that minimizes the weighted sum of the matching degree between the Euclidean distance of the color feature and the geometric feature, minus the weighted value of the reflection intensity value. The weighting coefficients of the color feature, geometric feature, and reflection intensity feature are dynamically adjusted according to the sensor accuracy and environmental conditions.

[0161] The path planning module is configured to plan a safe movement path for the lifting device actuator based on the three-dimensional coordinates and attitude of the lifting lug and obstacle information in the three-dimensional environment reconstruction model.

[0162] The control decision module is configured to send phased collaborative control commands to the spreader actuator based on a safe movement path.

[0163] The phased collaborative control instructions executed by the control decision module include, in sequence:

[0164] First-stage movement control: Control the movement of the lifting device actuator so that the center point of the electromagnetic adsorption module is aligned with the identified lifting lug target point in three-axis space;

[0165] Second stage adsorption control: After maintaining the position alignment, control the electromagnetic adsorption module to generate magnetic force, adsorb and lift the hanging lugs that have collapsed due to their own weight or stacking pressure to a natural hanging state.

[0166] The third stage of gripper control: After the lifting lug is attracted and lifted, the mechanical gripper module is controlled to perform the actions of opening, passing through the lifting lug, and closing and clamping.

[0167] The fourth stage of release and handling control: the electromagnetic adsorption module is de-energized to release the magnetic force, and at the same time the lifting device actuator is controlled to transport the grabbed ton bag to the target location;

[0168] The geometric feature matching degree of the improved weighted multi-feature fusion recognition algorithm in the recognition and localization module is obtained by calculating the weighted sum of the differences in normal vector features and curvature features, where the weighting factors of normal vector features and curvature features are used to adjust the contribution of the corresponding features.

[0169] The reflection intensity feature weighting coefficient is dynamically related to the reflection intensity value. Specifically, it is expressed as the product of the basic reflection intensity weighting value plus the gain coefficient and the portion of the average reflection intensity value that exceeds the reflection intensity threshold.

[0170] Furthermore, the intelligent control unit is the "core decision-making center" of the unmanned control system for ton bag lifting based on binocular vision cameras and LiDAR. Its core function is to receive and integrate multi-source perception data, and through precise lug identification and positioning, safe path planning, and phased collaborative control, output reliable control commands to the lifting device actuator. At the same time, it realizes closed-loop feedback and fault handling, which is the key to ensuring the unmanned and highly reliable operation of the entire system. The following is a detailed description of this unit from the overall structure to the details:

[0171] I. Overall Function Overview:

[0172] The intelligent control unit undertakes the core tasks of the entire process of "data processing - decision planning - command output - closed-loop control": First, it receives binocular visual RGB-D images and LiDAR 3D point cloud data transmitted by the sensing unit, and generates a dense 3D environment model with color information through spatiotemporal synchronization and feature-level fusion; Second, based on this model, it accurately locates the 3D coordinates and attitude of the lifting lug through an improved weighted multi-feature fusion algorithm, while extracting obstacle information in the environment; Then, combined with the motion constraints of the lifting device actuator, it plans a safe and efficient movement path; Finally, it sends phased collaborative control commands (adsorption-verification-grabbing-transportation) to the lifting device actuator, and achieves closed-loop control through secondary recognition by the sensing unit and status feedback of the lifting device to ensure the reliability of each action; In addition, the unit also has dynamic parameter adjustment, fault self-processing and data recording functions, adapting to the lifting needs of complex industrial environments and different batches of ton bags;

[0173] II. Submodule Composition and Functions:

[0174] (a) Data Fusion Module:

[0175] Data reception and preprocessing: Two types of core data output from the sensing unit are received synchronously via a gigabit Ethernet interface: RGB-D images from the binocular vision camera (resolution 2560×1440, frame rate 30fps) and 3D point cloud data from the LiDAR (16 / 32 lines, scanning frequency 10-20Hz). Preprocessing is performed first: Gaussian filtering (kernel size 3×3) is applied to the RGB-D images to remove noise, and histogram equalization is used to enhance color contrast (focusing on optimizing the distinction between the blue lugs and the white bag). Outlier removal (based on statistical filtering, confidence interval 95%), duplicate point removal, and downsampling (using voxel mesh downsampling, voxel size 5mm×5mm×5mm) are performed on the LiDAR point cloud. While retaining key features such as lugs and obstacles, the point cloud data volume is compressed by 40%-50%, reducing the bandwidth pressure for subsequent processing.

[0176] Spatiotemporal synchronization and coordinate registration:

[0177] Based on the hardware-triggered timestamp of the sensing unit (error ≤ 1ms), time alignment between the RGB-D image and point cloud data is achieved; then, a preset extrinsic parameter matrix (obtained from factory calibration, including the rotation matrix) is used. With translation vector Point cloud data in the lidar coordinate system Transform to the binocular vision camera coordinate system using the following formula:

[0178] ,in, These are the coordinates of the point in the transformed camera coordinate system; It is a 3×3 rotation matrix that describes the attitude deviation between the radar and the camera; (This is a 3×1 translation vector describing the positional deviation between the radar and the camera); After registration, the spatial positional error between the two types of data is ensured to be ≤3mm, laying the foundation for feature fusion.

[0179] Feature-level fusion and 3D modeling:

[0180] A fusion strategy of "color completion + depth correction" is adopted: the color information (RGB channel values) of the RGB-D image is mapped to the registered point cloud to solve the "colorless" defect of LiDAR, so that the lug (blue), bag (white), and obstacles (such as gray equipment) can be intuitively distinguished by color in the point cloud; at the same time, the high-precision depth value (range accuracy ±2cm@10m) of the LiDAR point cloud is used to correct the depth jumps in the edge areas of the RGB-D image (such as the junction of the lug and the bag), improving the reliability of the depth data; finally, a dense 3D environment model with a resolution of 5mm×5mm×5mm is generated. The model contains three features: color, depth, and reflection intensity, providing complete environmental information for subsequent lug recognition and path planning.

[0181] (ii) Identification and Positioning Module:

[0182] Multi-feature extraction:

[0183] Three core features of the lugs were extracted from the fused 3D environment model:

[0184] Color characteristics: Extract the feature values ​​of the blue pendant in the RGB color space. (The default color value is LAB color value (30) 70, -70)), each point in the calculation model and Euclidean distance Filter out candidate points that match the color ( );

[0185] Geometric features: Perform local plane fitting on the candidate point cloud (30-50 neighborhood points), and calculate the geometric features of each point. normal vector With curvature and the prior geometric model of the lug (The normal vector is pre-designed to be a long strip with a cross-section of 50-80mm × 5-8mm) curvature By comparison, the basic data of the four-degree geometric features are obtained;

[0186] Reflection intensity features: Extract each point LiDAR reflection intensity value The reflectance value of the metal parts embedded in the lifting lug is about 120-150, which is significantly higher than that of the bag body (40-60) and the environment (20-40). This is used as an important basis for the selection of lifting lugs.

[0187] Improved weighted multi-feature fusion recognition algorithm:

[0188] The optimal positioning point of the lug is calculated using the objective function. The formula is:

[0189] ,in, The optimal positioning point for the lifting lug is the three-dimensional coordinate of the center of the lifting lug. For any point in the three-dimensional model; This is the color feature weighting coefficient, which is set to 0.4-0.6 under normal lighting conditions and 0.1-0.3 under low / strong light conditions (to reduce the dependence on color features). For point Standard color of the lugs The smaller the Euclidean distance, the higher the color matching degree; This is the geometric feature weighting coefficient, which is 0.5-0.7 in dusty environments (because geometric features are more stable) and 0.3-0.5 in clean environments. For point With the prior geometric model of the lug The geometric feature matching degree is such that the smaller the value, the higher the geometric matching degree. The weighting coefficient for reflection intensity features is 0.3-0.5, which is used in the scenario of metal object recognition. For point The higher the reflection intensity value, the more likely it is to be a metal part inside the lug;

[0190] Among them, geometric feature matching degree Calculated using the following formula:

[0191] ,

[0192] in, The normal vector feature weight factor is set to 0.6-0.7 (the normal vector is more sensitive to the hanging lug posture); For point The unit normal vector; for The unit normal vector at the corresponding position; Let be the Euclidean distance between their normal vectors; The curvature feature weighting factor is set to 0.3-0.4. For point The curvature value; for The curvature value at the corresponding position; This is the absolute difference in curvature between the two.

[0193] Dynamic correlation of reflection intensity weight The correlation function is dynamically adjusted based on the average reflectance of the currently identified area:

[0194] ,in, The basic weight value for reflection intensity is set to 0.1-0.2; The reflection intensity gain coefficient is set to 0.005-0.01. To identify the average reflection intensity value of the point cloud within the region (statistical region diameter 200mm); The reflection intensity threshold is set to 80-100 (the maximum reflection intensity of a non-metallic object); Ensure that enhancement is only performed when the average regional reflectance is above the non-metallic threshold. Enhance the priority of metal part recognition;

[0195] Calculation of lug posture:

[0196] based on The point cloud of the surrounding lugs is analyzed, and the major axis (corresponding to the length direction of the lugs) and minor axis (corresponding to the width direction of the lugs) are extracted using principal component analysis (PCA). The attitude angles of the lugs (rotation angles around the X, Y, and Z axes, with an accuracy of ±0.5°) are calculated, and finally, the three-dimensional coordinates of the lugs are output. "Accuracy ±3mm + attitude angle" provides precise target parameters for scaffold alignment;

[0197] (III) Path Planning Module:

[0198] Obstacle extraction and environment modeling:

[0199] Obstacles are extracted from the dense 3D environment model: Areas with a reflection intensity <80 and a color other than blue / white (excluding lifting lugs and ton bags) are marked as obstacles (such as equipment, shelves, and other ton bags), and the minimum bounding box (AABB box) of the obstacle is generated, recording its spatial coordinates and dimensions; at the same time, combined with the physical dimensions of the lifting device actuator (such as a gripper unfolding width of 300mm and an overall lifting device height of 500mm), a "safe distance buffer zone" (500mm around the obstacle) is set to form an environmental constraint model of "passable area - prohibited area";

[0200] Initial path planning:

[0201] An improved A* algorithm is used to plan the initial path of the spreader from its current position to the target position of the lifting lug:

[0202] Heuristic function design:

[0203] ,in, (Euclidean distance weight) (Attitude adjustment cost weighting) The attitude angle deviation between the current path point and the target point (reducing the probability of sharp turns); , , Path node 3D coordinates; , , The three-dimensional coordinates of the target position of the lifting lug;

[0204] Path constraints: Path node spacing 50mm, maximum lifting / lowering speed of the spreader 0.5m / s, horizontal movement speed 1m / s, maximum acceleration 0.2m / s². 2 Ensure the path conforms to the physical limits of the spreader's motion;

[0205] Real-time path correction:

[0206] It receives point cloud data from real-time LiDAR scanning (update frequency 10Hz). If a new obstacle is detected entering the "safe distance buffer zone" (such as a device that suddenly intrudes), it immediately triggers local path replanning: starting from the current position of the spreader and ending at the position of the target lifting lug, it recalculates the local path (path length ≤ 2m), with a replanning time ≤ 100ms, to ensure that the spreader avoids obstacles in time; if the obstacle completely blocks the original path, it sends a "pause command" to the control decision module, and replans after the obstacle is removed.

[0207] (iv) Control Decision Module:

[0208] Phased collaborative instruction generation:

[0209] Based on the positioning results of the lifting lugs and the path planning results, a four-stage collaborative control command is generated, with each stage having "trigger condition - execution action - verification standard":

[0210] Phase 1 (Movement Control): Triggering condition is completion of path planning; Execution action: Control the lifting device to move along the planned path, so that the center point of the electromagnetic adsorption module is aligned with... Alignment in three-axis space (alignment error ≤ ±2mm); Verification standard: alignment error feedback from sensing unit < 2mm;

[0211] Phase Two (Adsorption Control): Triggering condition is successful motion control verification; Execution action: The electromagnetic adsorption module is powered on (current 10A, generating magnetic force ≥200N), adsorbing and lifting the lugs; Verification standard: The sensing unit calculates the rate of change of vertical height of the point cloud in the lug area. The formula is:

[0212] ,in, The average z-coordinate of the point cloud in the lug region before adsorption. The average z-coordinate after adsorption; preset threshold 15%-20%. The threshold is then used to determine if the lift was successful;

[0213] Phase 3 (Gripper Control): Trigger condition is successful adsorption verification; Action executed: The mechanical gripper module first opens (opening degree 300mm), then passes through the lifting lug (lifting device descends 50mm), and finally closes and clamps (clamping force 50-200N, adaptively adjusted according to the weight of the ton bag, 100N for a weight of 1 ton); Verification standard: The gripper pressure sensor feedback that the clamping force reaches the preset value.

[0214] Phase 4 (Release and Handling): Triggering condition is successful gripper control verification; Execution action: The electromagnetic adsorption module is de-energized (releasing magnetic force), the lifting device moves along the planned handling path (to the target unloading position), and upon arrival, the gripper opens to release the lifting lugs; Verification standard: The lidar feedback indicates that the ton bag has detached from the lifting device (distance > 100mm).

[0215] Closed-loop control and feedback processing receive real-time status feedback from the lifting device actuator (such as electromagnetic adsorption current, gripper clamping force, and lifting device position) and secondary identification results from the sensing unit, dynamically adjusting commands: for example, if the alignment error in movement control is >2mm, the lifting device position is automatically fine-tuned (adjusted by 5mm each time until the error is <2mm); if after adsorption... If the threshold is not reached, a retry is triggered (power is turned off for 1 second and then restored, and the maximum number of retries is 3; if it still fails, an alarm will be triggered).

[0216] Emergency troubleshooting:

[0217] Preset fault types and handling strategies:

[0218] Data interruption detection: Immediately control the spreader to stop moving and switch to "local cache mode". If the data is restored within 10 seconds, the operation will continue; otherwise, an alarm will be triggered.

[0219] Insufficient clamping force of the gripper: Increase the pressure of the drive device (increase the air pressure by 0.1MPa when pneumatically driven, and increase the current by 1A when electrically driven). If it is still insufficient, the lifting lug is considered to be damaged, and the operation is stopped and an alarm is triggered.

[0220] If the path encounters a fixed obstacle: Send a "manual intervention request" to the operation and maintenance terminal, along with an image of the obstacle's location. Restart path planning after the obstacle is removed.

[0221] III. Key Technology Principles:

[0222] (I) Principle of Feature-Level Fusion of Multi-Source Data:

[0223] The core concept is "complementary enhancement": binocular vision has the advantage of color and texture information (which can quickly distinguish between the lugs and the bag body), but its depth accuracy is greatly affected by lighting; lidar has the advantage of high-precision distance and reflection intensity (which can accurately locate spatial positions and identify metal parts), but it lacks color information; through spatiotemporal synchronization and coordinate registration, the two types of data are mapped to the same coordinate system, and then through feature-level fusion of "color-completed point cloud + depth-corrected vision", the color discrimination ability of vision is preserved, while the reliability of depth data is improved. This allows the 3D model to simultaneously possess "high recognition (color) + high precision (depth) + material differentiation (reflection intensity)", providing triple feature support for lug recognition, and improving the robustness of recognition by 60% compared to a single sensor;

[0224] (II) Optimization principle of dynamic weighted multi-feature recognition:

[0225] Traditional fixed-weight algorithms are prone to failure in complex environments (e.g., unreliable color features in poor lighting conditions, and easily confused geometric features in dusty conditions). This unit employs an "environmental perception-weight adaptation" mechanism: real-time monitoring of light intensity (obtained from RGB image brightness values) and dust concentration (obtained from point cloud density changes), dynamically adjusting... , , —For example, in low-light environments It dropped to 0.1. Increased to 0.5 (relying on metal reflection for recognition); in dusty environments The accuracy was increased to 0.7 (dependent on geometric features) to ensure that the optimal feature combination can be selected in different environments, and the accuracy of ear recognition is stable at over 98% (compared to only 75%-85% for traditional algorithms).

[0226] (III) Reliability Principle of Phased Closed-Loop Control:

[0227] It adopts a step-by-step logic of "verification-execution," with an independent verification node at each stage: only after the previous stage passes the verification by the sensing unit or the status of the lifting device will it proceed to the next stage, avoiding "one mistake leading to complete failure"; for example, after adsorption, it must pass... Verification ensures the lifting lugs are pulled up to a grabbable position before initiating the gripper action, preventing the gripper from "grabbing empty" due to lug collapse. Simultaneously, real-time feedback and adjustment commands (such as alignment deviation correction and clamping force fine-tuning) compensate for mechanical errors and environmental interference (such as spreader vibration and slight bag offset), ensuring a success rate of ≥99% for each step and ultimately achieving a lug grabbing success rate of ≥96%.

[0228] (iv) Real-time obstacle avoidance path planning principle:

[0229] The strategy employs a two-tiered approach: global planning and local correction. Initially, the A* algorithm plans the globally optimal path to ensure overall efficiency. Real-time LiDAR scanning dynamically updates obstacle information, triggering local replanning to address dynamic obstacles (such as suddenly intruding equipment). Simultaneously, motion constraints (speed, acceleration, and safety distance) of the spreader are incorporated into path planning to avoid planning "unexecutable" paths (such as sharp turns or excessively narrow passages). This ensures that the path is both safe and conforms to the spreader's physical limits, with an obstacle avoidance response time ≤100ms and a collision risk reduced to below 0.1%.

[0230] IV. Collaboration process with other modules of the system:

[0231] (a) Coordination with the sensing unit:

[0232] Data Interaction: The intelligent control unit receives preprocessed RGB-D images (H.265 compressed format), point cloud data (PCAP format), and timestamps from the sensing unit via TCP / IP protocol, with a data transmission delay of ≤50ms. Simultaneously, it sends "parameter configuration commands" to the sensing unit, such as adjusting the camera exposure time (1000μs in low light and 10μs in strong light) and the LiDAR scanning frequency (20Hz in dense environments and 10Hz in open environments) to optimize data acquisition quality.

[0233] Verification command issuance: During the suction and gripping phase of the lifting device, a "secondary identification command" is sent to the sensing unit, requiring it to focus on the lifting lug area to collect data (such as scanning the lifting lug point after suction and cloud computing). The sensing unit will feed back the verification result (success / failure + specific parameters) to the control decision module as the basis for triggering the next stage of action;

[0234] (ii) Coordination with the spreader actuator:

[0235] Command issuance and status feedback: The system sends phased control commands (such as electromagnetic adsorption power on / off, gripper opening and closing angle, and spreader moving speed) to the spreader actuator via the CAN bus, with a command cycle of 10ms. At the same time, it receives real-time status feedback from the spreader (electromagnetic adsorption current, gripper clamping force, spreader current coordinates, and drive device temperature), forming a closed-loop control of "command-feedback-adjustment".

[0236] Action timing coordination: Strictly control the action timing of each module of the lifting device. For example, the electromagnetic adsorption module is only calculated 3 seconds after being powered on (to ensure magnetic stability). After the gripper closes, hold it for 2 seconds (to ensure stable clamping) before cutting off the power to the electromagnetic adsorption module to avoid errors caused by too rapid a transition.

[0237] (iii) Compatibility with the features of specialized ton bags:

[0238] Preset parameter matching: Preset the characteristic parameters of the special ton bag in the intelligent control unit, such as the standard color of the lifting lugs. (LAB(30,70,-70)), Lug geometry model (Dimensions 50-80mm x 5-8mm), metal part reflectivity range (120-150), ensuring that the recognition algorithm can accurately match the special ton bag;

[0239] Batch adaptation adjustment: If you change to a different specification of special ton bag (e.g., increase the lifting lug width from 50mm to 80mm), you can modify it through the human-machine interface (HMI). Parameters and gripper clamping force thresholds do not require redeveloping the algorithm, and the adaptation time is short. minute.

[0240] In this embodiment, the system performs the following steps during the hoisting of the ton bag:

[0241] S1: Collect multi-source perception data of the work area using a binocular vision camera and LiDAR;

[0242] S2: The multi-source sensing data is fused and processed to reconstruct the three-dimensional environment, and an improved weighted multi-feature fusion recognition algorithm is used to identify and locate the three-dimensional coordinates and attitude of the ton bag and its lifting lugs.

[0243] S3: Plan the safe movement path of the spreader actuator to the target lifting lug, and control the spreader actuator to move to the predetermined adsorption position;

[0244] S4: Control the electromagnetic adsorption module to be powered on, and use magnetic force to lift the lug with the embedded metal parts.

[0245] S5: After confirming that the lifting lug has been successfully lifted using a verification and recognition method, control the mechanical gripper module to complete the gripping of the lifting lug;

[0246] S6: Control the electromagnetic adsorption module to shut off power and control the lifting device actuator to move the ton bag to the target location.

[0247] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An unmanned control system for ton bag lifting based on binocular vision camera and lidar, characterized in that: Includes intelligent control unit, spreader actuator, sensing unit and special ton bag; The sensing unit includes a binocular vision camera and a lidar. The sensing unit is installed above the lifting device actuator and is used to collect depth visual information and three-dimensional point cloud information of the work area. The intelligent control unit is communicatively connected to the sensing unit and the lifting device actuator, and is used to process and fuse multi-source sensing data to generate control commands. The intelligent control unit includes: The data fusion module is configured to receive and fuse RGB-D images from a binocular vision camera and 3D point cloud data from a LiDAR, and generate a dense 3D environment reconstruction model with color information through calibration and registration. The identification and positioning module is configured to accurately locate the three-dimensional coordinates and posture of the lug based on a dense three-dimensional environment reconstruction model and an improved weighted multi-feature fusion identification algorithm. The objective function of the improved weighted multi-feature fusion identification algorithm is expressed as: finding the point that minimizes the weighted sum of the matching degree between the Euclidean distance of the color feature and the geometric feature, minus the weighted value of the reflection intensity value. The weighting coefficients of the color feature, geometric feature, and reflection intensity feature are dynamically adjusted according to the sensor accuracy and environmental conditions. The path planning module is configured to plan a safe movement path for the lifting device actuator based on the three-dimensional coordinates and attitude of the lifting lug and obstacle information in the three-dimensional environment reconstruction model. The control decision module is configured to send phased collaborative control commands to the spreader actuator based on a safe movement path. The lifting device actuator includes an independently controllable electromagnetic adsorption module and a mechanical gripper module; The lifting lugs of the special ton bag have embedded metal parts that can be magnetically attracted, and the color of the lifting lugs of the special ton bag has a high contrast with the color of the main body of the bag.

2. The unmanned control system for ton bag lifting based on binocular vision camera and lidar as described in claim 1, characterized in that: The phased collaborative control instructions executed by the control decision module include, in sequence: First-stage movement control: Control the movement of the lifting device actuator so that the center point of the electromagnetic adsorption module is aligned with the identified lifting lug target point in three-axis space; Second stage adsorption control: After maintaining the position alignment, control the electromagnetic adsorption module to generate magnetic force, adsorb and lift the hanging lugs that have collapsed due to their own weight or stacking pressure to a natural hanging state. The third stage of gripper control: After the lifting lug is attracted and lifted, the mechanical gripper module is controlled to perform the actions of opening, passing through the lifting lug, and closing and clamping. The fourth stage of release and handling control: the electromagnetic adsorption module is de-energized to release the magnetic force, and at the same time the lifting device actuator is controlled to move the grabbed ton bag to the target location.

3. The unmanned control system for ton bag hoisting based on binocular vision camera and lidar as described in claim 2, characterized in that: After the second-stage adsorption control, the sensing unit restarts to perform verification and identification. Only after confirming that the lug has been successfully adsorbed and lifted and is in a graspable state will the control decision module trigger the third-stage gripper control. Verification and identification are achieved by calculating the average height change rate of the point cloud in the lug area in the vertical direction before and after adsorption. When the height change rate is greater than the preset height change threshold, the adsorption and lifting are determined to be successful.

4. The unmanned control system for ton bag hoisting based on binocular vision camera and lidar as described in claim 1, characterized in that: The geometric feature matching degree of the improved weighted multi-feature fusion recognition algorithm in the recognition and positioning module is obtained by calculating the weighted sum of the differences in normal vector features and curvature features, wherein the weighting factors of normal vector features and curvature features are used to adjust the contribution of the corresponding features.

5. The unmanned control system for ton bag hoisting based on binocular vision camera and lidar as described in claim 1, characterized in that: The reflection intensity feature weight coefficient is dynamically related to the reflection intensity value, specifically expressed as the product of the basic reflection intensity weight value plus the gain coefficient and the portion of the average reflection intensity value that exceeds the reflection intensity threshold.

6. The unmanned control system for ton bag hoisting based on binocular vision camera and lidar as described in claim 1, characterized in that: The data from the lidar is also used for real-time scanning during the movement of the lifting device actuator, and the path planning module dynamically corrects the movement path based on the real-time point cloud data to achieve obstacle avoidance.

7. The unmanned control system for ton bag lifting based on binocular vision camera and lidar as described in claim 1, characterized in that: The mechanical gripper module is a gripper or hook mechanism arranged symmetrically on both sides, and its opening and closing action is achieved by an electric drive device or a pneumatic drive device.

8. The unmanned control system for ton bag lifting based on binocular vision camera and lidar according to claim 1, characterized in that: The lifting lugs of the special ton bag are blue, the main body of the special ton bag is white, and the outline of the special ton bag is sewn with yellow edging.

9. The unmanned control system for ton bag lifting based on binocular vision camera and lidar according to claim 1, characterized in that: The system performs the following steps during the hoisting of ton bags: S1: Collect multi-source perception data of the work area using a binocular vision camera and LiDAR; S2: The multi-source sensing data is fused and processed to reconstruct the three-dimensional environment, and an improved weighted multi-feature fusion recognition algorithm is used to identify and locate the three-dimensional coordinates and attitude of the ton bag and its lifting lugs. S3: Plan the safe movement path of the spreader actuator to the target lifting lug, and control the spreader actuator to move to the predetermined adsorption position; S4: Control the electromagnetic adsorption module to be powered on, and use magnetic force to lift the lug with the embedded metal parts. S5: After confirming that the lifting lug has been successfully lifted using a verification and recognition method, control the mechanical gripper module to complete the gripping of the lifting lug; S6: Control the electromagnetic adsorption module to shut off power and control the lifting device actuator to move the ton bag to the target location.

Images