An adaptive collaborative loading and unloading method based on multi-modal visual perception

Abstract

The application belongs to the technical field of truss loading and unloading equipment cooperative control and machine vision perception, and discloses a self-adaptive cooperative loading and unloading method based on multi-modal visual perception, which uses a 3D global camera, a local 3D auxiliary camera, multiple trusses, an executor carried on the trusses, a frame track for the movement of the trusses and carriers located on both sides of the carriage; the method is executed in the following steps: collecting data based on the 3D global camera to determine the operation mode; establishing initial partitioning based on the area of the goods and the constraint of the executor-carrier; establishing a density integral dynamic partitioning and a truss successive decreasing mechanism; dynamic adjustment of the stroke of the telescopic rod and three-dimensional cooperative motion path planning; multi-modal visual cooperative precise operation and safety protection self-adaptive loading and unloading. The application realizes the self-adaptive matching of the loading and unloading equipment and the distribution of the goods, and improves the loading and unloading efficiency.

Description

Technical Field

[0001] This invention belongs to the field of collaborative control and machine vision perception technology for truss loading and unloading equipment, specifically involving an adaptive collaborative loading and unloading method for multiple trusses and multiple actuators based on multimodal visual perception. Background Technology

[0002] In modern logistics and industrial production, the loading and unloading of bagged materials is a key link connecting warehousing and transportation, and its efficiency and safety are directly related to the overall operational efficiency of the supply chain.

[0003] Existing bagged material loading and unloading equipment mostly adopts a single-girder, single-actuator operating structure, which cannot adaptively adjust according to the actual size of the truck bed and the distribution of goods. This results in low loading and unloading efficiency, and the overall efficiency is insufficient to meet the high-efficiency flow requirements of modern logistics. In addition, the telescopic boom of existing equipment must be retracted to a fixed height before horizontal movement. During the later stages of unloading, when the goods have been significantly lowered, a large number of ineffective lifting and lowering operations are still performed, generating a lot of meaningless and ineffective vertical lifting and lowering movements, which wastes valuable operation time. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive collaborative loading and unloading method for multi-gantry and multi-actuator systems based on multimodal visual perception, so as to solve the problem of low loading and unloading efficiency of gantry robots in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an adaptive collaborative loading and unloading method based on multimodal visual perception, which uses a 3D global camera, a local 3D auxiliary camera, multiple trusses, actuators mounted on the trusses, a frame track for moving the trusses, and vehicles located on both sides of the carriage. The 3D global camera establishes a three-dimensional Cartesian coordinate system with its own optical center as the origin; the parameters of the carriage are as follows: total length in the X direction is L, total width in the Y direction is W; the side boundaries are Y... left and Y right A side railing is installed along the entire length of the X-axis, with a height of H. wall Multiple trusses move horizontally along the X-axis, and each truss extends across the full width of the carriage along the Y-axis. It is equipped with two sets of actuators on the left and right sides, which can slide along the Y-axis and rise and fall along the Z-axis. All three axes can move simultaneously. The actuator is equipped with a local 3D auxiliary camera at the end of the actuator to collect three-dimensional data of the bagged material at close range, and the actuator drives the gripper to complete the precise gripping of the goods based on the collected data. This method is executed in the following order: Step S1: Determine the operating mode based on data acquired by the 3D global camera; Step S2: Initial partitioning based on cargo area and the actuator-vehicle constraint; Step S3: Establish a dynamic zoning of density integrals and a successive reduction mechanism for truss beams; Step S4: Dynamic adjustment of telescopic pole stroke and three-dimensional collaborative motion path planning; Step S5: Multimodal vision collaborative precision operation and safety protection adaptive loading and unloading.

[0006] Preferably, in step S1, the 3D global camera acquires the three-dimensional point cloud data of the carriage in real time, and calculates the following parameters: L is the total length of the carriage in the X direction, W is the total width of the carriage in the Y direction, H is the maximum loading height of the carriage, and H wall H is the height of the side railing of the carriage. stack H represents the total height of the stacked goods. cargo(x,y) The real-time stacking height H of the goods at location (x,y) railing(x,y) The height of the fence at position (x, y) is given by Y. left and Y right The value at point H wall The remaining positions are all 0; Y left and Y right Let Y be the Y coordinates of the left and right boundaries of the carriage, respectively; calculate the carriage's vacancy rate R. free .

[0007] Ideally, the vacancy rate R of the carriages is... free =(Total volume of carriage - Total volume of cargo) / Total volume of carriage; The preset idle rate threshold k ranges from 0 to 1; R free Comparison with the preset idle rate threshold k: If R free If the value is greater than k, then step S2 is executed and the system is run in unloading mode. If R free If k ≤ k, then execute step S2 and run in loading mode.

[0008] Preferably, step S2 specifically includes the following steps: Step S2.1, Extract the effective working area and vehicle distribution: Calculate the coverage area A of the cargo on the XY projection plane from the 3D point cloud data of the carriage collected by the 3D global camera. cargo The position and side of the vehicle are marked simultaneously: the coordinates of the vehicle on the left are X. L(i) (i=1,..., N) L The vehicle coordinates X on the right side R(j) (j=1,..., N) R ), N L With N R It is a positive integer, used as a constraint for setting partition boundaries and allocating the executor; Step S2.2, calculate the theoretical number of partitions: the optimal working area of ​​a single truss is A, D is the optimal independent working width in the X direction of a single truss, and each truss is equipped with one actuator on each side; if the working mode is unloading mode, the theoretical number of partitions is N. theory =A cargo / A,N theory Rounding the value up, the corresponding number is N. theory Based on the aforementioned truss; if the operation mode is loading mode, the theoretical number of zones is N. theory =L / D, N theory Rounding the value up, the corresponding number is N. theory The truss beam described above; Step S2.3, the vehicle constraints and the partition adjustment: calculate the maximum number of trusses that the vehicles on the left and right sides can support respectively, with the left side supporting N. L ×2 of the aforementioned trusses, with N supports on the right side. R ×2 of the aforementioned trusses; Step S2.4, binding the actuator to the vehicle: Calculate the X-direction distance from the center of each partition to all vehicles on the same side: left distance D L(j,i) =|X L(i) - X center(j) |, distance D to the right R(j,i) =|X R(i) - X center(j) |; Match the actuators on the same side with the vehicle according to the principle of closest proximity: If N L <N init N init If the initial number of trusses is used, then adjacent left-side actuators are allowed to share one left-side vehicle, and a maximum of two left-side actuators can share one vehicle. If N R <N init If so, adjacent right-side actuators are allowed to share one right-side vehicle, and a maximum of two right-side actuators can share one right-side vehicle; after completing this step, proceed to step S3.

[0009] Preferably, in step S2.3, (1) If N theory ≤N L ×2 and N theory ≤N R ×2, then the initial number of trusses N init =N theory The number of partitions and the number of trusses are determined according to theoretical values, and all actuators are working normally; (2) If N theory >N L ×2 and N theory ≤NR ×2, then the initial number of trusses N init =N theory If the carriers on the left are insufficient, the excess actuators on the left need to be deactivated. The deactivated actuators are moved into the girder storage area, and the girder where they are located is loaded and unloaded by the actuators on the right alone. (3) If N theory ≤N L ×2 and N theory >N R ×2, then the initial number of trusses N init =N theory The number of vehicles on the right is insufficient, so the excess actuators (N) on the right need to be deactivated. theory - N R (×2) The deactivated actuators are moved into the storage areas at both ends of the truss, and the truss on which they are located is loaded and unloaded by the actuator on the left side alone. (4) If N theory >N L ×2 and N theory >N R If the value is ×2, then the number of partitions needs to be reduced, and the required initial number of trusses N needs to be calculated. init =min(N theory -N R ×2, N theory -N L ×2); (5) If the operation mode is unloading, each of the above-mentioned trusses is evenly distributed along the effective coverage area of ​​the goods; if the operation mode is loading, each of the above-mentioned trusses is evenly arranged along the X-axis of the carriage to form N now =N init An initial partition.

[0010] Preferably, step S3 specifically includes the following steps: Step S3.1, construct the material density function inside the carriage: In each cycle, the 3D global camera acquires the latest 3D point cloud data of the carriage; the Z-axis cross-sectional area of ​​all point clouds at each position is aggregated along the X-axis to construct the material density function p(x,t); the total volume of material on the carriage is calculated according to formula V total(t) =∫p(x,t)dx; after completing this step, proceed to step S3.2; Step S3.2: Divide the material into equal dynamic zones according to the total amount of material, and solve the zone integral equation: The boundary positions of each zone are determined; the zone width is automatically narrowed in areas with dense material and automatically widened in areas with sparse material; the material volume of each zone is... After completing this step, proceed to step S3.3. Step S3.3, Adjustment of boundary lines for each zone: Calculate the material volume ΔV(t) within the buffer zone of each boundary line, where α is the volume change coefficient, ranging from 0 to 1. Let the position of the kth boundary line be x. k If the buffer band half-width is δ (a preset constant), then That is, the sum of the material volumes within the δ range on both sides of each dividing line; If ΔV(t)≥α·V total(t) / N now Then, proceed to step S3.2 to resolve the boundary equations to translate the boundary, N now Remain unchanged; If ΔV(t) < α·V total(t) / N now If the current boundary remains unchanged, proceed to step S3.4 after completing this step. Step S3.4, truss beams decrease successively: set V min Preset unloading and emptying thresholds for individual zones, based on the total volume of multiple bags; V max A loading saturation threshold is preset for a single partition; each cycle iterates through all partitions to determine whether the truss reduction is triggered.

[0011] Preferably, in step S3.4, all partitions are traversed according to the following steps: Step ①, Determine the volume threshold for loading / unloading mode partitions: If the operation mode is loading, and there exists a partition j such that V j(t) ≥V max If the candidate partition triggers the reduction of the truss beam, proceed to step ② to perform load verification; if all partitions in this mode do not meet this condition, the reduction will not be triggered in this cycle, and proceed to step S3.3. If the operation mode is unloading, and there exists a partition j such that V j(t) ≤V min If the candidate partition triggers the reduction of the truss beam, proceed to step ② to perform load verification; if all partitions in this mode do not meet this condition, the reduction will not be triggered in this cycle, and proceed to step S3.3. Step 2, perform load ratio verification: After the candidate trigger decreases, it is necessary to verify that N now Reduce to N now After -1, check whether the remaining trusses can bear the current remaining total workload; if the check passes, proceed to step ③ to decrease; if the check fails, suspend the decrease in this cycle and wait for more partitions to complete naturally before starting the check again from step ① in the next cycle. Load verification formula for loading mode: V remain_load =Σmax(0,V max -V j(t)), that is, the sum of the volumes that each of the aforementioned partitions still needs to be filled; Verification condition: V remain_load / (N now -1)≤V fair_load ; Unloading mode load verification formula: V remain_unload =ΣV j(t) That is, the sum of the volumes of all the aforementioned partitions that have not yet been removed; Verification condition: V remain_unload / (N now -1)≤V fair_load ; Where V fair_load This is a preset upper limit on the additional workload that a single truss can withstand; Step ③, perform the decrement: reduce N now Updated to N now -1, with the new N now Repeat step S3.2 to solve the partition boundary equations; Step 4: Handle the case where multiple zones meet the criteria simultaneously and loop: If multiple zones simultaneously meet the triggering condition in this round, only the decrease of one zone is processed, and the rest are deferred to the next cycle; each time only one of the aforementioned trusses is decreased, and the total number of decrease levels is N. max After reduction, the excess trusses are moved into the storage area along the X-axis; steps S2.4 to S3 are repeated until N is reached. now =0, all the aforementioned trusses have been stored.

[0012] Preferably, step S4 specifically includes the following steps: Step S4.1, adaptive adjustment of the telescopic rod stroke: Each time the actuator moves horizontally, it only needs to keep the end of the gripper above the height of the effective obstacles along the current path; the height of the effective obstacles changes dynamically with the loading and unloading progress; when the cargo is significantly reduced in the later stages of unloading, the actuator only needs to retract to a position above the fence for safe passage, without needing to return to its highest position when the vehicle is full; the actual retraction height each time is determined by H in step S4.2. eff(x,y) The path is dynamically provided at each location; after completing this step, proceed to step S4.2. Step S4.2, construct a dual-condition effective obstacle height field: the actuator crosses the side fence of the carriage each time it travels between the vehicle and the target point of the carriage; The carriage contains both side railings of varying heights and bagged materials; the height of the side railings is H. wall Located in Y left and Y right At the location, the height H of the bagged material cargo(x,y) The effective obstacle height field is obtained by extracting the obstacle height from the 3D point cloud in real time and taking the larger value of the two values, plus the additional height of the actuator's current state; after completing this step, proceed to step S4.3. No-load state (the actuator is not holding any material):

[0013] Loaded state (the actuator holds material):

[0014] The meanings of the parameters in the formula: H cargo(x,y) H represents the real-time material stacking height extracted from the 3D point cloud. railing(x,y) The height of the fence is in Y. left and Y right H is the location wall The rest are 0; dL is the preset safety clearance; H load The thickness of the material held by the actuator in the Z-axis direction is obtained from a local 3D camera point cloud; during the unloading process, H cargo(x,y) Decrease, H eff As it decreases, the clearance elevation also decreases accordingly; Step S4.3, construct loading and unloading paths (both including empty and loaded sections): the path of the actuator when loading and unloading materials each time needs to cross the side fence of the carriage; Loading path: First segment unloaded, the actuator starts from the current position at H... eff_empty(x,y) For height constraints, proceed to the fixed vehicle position P. carrier The first section grabs the material; the second section carries the load, and after grabbing the material, it moves at H... eff_loaded(x,y) For height constraints, from the vehicle position P carrier Directly to the stacking point P inside the carriage place After releasing the material, it becomes unloaded; Unloading path: First segment unloaded, the actuator uses H eff_empty(x,y) For height constraints, from the vehicle position P carrier Proceed to the grab point P inside the carriage grasp The second stage carries the load and grabs the material at an H-axis speed. eff_loaded(x,y) For height constraints, from P grasp Return to the vehicle position P carrier Release the material; after completing this step, proceed to step S4.4; Step S4.4, Solving for the shortest path in the peak segment: Sample along the XY path to obtain H at each point. effThe value is used to form a one-dimensional obstacle height profile; all local maxima (obstacle peaks) in the profile are extracted, and the complete path is divided into several independent segments using these as dividing nodes; each segment uses the maximum obstacle peak height of the segment as the minimum passage constraint, and the three axes move simultaneously (the truss moves along the X-axis, the actuator slides along the Y-axis, and the actuator rises and falls along the Z-axis) to complete the spatial linear motion of the segment. The trajectories of each segment are spliced ​​together to obtain the complete three-dimensional shortest path; the height is raised only at the obstacle peak position, and each segment retains only the minimum necessary passage height. The actuator does not need to be retracted to a fixed height throughout the entire process; after completing this step, proceed to step S5.

[0015] Segmented triaxial oblique line length: If the segment is a target grabbing and releasing entry / exit segment, or the maximum obstacle height H in this segment... q >z start ,but ; If the segment is not the target entry / exit segment and H q ≤z start Then L q =S q ; In the formula S q H is the length of the XY projection of this segment. q z is the maximum obstacle height within this segment. start This is the Z-coordinate of the starting point of this segment.

[0016] Preferably, in step S5, each cycle is executed in the following order of sub-steps: (a) The 3D global camera updates the 3D point cloud in real time, constructs the material density function p(x,t), and calculates V by integration. total With each partition V j ; (b) Perform the two-level partition control in steps S3.3 and S3.4; (c) Construct corresponding H according to each working condition. eff Execute step S4 to plan the 3D path; (d) Issue X-axis commands for the truss and Y-axis and Z-axis commands for the actuators; (e) The local 3D auxiliary camera monitors the material positioning in real time, and the actuator completes precise gripping and placement; the idle truss is stored in the storage area set at both ends of the frame X-axis, and the idle actuator is stored in the storage area set at both ends of the truss; repeat the above sub-steps until N. now =0, all trusses have been stored, and the operation is complete.

[0017] Preferably, each actuator on the left and right sides of each truss is bound to a carrier, and each carrier is shared by a maximum of two actuators. The two actuators sharing the same carrier use the carrier alternately. Beneficial effects

[0018] This invention addresses the problem in existing technologies where a single truss and single actuator cannot adaptively adjust to the actual dimensions of the cargo compartment and the distribution of goods. It employs a multi-truss, multi-actuator parallel operation architecture, constructing a material density function based on 3D point cloud data acquired by a 3D global camera. The operating area is dynamically divided according to the actual distribution of goods, achieving adaptive matching between loading / unloading equipment and cargo distribution, thus improving loading / unloading efficiency. Furthermore, it addresses the issue that existing equipment requires the telescopic boom to retract to a fixed height before horizontal movement, and that a large amount of ineffective vertical lifting motion still occurs in the later stages of unloading when the cargo has significantly decreased in height. This invention constructs a dual-condition effective obstacle height field. When the actuator moves horizontally, it only needs to maintain a position higher than the effective obstacle height along the current path. Each path segment retains only the minimum necessary passage height. During unloading, the lifting height automatically decreases as the cargo height decreases, eliminating ineffective lifting motions, shortening the operating path, and improving operating efficiency. Attached Figure Description

[0019] Figure 1 This is a top view of the carriage in this invention; Figure 2 This is a schematic diagram of loading / unloading for the present invention; the installation positions of the global 3D camera and the local 3D camera marked in the figure are only exemplary arrangements, and the camera arrangements are not limited to the illustrated scheme; Figure 3 This is a three-dimensional schematic diagram of the carriage of the present invention; Figure 4 This is a top view of the carriage in an embodiment of the present invention; Figure 5 This is a flowchart of the present invention. Detailed Implementation

[0020] Embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0021] This invention provides an adaptive collaborative loading and unloading method based on multimodal visual perception. This method utilizes a 3D global camera, local 3D auxiliary cameras, multiple trusses, actuators mounted on the trusses, frame tracks for truss movement, and vehicles located on both sides of the carriage, such as... Figure 1 , Figure 3 As shown in Figure 2, an exemplary installation method for a 3D global camera and a local 3D auxiliary camera is illustrated. The 3D global camera establishes a three-dimensional Cartesian coordinate system with its own optical center as the origin; the parameters of the carriage are as follows: the total length of the X-axis depth is L, and the total width of the Y-axis is W; the two side boundaries Y_left and Y_right are provided with side railings along the entire length of the X-axis, with a height of H_wall; multiple trusses move horizontally along the X-axis, and each truss extends across the entire width of the carriage along the Y-axis, equipped with two sets of actuators on the left and right sides, which can slide along the Y-axis and rise and fall along the Z-axis, and the three axes can move simultaneously; The actuator is equipped with a local 3D auxiliary camera at the end of the actuator to collect three-dimensional data of the bagged material at close range, and the actuator drives the gripper to complete the precise gripping of the goods based on the collected data. like Figure 5 As shown, the method is executed in the following steps in sequence: Step S1: Determine the operating mode based on data acquired by the 3D global camera; Step S2: Initial partitioning based on cargo area and the actuator-vehicle constraint; Step S3: Establish a dynamic zoning of density integrals and a successive reduction mechanism for truss beams; Step S4: Dynamic adjustment of telescopic pole stroke and three-dimensional collaborative motion path planning; Step S5: Multimodal vision collaborative precision operation and safety protection adaptive loading and unloading. Example 1

[0022] Initially, the total volume of goods in the carriage is 100 cubic meters, evenly distributed within the carriage. The preset idle rate threshold k is 0.5, the volume change coefficient α is 0.3, the safety distance dL is 0.2 meters, and the preset upper limit V of the additional workload that a single truss can bear is... fair_load It is 40 cubic meters.

[0023] Step S1: Determine the operating mode based on data acquired by the 3D global camera; A 3D camera collects real-time 3D point cloud data of the vehicle compartment, obtaining the following data: the total length L in the X direction of the vehicle is 12 meters, the total width W in the Y direction of the vehicle compartment is 3 meters, and the height H of the side railings on both sides of the vehicle is... wall =2.5 meters, the real-time stacking height H of the goods at (x,y) cargo(x,y H railing(x,y) Let (x, y) be the height of the fence at position (x, y), in Y left and Y right The value is 2.5 meters at one location and 0 meters at all other locations; Y left and Y right These are the Y-coordinates of the left and right boundaries of the carriage, respectively. The carriage's vacancy rate, R... free R is 0.05. free =0.05<0.5, execute step S2 and run in unloading mode.

[0024] Step S2: Initial partitioning based on cargo area and actuator-vehicle constraints; Step S2.1: Extract the effective working area and vehicle distribution; Calculate the coverage area A of the cargo on the XY projection plane from the 3D point cloud data of the carriage acquired by the 3D global camera. cargo It is 36 square meters.

[0025] There are N vehicles on the left. L =3, X-axis coordinate of the left vehicle X L(1) =1.5, X L(2) =6,X L(3) =10.5; There are N vehicles on the right. R =2, Y-axis coordinate of the right vehicle X R(1) =3,X R(2) =11.

[0026] Step S2.2: Calculate the theoretical number of zones: The optimal working area of ​​a single truss is A = 12 square meters, and each truss is equipped with one actuator on the left and one on the right. The current operating mode is unloading mode, with a theoretical number of zones N. theory =A cargo / A=36 / 12=3, which means 3 trusses are needed.

[0027] Step S2.3, Vehicle Constraints and Zoning Adjustment: Calculate the maximum number of trusses that the left and right vehicles can support respectively. The left side can support 3×2=6 trusses (each vehicle can be shared by a maximum of 2 left actuators), and the right side can support 2×2=4 trusses (each vehicle can be shared by a maximum of 2 right actuators). The following cases are handled: N theory =3≤6 and N theory =3≤4, determine the number of partitions N according to theoretical values. init =N theory =3, determine the number of truss beams N according to theoretical values. now =3, all actuators are working normally.

[0028] The current operating mode is unloading, such as Figure 4 The trusses shown are evenly distributed along the effective cargo coverage area to form three initial zones.

[0029] Zone 1: X = [0, 4] meters, material volume V1 ≈ 33.3 cubic meters Zone 2: X = [4,8] meters, material volume V2 ≈ 33.3 cubic meters Zone 3: X = [8, 12] meters, material volume V3 ≈ 33.3 cubic meters Step S2.4: Binding the actuator to the vehicle; Calculate the X-direction distance from the center of each zone to all vehicles on the same side: Left side distance D L(j,i) =|X L(i) -X center(j) |, distance D to the right R(j,i) =|X R(i) -X center(j) |; Match the actuators and vehicles on the same side according to the principle of proximity.

[0030] The center of partition 1 is X=2m, the center of partition 2 is X=6m, and the center of partition 3 is X=10m.

[0031] Section 1 is closest to vehicle L1, at a distance of 0.5 meters, therefore the left actuator of section 1 is bound to vehicle L1; Partition 2 is closest to vehicle L2, at a distance of 0 meters, therefore the left actuator of partition 2 is bound to vehicle L2; Section 3 is closest to vehicle L3, at a distance of 0.5 meters, therefore the left actuator of section 3 is bound to vehicle L3; Partition 1 is closest to vehicle R1, at a distance of 1 meter, therefore the left actuator of partition 1 is bound to vehicle R1; Section 2 is closest to vehicle R1, at a distance of 3 meters, therefore the left actuator of section 2 is bound to vehicle R1; Section 3 is closest to vehicle R2, at a distance of 1 meter, therefore the left actuator of section 3 is bound to vehicle R2; The right actuators of partition 1 and partition 2 alternately use vehicle R1. After completing this step, proceed to step S3.

[0032] Step S3: Establish a dynamic zoning of density integrals and a successive reduction mechanism for truss beams; Step S3.1: Construct the material density function; In each cycle, the 3D global camera acquires the latest 3D point cloud data of the carriage. The Z-axis cross-sectional area of ​​all point clouds at each location is aggregated along the X-axis to construct the material density function p(x,t). The total material volume V on the carriage is... total(t) =∫p(x,t)dx=100 cubic meters. After completing this step, proceed to step S3.2.

[0033] Step S3.2: Dynamically partition the work area and other volumes; Initially, the cargo inside the carriage is evenly distributed, and the total volume of the carriage is V. total(t) =100m 3 N now =3. Dynamically partitioned according to the principle of equal volume. The work volume allocated to each area is The region boundaries are determined by integrating the density function p(x,t) along the X-axis, ensuring a uniform working volume across all regions. The region division results are as follows: Zone 1: X=[0,4]m, the volume of goods V1 in the zone is approximately 33m³. 3 , Zone 2: X = [4,8]m, the cargo volume V2 within the zone is approximately 33m³. 3 , Zone 3: X = [8, 12] m, the cargo volume V3 within the zone is approximately 33 m³. 3 , The system automatically narrows the area width in densely packed areas and widens the area width in sparsely packed areas to ensure a uniform distribution of workload among the actuators. After completion, proceed to step S3.3.

[0034] Step S3.3: Fine-tune the dividing line; For the volume of goods within the integration area on both sides of each dividing line To conduct statistics.

[0035] Taking the boundary line X = 4m between region 1 and region 2 as an example, the volume change coefficient α is 0.3. Based on the 3D global camera point cloud computing, X = [3.7, 4.3], the volume of goods near this region ΔV(t) = 2.5m³. 3 α·V total(t) / N now =0.3×100 / 3≈11.1m 3 ΔV(t) = 2.5 < 11.1, so keep the current boundary unchanged.

[0036] Taking the boundary line X = 8m between region 2 and region 3 as an example, the calculated value is ΔV(t)≈2.3m. 3 V(t) = 2.0 < 11.1, so keep the current boundary unchanged.

[0037] The current boundary position is reasonable and no adjustment is needed. Proceed to step S3.4 after completion.

[0038] Step S3.4: Decrease the number of regions; Preset parameters for unloading mode: V min =2m 3 (Threshold for completion of unloading operations in a single area), V fair_load =40m 3 (The maximum amount of unloading work that a single area can handle).

[0039] Each decrease is performed in the following three steps: Step ① Threshold determination: 3D global camera detects V in each region j(t) If there is V j(t) ≤V min If so, then the region is a candidate decreasing region.

[0040] Step ② Load transfer verification (unloading mode): V remain_unload / (Nnow -1)≤V fair_load Only after the verification passes can the rank be decreased; otherwise, the current rank is maintained and the rank is waited for.

[0041] Step ③ execute the decrement: N now =N now -1, press new N now Return to step S3.2 for equal-volume repartitioning.

[0042] First decrease: After several unloading cycles, the current status is: V 1(t) =25m 3 V 2(t) =27m 3 V 3(t) =1.8m 3 V total ≈53.8m 3 .

[0043] Step ① Threshold determination: V 3(t) ≤V min =2, the condition is met, and the number of candidates decreases.

[0044] Step ② Load transfer verification: V remain_unload / (N now -1) =52 / (3-1)=26≤V fair_load =40, verification passed.

[0045] Step ③ execute the decrement: N now =3-1=2. Press N. now =2 equal-volume re-partitioning: new partition 1 X=[0,6]m, new partition 2 X=[6,12]m, each partition is approximately 26.9m³.

[0046] Second decrease: After the new partition is completed, the truss continues to unload automatically. 1(t) =24m 3 V 2(t) =1.5m 3 V total ≈25.5m 3 .

[0047] Step ① Threshold determination: V 2(t) =1.5≤V min =2, the condition is met, and the number of candidates decreases.

[0048] Step ② Load transfer verification: V remain_unload / (N now -1) =24 / (2-1)=24≤V fair_load =40, verification passed.

[0049] Step ③ execute the decrement: N now =2-1=1. Press N. now =1 equal volume rezoning: the entire region X=[0,12]m, V total =25.5m 3 .

[0050] Third decrease: Continue unloading, assuming at some point V total =1.5m 3 .

[0051] Step ① Threshold determination: V total =1.5≤V min =2, the condition is met, and the number of candidates decreases.

[0052] Step ② Load transfer verification: N now When the value is 1, the operation ends immediately upon decrementing, requiring no transfer or verification, and can be executed directly.

[0053] Step ③ execute the decrement: N now =1-1=0. All trusses have been stored and unloading is complete.

[0054] Step S4: Actuator stroke dynamic optimization and 3D cooperative motion path planning algorithm: This algorithm is the basic algorithm for actuator motion control in each work cycle, and it is called every time the actuator needs to move.

[0055] Step S4.1: Adaptive adjustment of the telescopic rod stroke; Each time the actuator moves horizontally, it only needs to keep the end of the gripper above the height of the effective obstacles along the current path. This effective obstacle height changes dynamically with the loading and unloading progress: when the cargo volume decreases significantly in the later stages of unloading, the actuator only needs to retract to a position above the barrier for safe passage, without needing to return to its highest position when the truck is full. The actual retraction height each time is determined by H in step S4.2. eff(x,y) The path is dynamically provided at each location. After completing this step, proceed to step S4.2.

[0056] Step S4.2: Construct the effective obstacle height field for dual actuators; In this embodiment, the loading and unloading channel has only one obstacle peak on the side railing of the carriage, with no other material stacking peaks. The material stacking height H_cargo(x,y) and railing height H_wall(x,y) are detected in real time by 3D point cloud, with dL being 0.2 meters; the material thickness H grasped by the actuator is estimated by local 3D camera pre-scanning. load =0.2 meters. The effective obstacle height field is constructed by combining the current state of the actuator (unloaded / loaded) as input.

[0057]

[0058]

[0059] Step S4.3: Construct the unloading route; Unloading path: The unloaded path is for the actuator to move from the carrier position (4.5, 4.0, 0.5) to the grab point (2.0, 6.0, 1.2) inside the carriage; the loaded path is for the actuator to grab the material and then return from the grab point (2.0, 6.0, 1.2) to the carrier position (4.5, 4.0, 0.5) to release the material. After completing this step, proceed to step S4.4.

[0060] Step S4.4: Solving spatial segment paths and 3D obstacle avoidance; The effective obstacle height values ​​at each location are sampled along the XY path to form a one-dimensional obstacle height profile. In this embodiment, there is only one obstacle peak at the side fence from the vehicle to the grab point.

[0061] The unloaded path is divided into two segments: the first segment is from the vehicle (4.5, 4.0, 0.5) to the fence peak (3.25, 5.0, 2.7). The second segment is from the fence peak (3.25, 5.0, 2.7) to the gripping point (2.0, 6.0, 1.2). .

[0062] The load path is divided into two segments: the first segment is from the grab point (2.0, 6.0, 1.2) to the fence peak point (3.25, 5.0, 2.9). The second segment extends from the fence peak (3.25, 5.0, 2.9) to the vehicle (4.5, 4.0, 0.5). .

[0063] Step S5: Multimodal vision-based collaborative precision operation and adaptive loading and unloading with safety protection; each cycle is executed in the following order of sub-steps: (a) 3D global camera scan to construct the material density function p(x,t), and integral calculation of V total and each partition V j ; (b) Perform two-layer zoning control – boundary fine-tuning and truss reduction control – in steps S3.3 and S3.4; (c) Call the algorithm in step S4 to dynamically calculate H based on the current operating conditions of each partition. eff Plan the optimal three-dimensional path; (d) Convert the path into X-axis movement commands, Y-axis slide commands, and Z-axis lifting commands and send them to each actuator; (e) A local 3D auxiliary camera monitors the material positioning in real time, and the actuator completes precise grasping and releasing.

[0064] Repeat steps (a) through (e) above until N. now =0, all trusses have been stored, and the operation is complete.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any minor modifications, equivalent substitutions, and improvements made to the above embodiments based on the technical essence of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive collaborative loading and unloading method based on multimodal visual perception, characterized in that, The method uses a 3D global camera, a local 3D auxiliary camera, multiple trusses, actuators mounted on the trusses, frame tracks for moving the trusses, and vehicles located on both sides of the carriage. The 3D global camera establishes a three-dimensional Cartesian coordinate system with its own optical center as the origin; the parameters of the carriage are as follows: total length in the X direction is L, total width in the Y direction is W; the side boundaries are Y... left and Y right A side railing is installed along the entire length of the X-axis, with a height of H. wall Multiple trusses move horizontally along the X-axis, and each truss extends across the full width of the carriage along the Y-axis. It is equipped with two sets of actuators on the left and right sides, which can slide along the Y-axis and rise and fall along the Z-axis. All three axes can move simultaneously. The actuator is equipped with a local 3D auxiliary camera at the end of the actuator to collect three-dimensional data of the bagged material at close range, and the actuator drives the gripper to complete the precise gripping of the goods based on the collected data. This method is executed in the following order: Step S1: Determine the operating mode based on data acquired by the 3D global camera; Step S2: Initial partitioning based on cargo area and the actuator-vehicle constraint; Step S3: Establish a dynamic zoning of density integrals and a successive reduction mechanism for truss beams; Step S4: Dynamic adjustment of telescopic pole stroke and three-dimensional collaborative motion path planning; Step S5: Multimodal vision collaborative precision operation and safety protection adaptive loading and unloading.

2. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 1, characterized in that, In step S1, the 3D global camera acquires real-time 3D point cloud data of the carriage, and calculates the following parameters: L is the total length of the carriage in the X direction, W is the total width of the carriage in the Y direction, H is the maximum loading height of the carriage, and H wall H is the height of the side railing of the carriage. stack H represents the total height of the stacked goods. cargo(x,y) H represents the real-time stacking height of goods at location (x, y). railing(x,y) The height of the side fence at position (x, y) is given by Y. left and Y right The value at point H wall The remaining positions are all 0; Y left and Y right Let Y be the Y coordinates of the left and right boundaries of the carriage, respectively; calculate the carriage's vacancy rate R. free .

3. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 2, characterized in that, Carriage vacancy rate R free =(Total volume of carriages - Total volume of cargo) / Total volume of carriages, with the preset idle rate threshold k ranging from 0 to 1; R free Comparison with the preset idle rate threshold k: If R free If the value is greater than k, then step S2 is executed and the system is run in unloading mode. If R free If k ≤ k, then execute step S2 and run in loading mode.

4. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S2.1, Extract the effective working area and vehicle distribution: Calculate the coverage area A of the cargo on the XY projection plane from the 3D point cloud data of the carriage collected by the 3D global camera. cargo The position and side of the vehicle are marked simultaneously: the coordinates of the vehicle on the left are X. L(i) (i=1,..., N) L The vehicle coordinates X on the right side R(j) (j=1,..., N) R ), N L With N R It is a positive integer, used as a constraint for setting partition boundaries and allocating the executor; Step S2.2, calculate the theoretical number of partitions: the optimal working area of ​​a single truss is A, D is the optimal independent working width in the X direction of a single truss, and each truss is equipped with one actuator on each side; if the working mode is unloading mode, the theoretical number of partitions is N. theory =A cargo / A,N theory Rounding the value up, the corresponding number is N. theory Based on the aforementioned truss; if the operation mode is loading mode, the theoretical number of zones is N. theory =L / D, N theory Rounding the value up, the corresponding number is N. theory The truss beam described above; Step S2.3, the vehicle constraints and the partition adjustment: calculate the maximum number of trusses that the vehicles on the left and right sides can support respectively, with the left side supporting N. L ×2 of the aforementioned trusses, with N supports on the right side. R ×2 of the aforementioned trusses; Step S2.4, binding the actuator to the vehicle: Calculate the X-direction distance from the center of each partition to all vehicles on the same side: left distance D L(j,i) =|X L(i) - X center(j) |, distance D to the right R(j,i) =|X R(i) - X center(j) |; Match the actuators on the same side with the vehicle according to the principle of closest proximity: If N L <N init N init If the initial number of trusses is used, then adjacent left-side actuators are allowed to share one left-side vehicle, and a maximum of two left-side actuators can share one vehicle. If N R <N init Then, adjacent right-side actuators are allowed to share one right-side vehicle, and a maximum of two right-side actuators can share one right-side vehicle.

5. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 4, characterized in that, In step S2.3, (1) If N theory ≤N L ×2 and N theory ≤N R ×2, then the initial number of trusses N init =N theory The number of partitions and the number of trusses are determined according to theoretical values, and all actuators are working normally; (2) If N theory >N L ×2 and N theory ≤N R ×2, then the initial number of trusses N init =N theory If the carriers on the left are insufficient, the excess actuators on the left need to be deactivated. The deactivated actuators are moved into the girder storage area, and the girder where they are located is loaded and unloaded by the actuators on the right alone. (3) If N theory ≤N L ×2 and N theory >N R ×2, then the initial number of trusses N init =N theory The number of vehicles on the right is insufficient, so the excess actuators (N) on the right need to be deactivated. theory - N R (×2) The deactivated actuators are moved into the storage areas at both ends of the truss, and the truss on which they are located is loaded and unloaded by the actuator on the left side alone. (4) If N theory >N L ×2 and N theory >N R If the value is ×2, then the number of partitions needs to be reduced, and the required initial number of trusses N needs to be calculated. init =min(N theory -N R ×2, N theory -N L ×2); (5) If the operation mode is unloading, each of the above-mentioned trusses is evenly distributed along the effective coverage area of ​​the goods; if the operation mode is loading, each of the above-mentioned trusses is evenly arranged along the X-axis of the carriage to form N now =N init An initial partition.

6. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 1, characterized in that, Step S3 specifically includes the following steps: Step S3.1, construct the material density function inside the carriage: In each cycle, the 3D global camera acquires the latest 3D point cloud data of the carriage; the Z-axis cross-sectional area of ​​all point clouds at each position is aggregated along the X-axis to construct the material density function p(x,t); the total volume of material on the carriage is calculated according to formula V total(t) Calculate ∫p(x,t)dx; Step S3.2: Divide the material into equal dynamic zones according to the total amount of material, and solve the zone integral equation: The boundary positions of each zone are determined; the zone width is automatically narrowed in areas with dense material and automatically widened in areas with sparse material; the material volume of each zone is... ; Step S3.3, Adjustment of boundary lines for each zone: Calculate the material volume ΔV(t) within the buffer zone of each boundary line, where α is the volume change coefficient, ranging from 0 to 1. Let the position of the kth boundary line be x. k If the buffer band half-width is δ (a preset constant), then That is, the sum of the material volumes within the δ range on both sides of each dividing line; If ΔV(t)≥α·V total(t) / N now Then, proceed to step S3.2 to resolve the boundary equations to translate the boundary, N now Remain unchanged; If ΔV(t) < α·V total(t) / N now If so, then the current boundary remains unchanged; Step S3.4, truss beams decrease successively: set V min Preset unloading and emptying thresholds for individual zones, based on the total volume of multiple bags; V max A loading saturation threshold is preset for a single partition; each cycle iterates through all partitions to determine whether the truss reduction is triggered.

7. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 6, characterized in that, In step S3.4, traverse all partitions according to the following steps: Step ①, Determine the volume threshold for loading / unloading mode partitions: If the operation mode is loading, and there exists a partition j such that V j(t) ≥V max If the candidate partition triggers the reduction of the truss beam, proceed to step ② to perform load verification; if all partitions in this mode do not meet this condition, the reduction will not be triggered in this cycle, and proceed to step S3.

3. If the operation mode is unloading, and there exists a partition j such that V j(t) ≤V min If the candidate partition triggers the reduction of the truss beam, proceed to step ② to perform load verification; if all partitions in this mode do not meet this condition, the reduction will not be triggered in this cycle, and proceed to step S3.

3. Step 2, perform load ratio verification: After the candidate trigger decreases, it is necessary to verify that N now Reduce to N now After -1, check whether the remaining trusses can bear the current remaining total workload; if the check passes, proceed to step ③ to decrease; if the check fails, suspend the decrease in this cycle and wait for more partitions to complete naturally before starting the check again from step ① in the next cycle. Load verification formula for loading mode: V remain_load =Σmax(0,V max -V j(t) ), that is, the sum of the volumes that each of the aforementioned partitions still needs to be filled; Verification condition: V remain_load / (N now -1)≤V fair_load ; Unloading mode load verification formula: V remain_unload =ΣV j(t) That is, the sum of the volumes of all the aforementioned partitions that have not yet been removed; Verification condition: V remain_unload / (N now -1)≤V fair_load V fair_load This is a preset upper limit on the additional workload that a single truss can withstand; Step ③, perform the decrement: reduce N now Updated to N now -1, with the new N now Repeat step S3.2 to solve the partition boundary equations; Step 4, handle the case where multiple zones meet the criteria at the same time and cycle: if multiple zones meet the triggering conditions at the same time in this round, only the reduction of one zone is processed, and the rest are postponed to the next cycle; only one of the trusses is reduced each time; after reduction, the excess trusses are moved into the storage area along the X-axis; Repeat steps S2.4 to S3 until N. now =0, all the aforementioned trusses have been stored.

8. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S4.1, adaptive adjustment of telescopic rod stroke: Each time the actuator moves horizontally, it only needs to keep the end of the gripper at a position higher than the effective obstacle height along the current path; the effective obstacle height changes dynamically with the loading and unloading progress; when the cargo is greatly reduced in the later stage of unloading, the actuator only needs to be retracted to a position higher than the fence to pass safely, without having to return to the highest position when the vehicle is full; Step S4.2, construct a dual-condition effective obstacle height field: the actuator crosses the side fence of the carriage each time it travels between the vehicle and the target point of the carriage; The carriage contains both side railings of varying heights and bagged materials; the height of the side railings is H. wall Located in Y left and Y right At the location, the height H of the bagged material cargo(x,y) The effective obstacle height field is obtained by extracting the 3D point cloud in real time, taking the larger value of the two values, and adding the additional height of the actuator's current state. Step S4.3, construct loading and unloading paths: the path of the actuator when loading and unloading materials each time needs to cross the side fence of the carriage; Loading path: First segment unloaded, the actuator starts from the current position at H... eff_empty(x,y) For height constraints, proceed to the fixed vehicle position P. carrier The first section grabs the material; the second section carries the load, and after grabbing the material, it moves at H... eff_loaded(x,y) For height constraints, from the vehicle position P carrier Directly to the stacking point P inside the carriage place After releasing the material, it becomes unloaded; Unloading path: First segment unloaded, the actuator uses H eff_empty(x,y) For height constraints, from the vehicle position P carrier Head to the grab point P inside the carriage grasp The second stage carries the load and grabs the material at an H-axis speed. eff_loaded(x,y) For height constraints, from P grasp Return to the vehicle position P carrier Release material; Step S4.4, Solving for the shortest path in the peak segment: Sample along the XY path to obtain H at each point. eff The value is used to form a one-dimensional obstacle height profile; all local maxima points in the profile are extracted, and the complete path is divided into several independent segments using these as dividing nodes; each segment uses the maximum obstacle peak height of the segment as the minimum passage constraint, and the three axes move simultaneously to complete the spatial linear motion of the segment. The trajectories of each segment are spliced ​​together to obtain the complete three-dimensional shortest path; the height is raised only at the obstacle peak position, and each segment of the path retains only the minimum necessary passage height. The actuator does not need to be retracted to a fixed height position throughout the entire process.

9. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 6, characterized in that, In step S5, each cycle is executed in the following order of sub-steps: (a) The 3D global camera updates the 3D point cloud in real time, constructs the material density function p(x,t), and calculates V by integration. total With each partition V j ; (b) Perform the two-level partition control in steps S3.3 and S3.4; (c) Construct corresponding H according to the working conditions of each section eff Execute step S4 to plan the 3D path; (d) Issue X-axis commands for the truss and Y-axis and Z-axis commands for the actuators; (e) The local 3D auxiliary camera monitors the material positioning in real time, and the actuator completes precise gripping and placement; the idle truss is stored in the storage area set at both ends of the frame X-axis, and the idle actuator is stored in the storage area set at both ends of the truss; repeat the above sub-steps until N. now =0, all trusses have been stored, and the operation is complete.

10. The adaptive collaborative loading and unloading method based on multimodal visual perception according to claim 1, characterized in that, Each actuator on the left and right sides of each truss is attached to a carrier, and each carrier can be shared by a maximum of two actuators. The two actuators sharing the same carrier use the carrier alternately.

Images