A method, device, and equipment for detecting obstacles above goods, and a storage medium

Abstract

The application discloses a kind of goods upper obstacle detection method, device, equipment and storage medium, the method comprises: by being fixed to downward depth camera and lateral auxiliary sensor of mechanical arm tail end, the fusion point cloud data of three-dimensional space above goods is collected;From the fusion point cloud, the upper detection area above the upper surface of goods is segmented;The point cloud in the upper detection area is clustered and feature is extracted to identify and track obstacle;The path point of the mechanical arm tail end descending path is discretized, the nearest distance of each path point and the bounding box of obstacle is calculated, and collision detection is carried out;And according to the collision detection result, output hierarchical early warning.The application covers the detection blind area of goods directly above, through real-time collision verification and dynamic obstacle tracking of path level, significantly reduces the false detection rate and collision risk, and improves the safety of robot loading and unloading operation.

Description

Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to a method, apparatus, equipment, and storage medium for detecting obstacles above goods. Background Technology

[0002] When industrial robots perform tasks such as grasping, loading, unloading, and palletizing, obstacle detection in the space above the end effector is a core element in ensuring its safe operation. Taking grasping operations as an example, the robotic arm's end effector needs to approach the goods from directly above. This path may contain obstacles such as shelf beams, suspension cables, and extended structures of adjacent equipment. If the detection fails, it can easily lead to collisions, resulting in equipment damage or even safety accidents.

[0003] Currently, relevant detection technologies mainly rely on two types of solutions: LiDAR and depth cameras. LiDAR solutions, as described in the paper "Research on Obstacle Detection for AGV Movement Path Avoidance Based on Multi-Data Fusion," construct an environmental point cloud map through laser scanning. However, its scanning plane is relatively fixed, resulting in a large blind spot in the vertical space directly above the goods. Single-line LiDAR also struggles to effectively perceive three-dimensional obstacles. Depth camera solutions, as mentioned in the paper "AGV Visual Navigation System Based on DAMOYOLO-S—Real-time Obstacle Detection," use lightweight models for real-time image detection. However, these methods primarily target obstacles on the same plane on the ground, and due to limitations in camera perspective, they cannot fully cover the three-dimensional space above the end effector.

[0004] In summary, existing technologies have the following main shortcomings: First, the blind spot in the overhead space is large; current solutions mostly use horizontal scanning, lacking effective coverage of the vertical area directly above the goods. Second, the end-effector movement path is not specifically modeled, failing to distinguish between the robotic arm's descent path and horizontal movement path, and lacking targeted analysis of obstacle risks encountered during vertical movement in grasping operations. Third, the response to dynamic obstacles is lagging, making it difficult to achieve real-time tracking and motion prediction for dynamic targets such as suspended cables and swing beams. Fourth, the false detection rate is high; complex lighting conditions and reflective materials easily introduce sensor noise, leading to frequent false alarms. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing technology has problems such as large blind spots in the detection of the upper space, inaccurate assessment of collision risk of the vertical movement path of the end effector, slow response to dynamic obstacles, and high false detection rate.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a method for detecting obstacles above cargo, the method comprising: Spatial point cloud data of the target area is collected by a downward depth camera fixed to the end of the robotic arm and a lateral auxiliary sensor fixed to the forearm of the robotic arm, and the spatial point cloud data is fused into unified fused point cloud data. The inspection area above the upper surface of the cargo is segmented from the fused point cloud data; Clustering and feature extraction are performed on the point cloud within the upper inspection area to identify and track at least one obstacle; The descent path of the robotic arm end effector is discretized into path points, and the nearest distance between each path point and the bounding box of each obstacle is calculated for collision detection. Based on the collision detection results, a graded warning signal is output.

[0007] As an optional implementation, in the first aspect of the invention, the optical axis of the downward depth camera is perpendicular to the plane of the end flange of the robotic arm and points downward; the optical axis of the lateral auxiliary sensor points to the side and downward, forming a preset downward angle with the horizontal plane.

[0008] As an optional implementation, in the first aspect of the present invention, the step of fusing the spatial point cloud data into unified fused point cloud data includes: Based on the offline calibrated extrinsic transformation matrix and the real-time acquired robotic arm pose, the first point cloud data collected by the lateral auxiliary sensor is transformed into the camera coordinate system of the current depth camera and fused with the second point cloud data collected by the depth camera.

[0009] As an optional implementation, in a first aspect of the invention, segmenting the upper inspection area above the upper surface of the cargo from the fused point cloud data includes: The Random Sample Consensus (RANSAC) algorithm is used to perform plane fitting on the point cloud corresponding to the upper surface of the cargo in the fused point cloud data to obtain the reference plane. Using the reference plane as a reference, the spatial region that is vertically upward and more than a preset first safety distance threshold from the reference plane is divided into the upper inspection area.

[0010] As an optional implementation, in the first aspect of the invention, the step of clustering and feature extraction of the point cloud within the upper inspection area to identify and track at least one obstacle includes: The density-based noise spatial clustering (DBSCAN) algorithm is used to cluster the point cloud in the upper inspection area to obtain at least one independent obstacle point cloud cluster. Extract the geometric features of each obstacle point cloud cluster, including three-dimensional dimensions, principal axis orientation, linearity, and mean curvature; Based on the geometric features, the obstacle point cloud clusters are classified into beams, cables, or irregular obstacles.

[0011] As an optional implementation, in the first aspect of the invention, the step of clustering and feature extraction of the point cloud within the upper inspection area to identify and track at least one obstacle further includes: A Kalman filter is used to estimate the motion state and predict the trajectory of the classified obstacles in order to track dynamic obstacles.

[0012] As an optional implementation, in a first aspect of the invention, discretizing the descent path of the robotic arm end effector into path points and calculating the nearest distance between each path point and the bounding box of each obstacle for collision detection includes: Generate an axially aligned bounding box for each identified and tracked obstacle; The descent path from the safe height above the grab point to the target grab point is discretized into a series of dense path points; Calculate the Euclidean distance from each of the path points to each of the bounding boxes, and use this distance as the nearest distance; When the nearest distance to any of the path points is less than a preset second safety distance threshold, it is determined that the descent path has collided with the corresponding obstacle.

[0013] As an optional implementation, in the first aspect of the present invention, the step of outputting a graded warning signal based on the collision detection result includes: The warning level is determined based on the minimum collision distance and the dynamic properties of the obstacle; The output includes the warning signal containing the warning level, obstacle identification, type, location, and speed; Specifically, when the minimum collision distance is less than twice the second safety distance threshold but greater than the second safety distance threshold, and the obstacle is static, the warning level is determined to be Level 1; when the minimum collision distance is less than the second safety distance threshold, or when the minimum collision distance is less than twice the second safety distance threshold but greater than the second safety distance threshold and the obstacle is approaching, the warning level is determined to be Level 2; when the minimum collision distance is less than half of the second safety distance threshold, or when the intrusion speed of the obstacle exceeds a preset threshold, the warning level is determined to be Level 3.

[0014] A second aspect of the present invention discloses a device for detecting obstacles above cargo, the device comprising: The point cloud acquisition and fusion module is used to acquire spatial point cloud data of the target area through a downward depth camera fixed to the end of the robotic arm and a lateral auxiliary sensor fixed to the forearm of the robotic arm, and to fuse the spatial point cloud data into unified fused point cloud data. The spatial segmentation module is used to segment the upper inspection area above the upper surface of the cargo from the fused point cloud data; An obstacle recognition and tracking module is used to cluster and extract features from the point cloud in the upper inspection area to identify and track at least one obstacle. The descent path collision detection module is used to discretize the descent path of the robotic arm end effector into path points, and calculate the nearest distance between each path point and the bounding box of each obstacle for collision detection. The obstacle warning output module is used to output graded warning signals based on the collision detection results.

[0015] A third aspect of the present invention discloses a robotic device, the robotic device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the obstacle detection method above cargo disclosed in the first aspect of the present invention.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements some or all of the steps in the obstacle detection method above cargo disclosed in the first aspect of the present invention.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, a downward depth camera and a lateral auxiliary sensor fixed to the end of the robotic arm specifically cover the detection blind spot directly above the cargo, achieving full coverage and high-precision perception of the three-dimensional space of the end effector's descent path. By performing dynamic collision detection between the planned path and the obstacle bounding box, accurate collision risk assessment at the path level is achieved, effectively avoiding false alarms and missed alarms. By introducing a Kalman filter to estimate the motion state and predict the trajectory of obstacles, dynamic obstacles such as swinging cables can be tracked in real time, providing early warnings. By outputting graded warnings based on the collision detection results and triggering path corrections or emergency shutdowns accordingly, the safety and automation of the robot's loading and unloading operations are significantly improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for detecting obstacles above cargo disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a cargo overhead obstacle detection device disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a robot device disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer storage medium disclosed in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention discloses a method, apparatus, device, and storage medium for detecting obstacles above cargo. It uses a downward depth camera and lateral auxiliary sensors fixed to the end effector of a robotic arm to acquire fused point cloud data of the three-dimensional space above the cargo. The method then segments the area to be inspected above the cargo's upper surface from the fused point cloud. The point cloud within this area is clustered and its features extracted to identify and track obstacles. The descent path of the robotic arm's end effector is discretized into path points, and the shortest distance between each path point and the bounding box of the obstacle is calculated for collision detection. A tiered warning is output based on the collision detection results. This invention specifically covers the detection blind spot directly above the cargo. Through path-level real-time collision verification and dynamic obstacle tracking, it significantly reduces the false detection rate and collision risk, improving the safety of robotic loading and unloading operations. These are described in detail below.

[0024] Example 1 Please see Figure 1 , Figure 1 This is a schematic flowchart of a method for detecting obstacles above cargo disclosed in an embodiment of the present invention. Figure 1 As shown, the obstacle detection method above the cargo may include the following operations: 101. Spatial point cloud data of the target area is collected by a downward depth camera fixed to the end of the robotic arm and a lateral auxiliary sensor fixed to the forearm of the robotic arm, and the spatial point cloud data is fused into unified fused point cloud data.

[0025] Optionally, the depth camera is fixed above the end flange of the robotic arm by a rigid mounting plate, with its optical axis strictly perpendicular to the flange plane and pointing downwards. The optical center of the camera is positioned at a height of 300-500 mm above the end effector's gripping plane. The lateral auxiliary sensor is fixed to the side of the robotic arm's forearm by a bracket, with its optical axis pointing slightly downwards at a 15°-30° angle to the horizontal plane, used to cover the blind spot in front of the depth camera.

[0026] Optionally, a hardware-software integrated spatiotemporal synchronization fusion scheme is adopted: time synchronization adopts hardware synchronization based on encoder triggering, with the robot arm motion encoder pulse locking the downward camera acquisition time, and lateral sensor data being aligned through timestamp interpolation, completely eliminating the time difference of multi-sensor acquisition; spatial coordinate unification is achieved by converting all point cloud data to the camera coordinate system of the downward depth camera at the current moment through offline calibrated extrinsic parameter transformation matrix and real-time acquired robot arm pose, while a dual denoising algorithm combining direct-pass filtering and statistical filtering is adopted to remove discrete noise points generated by industrial scene illumination, cargo reflection, and dust, filtering invalid point clouds, and finally forming high-precision, low-noise, and time-aligned fused point cloud data, solving the problems of point cloud misalignment and large noise interference in traditional fusion methods.

[0027] Specifically, the core principle of point cloud coordinate unification is a quadratic coordinate transformation, which calibrates all sensor point cloud data to the current camera coordinate system, achieving accurate fusion of multi-source point cloud data. The entire transformation process relies on the extrinsic parameter transformation matrix obtained from offline calibration, and the specific transformation procedure is as follows: First, using the robot's base coordinate system as the intermediate reference, the raw point cloud data P_sensor collected by the lateral sensor is transformed from the sensor coordinate system to the robot's base coordinate system through the extrinsic transformation matrix T_sensor_base of the sensor relative to the base. To achieve fusion and matching between this point cloud and the point cloud data from the downward camera, a secondary coordinate transformation is also required on the point cloud in the base coordinate system, transforming it to the downward camera coordinate system.

[0028] The entire process combines the real-time pose matrix T_current of the robotic arm to complete dynamic correction, ultimately unifying all point cloud data into the downward camera coordinate system at the current moment, resulting in fused target point cloud data. The complete fusion transformation formula is: P_fused = T_current⁻¹ × T_base_camera × T_sensor_base × P_sensor. The roles and definitions of each parameter in the formula are as follows: T_current⁻¹: The inverse matrix of the real-time pose of the robotic arm, its core function is to compensate for the coordinate system offset error generated during the movement of the robotic arm and eliminate the influence of the dynamic posture change of the robotic arm on the coordinate accuracy; T_base_camera: The fixed transformation matrix from the robot base coordinate system to the downward camera coordinate system, obtained from offline calibration, and belongs to the inherent fixed parameters of the device; P_fused: The fused point cloud data in the downward camera coordinate system obtained after two-layer coordinate transformation and dynamic compensation, which is the effective output data after coordinate unification.

[0029] 102. Segment the upper inspection area above the upper surface of the cargo from the fused point cloud data.

[0030] Optionally, the Random Sample Consensus (RANSAC) algorithm is used to robustly fit the point cloud corresponding to the upper surface of the cargo in the fused point cloud data, obtaining a reference plane and a set of internal points. A local coordinate system is established using this reference plane as the Z=0 plane, with its normal vector pointing in the positive Z-axis direction. Then, based on the real-time identified 3D dimensions and stacking height of the cargo, the first safety distance threshold is adaptively and dynamically adjusted. For large-volume, high-stack cargo, the threshold is automatically increased to expand the detection range above; for small, flat cargo, the threshold is automatically decreased to eliminate invalid blank areas above. The spatial region vertically upwards and exceeding the dynamically adjusted first safety distance threshold from the reference plane is designated as the upper inspection area. Subsequent obstacle detection is performed only on the point cloud within this area, achieving accurate adaptation between the detection area and the operational scenario.

[0031] 103. Cluster and extract features from the point cloud in the upper inspection area to identify and track at least one obstacle.

[0032] Optionally, firstly, the density-based noise applied spatial clustering (DBSCAN) algorithm is used to cluster the point cloud in the upper inspection area, effectively separating each independent obstacle point cloud cluster.

[0033] Next, geometric features such as three-dimensional dimensions, principal axis direction, point cloud distribution (linearity) along the principal axis, and average curvature are extracted for each point cloud cluster. An innovative curvature abrupt change weight coefficient is added to differentiate between rigid and flexible obstacles: if one dimension is significantly larger than the other two, and the linearity is high, curvature is low, and there are no curvature abrupt changes, it is identified as a rigid beam; if it has high linearity, large curvature fluctuations, and frequent curvature abrupt changes, it is identified as a flexible suspension cable; those that do not meet the above characteristics are classified as irregular obstacles, solving the problem that traditional algorithms cannot accurately distinguish between rigid and flexible obstacles and that flexible obstacles are often missed in identification.

[0034] Finally, for dynamic obstacles, a Kalman filter with differentiated parameter configuration is used for tracking and prediction. For rigid beam obstacles, low-gain, low-noise steady-state filter parameters are configured to ensure tracking stability for static, low-speed moving obstacles; for flexible dynamic obstacles such as swinging cables, adaptive dynamic gain parameters are configured to adapt to their irregular swinging motion characteristics in real time. The filter's state variables include the obstacle's three-dimensional position and velocity. In each frame, the current state of the obstacle is first predicted through a state transition equation, and then updated using the actual observations of the current frame to obtain the posterior optimal estimate. Through this recursive process, the filter can continuously track obstacles and predict their short-term trajectories, significantly improving the tracking accuracy of dynamic obstacles.

[0035] Specifically, for dynamic obstacles in the scene, the system uses a Kalman filter to complete real-time tracking and motion state prediction. The filter uses the complete motion information of the obstacle as the state variable x, with the standard definition formula: x = [px, py, pz, vx, vy, vz]^T. This state variable covers the obstacle's three-dimensional spatial position parameters and three-dimensional motion velocity parameters, which can comprehensively characterize the obstacle's real-time motion state.

[0036] The Kalman filter's operating logic mainly includes two core iterative steps: prediction and update, to accurately solve for the obstacle state frame by frame. The first step is state prediction. When processing each frame of data, the system bases its prediction on the optimal obstacle state output from the previous time step, and then uses the state transition equation... The prior estimate of the current state is calculated to predict the current position and initial velocity of the obstacle.

[0037] The second step is state update. After prediction, the system incorporates the actual observation value z of the current frame. This observation value can be calculated through point cloud frame nearest neighbor matching and obstacle centroid association algorithms, accurately reflecting the true position and displacement information of obstacles. Using the Kalman gain algorithm, the predicted state value and the actual observation value are optimally fused and corrected, and finally, the posterior optimal estimate x of the current state is obtained. k | k And the corresponding covariance matrix, to complete the single-frame state update.

[0038] The system continuously and stably tracks the motion state of dynamic obstacles through a frame-by-frame recursive iterative prediction and update process. Simultaneously, the prior result x output by the filter prediction stage... k ₊1| k It can be used to predict the short-term movement trajectory of obstacles, thereby determining whether the movement path of the obstacle will collide with the robot's preset grasping operation path, providing data support for robot obstacle avoidance and path planning.

[0039] 104. Discretize the descent path of the robotic arm end effector into path points, and calculate the nearest distance between each path point and the bounding box of each obstacle for collision detection.

[0040] Optionally, an axially aligned bounding box is first generated for each identified and tracked obstacle, the boundaries of which are determined by the minimum and maximum values ​​of the cluster point cloud on each coordinate axis.

[0041] Then, the continuous descent path of the robotic arm's end effector from a safe height directly above the gripping point to the target gripping point is discretized into a series of dense path points, i.e., the position of the origin of the end-effector coordinate system. This step innovatively introduces a speed-adaptive discretization strategy: when the robotic arm descends at high speed, the sampling density of path points is increased to ensure no detection blind spots under high-speed motion; when the robotic arm is precisely aligned at low speed, the path points are appropriately sparsed to reduce computational overhead, balancing detection accuracy and real-time computational efficiency, and adapting to the robot's full-condition operation requirements.

[0042] Next, the Euclidean distance from each path point to the bounding box of each obstacle is calculated. If there exists any path point whose distance to the bounding box of an obstacle is less than a preset second safety distance threshold (dsafe, usually the radius of the end effector plus a safety margin), then it is determined that the descent path has collided with the obstacle, i.e., path interference exists.

[0043] 105. Based on the collision detection results, output a graded early warning signal.

[0044] Optionally, the system classifies warning levels based on the minimum collision distance dmin and the dynamic properties of the obstacle: Level I Attention: Triggered when dsafe < dmin < 2 * dsafe and the obstacle is static, only reporting information.

[0045] Level II Warning: Triggered when dmin < dsafe or when the obstacle is approaching at Level I distance, the drive system performs path correction, such as vertically raising or lowering the starting point or horizontally offsetting the grasping point.

[0046] Level III Danger: Triggered when dmin < 0.5 * dsafe or when the obstacle intrudes rapidly, immediately pausing the operation and performing an emergency stop.

[0047] The early warning module outputs a signal containing the obstacle ID, type, estimated position, speed, and specific early warning level to the upper-level system to complete the closed-loop from perception to safe execution.

[0048] It can be seen that implementing Figure 1 the above-described method for detecting obstacles above goods, the special sensor fixed downward at the end covers the detection blind area of the traditional solution, achieving precise perception of the three-dimensional space above the goods. By performing dynamic collision detection between the planned path and the obstacle bounding box, accurate risk assessment at the path level is achieved, effectively avoiding false alarms and missed detections. By using a Kalman filter to track and predict the trajectory of dynamic obstacles, it can respond in real time to complex scenarios such as swinging cables. Finally, through hierarchical early warning and flexible response strategies, the safety, reliability, and automation level of robot loading, unloading, and palletizing operations are significantly improved.

[0049] Embodiment 2 Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of an obstacle detection device above goods disclosed in an embodiment of the present invention. As Figure 2 shown, the obstacle detection device above goods may include: A point cloud acquisition and fusion module 201, configured to collect spatial point cloud data of a target area through a downward depth camera fixed at the end of the robotic arm and a lateral auxiliary sensor fixed on the forearm of the robotic arm, and fuse the spatial point cloud data into unified fused point cloud data; A space segmentation module 202, configured to segment an above待检区 (to-be-inspected area above) above the upper surface of the goods from the fused point cloud data; An obstacle recognition and tracking module 203, configured to perform clustering and feature extraction on the point cloud in the above待检区 (to-be-inspected area above) to identify and track at least one obstacle; A descent path collision detection module 204, configured to discretize the descent path of the end actuator of the robotic arm into path points, calculate the shortest distance between each path point and the bounding box of each obstacle for collision detection; The obstacle warning output module 205 is used to output graded warning signals based on the collision detection results.

[0050] For a detailed description of the above-mentioned obstacle detection device for cargo, please refer to the detailed description of the obstacle detection method for cargo in Embodiment 1 above, which will not be repeated here.

[0051] Example 3 Please see Figure 3 , Figure 3 This is a structural schematic diagram of a robot device disclosed in an embodiment of the present invention. Figure 3 As shown, the robot device may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the obstacle detection method above cargo disclosed in Embodiment 1 of the present invention.

[0052] Example 4 This invention discloses a computer-readable storage medium storing computer instructions, which, when invoked, are used to execute steps in the obstacle detection method above cargo disclosed in Embodiment 1 of this invention.

[0053] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0054] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0055] Finally, it should be noted that the method, apparatus, device, and storage medium for detecting obstacles above cargo disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting obstacles above cargo, characterized in that, The method includes: Spatial point cloud data of the target area is collected by a downward depth camera fixed to the end of the robotic arm and a lateral auxiliary sensor fixed to the forearm of the robotic arm, and the spatial point cloud data is fused into unified fused point cloud data. The inspection area above the upper surface of the cargo is segmented from the fused point cloud data; Clustering and feature extraction are performed on the point cloud within the upper inspection area to identify and track at least one obstacle; The descent path of the robotic arm end effector is discretized into path points, and the nearest distance between each path point and the bounding box of each obstacle is calculated for collision detection. Based on the collision detection results, a graded warning signal is output.

2. The method according to claim 1, characterized in that, The optical axis of the downward depth camera is perpendicular to the plane of the flange at the end of the robotic arm and points downward; the optical axis of the lateral auxiliary sensor points to the side and downward, at a preset downward angle to the horizontal plane.

3. The method according to claim 2, characterized in that, The process of fusing the spatial point cloud data into unified fused point cloud data includes: Based on the offline calibrated extrinsic transformation matrix and the real-time acquired robotic arm pose, the first point cloud data collected by the lateral auxiliary sensor is transformed into the camera coordinate system of the current depth camera and fused with the second point cloud data collected by the depth camera.

4. The method according to claim 3, characterized in that, The step of segmenting the upper inspection area above the upper surface of the cargo from the fused point cloud data includes: The Random Sample Consensus (RANSAC) algorithm is used to perform plane fitting on the point cloud corresponding to the upper surface of the cargo in the fused point cloud data to obtain the reference plane. Using the reference plane as a reference, the spatial region that is vertically upward and more than a preset first safety distance threshold from the reference plane is divided into the upper inspection area.

5. The method according to claim 4, characterized in that, The step of clustering and feature extraction of the point cloud within the upper inspection area to identify and track at least one obstacle includes: The density-based noise spatial clustering (DBSCAN) algorithm is used to cluster the point cloud in the upper inspection area to obtain at least one independent obstacle point cloud cluster. Extract the geometric features of each obstacle point cloud cluster, including three-dimensional dimensions, principal axis orientation, linearity, and mean curvature; Based on the geometric features, the obstacle point cloud clusters are classified into beams, cables, or irregular obstacles.

6. The method according to claim 5, characterized in that, The step of clustering and extracting features from the point cloud within the upper inspection area to identify and track at least one obstacle further includes: A Kalman filter is used to estimate the motion state and predict the trajectory of the classified obstacles in order to track dynamic obstacles.

7. The method according to claim 6, characterized in that, The step of discretizing the descent path of the robotic arm's end effector into path points and calculating the nearest distance between each path point and the bounding box of each obstacle for collision detection includes: Generate an axially aligned bounding box for each identified and tracked obstacle; The descent path from the safe height above the grab point to the target grab point is discretized into a series of dense path points; Calculate the Euclidean distance from each of the path points to each of the bounding boxes, and use this distance as the nearest distance; When the nearest distance to any of the path points is less than a preset second safety distance threshold, it is determined that the descent path has collided with the corresponding obstacle.

8. A device for detecting obstacles above cargo, characterized in that, The device includes: The point cloud acquisition and fusion module is used to acquire spatial point cloud data of the target area through a downward depth camera fixed to the end of the robotic arm and a lateral auxiliary sensor fixed to the forearm of the robotic arm, and to fuse the spatial point cloud data into unified fused point cloud data. The spatial segmentation module is used to segment the upper inspection area above the upper surface of the cargo from the fused point cloud data; An obstacle recognition and tracking module is used to cluster and extract features from the point cloud in the upper inspection area to identify and track at least one obstacle. The descent path collision detection module is used to discretize the descent path of the robotic arm end effector into path points, and calculate the nearest distance between each path point and the bounding box of each obstacle for collision detection. The obstacle warning output module is used to output graded warning signals based on the collision detection results.

9. A robotic device, characterized in that, The robotic device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the obstacle detection method above cargo as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the obstacle detection method above cargo as described in any one of claims 1-7.

Images