A multi-modal data fusion-based collision avoidance method and system for a stacker-reclaimer

Abstract

The application discloses a kind of based on multimodal data fusion's stacker-reclaimer anti-collision method and system, it is related to industrial equipment anti-collision technical field, including real-time acquisition target stacker-reclaimer and the laser radar point cloud of working area equipment, ultra-wideband positioning coordinates, inertial measurement unit attitude angle, high-definition video stream and control instruction;To multimodal data executes time stamp alignment and space coordinate system, obtains space-time synchronous perception data;Through feature extraction fusion obtains stacker-reclaimer pose, obstacle profile and trajectory;With pose and control instruction, predict future time window motion intention and reachable space;Reachable space and obstacle trajectory dynamic collision detection, calculate risk point, collision time and safety margin;Generation hierarchical anti-collision strategy.The application is through multimodal space-time synchronous fusion and the dynamic detection of motion intention prediction, improves the accuracy and foresight of anti-collision under complex environment.

Description

Technical Field

[0001] This invention belongs to the field of industrial equipment anti-collision technology, specifically a stacker-reclaimer anti-collision method and system based on multimodal data fusion. Background Technology

[0002] Stacker-reclaimers (CRRs) need to coordinate with surrounding equipment during operations in material yards, and collision avoidance is crucial for ensuring safe operation. Existing technologies often use a single sensor to collect environmental data or simply overlay raw data from multiple sensors for collision detection. Single-sensor solutions are susceptible to dust obstruction and lighting variations, making it difficult to reliably acquire the stacker-reclaimer's own position and the complete outline of obstacles. Overlaying multiple sensors fails to address data timing discrepancies and coordinate system differences, leading to conflicting perception information and insufficient accuracy after fusion. Current collision detection methods primarily rely on spatial overlap judgment between the stacker-reclaimer's current position and the static outline of obstacles, neglecting the influence of equipment movement trends and control commands. This lack of prediction of the future trajectory of dynamic obstacles makes it prone to missing potential risks during equipment acceleration and turning, resulting in warnings lagging behind actual hazards.

[0003] It is necessary to overcome the bottleneck of effectively integrating multi-source heterogeneous data, and solve the inconsistencies in time synchronization and spatial mapping of data such as LiDAR point clouds, ultra-wideband positioning, inertial attitude data, and video images, to form a unified and reliable perception foundation. Simultaneously, it is necessary to change the static collision detection logic, combining it with real-time control commands to predict the future movement range of the device, incorporating dynamic obstacle trajectories into risk assessment, and achieving proactive identification of potential collisions. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art; Therefore, this invention proposes a collision avoidance method for stacker-reclaimers based on multimodal data fusion, comprising: Real-time acquisition of operating status data of the target stacker-reclaimer and related equipment within the working area. The operating status data includes lidar scan point cloud, ultra-wideband positioning coordinates, inertial measurement unit attitude angle, high-definition video image stream, and real-time control commands of the equipment control system. The lidar scan point cloud, the ultra-wideband positioning coordinates, the inertial measurement unit attitude angle, and the high-definition video image stream are time-stamped and aligned with the spatial coordinate system to obtain spatiotemporally synchronized multimodal sensing data. Multimodal feature extraction and fusion operations are performed on the spatiotemporally synchronized multimodal sensing data to obtain the pose state information of the target stacker-reclaimer itself and the sensing contours and motion trajectories of dynamic obstacles in the working area; Based on the target stacker-reclaimer's own pose information and the real-time control commands, the target stacker-reclaimer's movement intention and reachable movement space within a preset time window in the future are predicted; The reachable motion space of the target stacker-reclaimer is dynamically collided with the perceived contour and motion trajectory of the dynamic obstacle. If a potential collision risk is detected, the collision risk point, the estimated collision time, and the minimum safety margin required for avoidance are calculated. Based on the collision risk points, the expected collision time, and the minimum safety margin, a hierarchical collision avoidance control strategy is generated.

[0005] Furthermore, the lidar scan point cloud, the ultra-wideband positioning coordinates, the inertial measurement unit attitude angles, and the high-definition video image stream are time-stamped and aligned with the spatial coordinate system to obtain spatiotemporally synchronized multimodal sensing data, including: A unified, high-precision timestamp is applied to each frame of lidar scan point cloud, each set of ultra-wideband positioning coordinates, each set of inertial measurement unit attitude angles, and each frame of high-definition video image stream. A global workspace coordinate system is established with a preset reference point on the target stacker-reclaimer body as the origin; Transform the laser radar scan point cloud from the radar's own coordinate system to the global workspace coordinate system; Transform the ultra-wideband positioning coordinates from the positioning base station coordinate system to the global workspace coordinate system; The attitude angle of the inertial measurement unit is used to perform attitude correction on the high-definition video image stream to eliminate image jitter caused by body shaking, and the corrected image pixel coordinates are mapped to a two-dimensional plane in the global workspace coordinate system; Ensure that all modal data are expressed in the global workspace coordinate system under the same high-precision timestamp, thus achieving spatiotemporal synchronization.

[0006] Furthermore, multimodal feature extraction and fusion operations are performed on the spatiotemporally synchronized multimodal sensing data to obtain the pose state information of the target stacker-reclaimer itself and the perceived contours and motion trajectories of dynamic obstacles within the working area, including: From the spatiotemporally synchronized multimodal sensing data, the geometric structure features of the lidar point cloud, the position stability features of the ultra-wideband coordinates, the attitude change rate features of the inertial measurement unit, and the visual semantic features of the high-definition video image are extracted. An attention-based feature fusion network is used to weight and fuse the geometric structural features, the positional stability features, the pose change rate features, and the visual semantic features to generate a unified environmental feature representation. Based on the unified environmental feature representation, the point cloud clusters and image regions of the target stacker-reclaimer are identified through an instance segmentation network. Combined with the ultra-wideband coordinates and the attitude angle of the inertial measurement unit, the three-dimensional position, orientation, boom pitch angle and rotation angle of the target stacker-reclaimer are calculated as the pose state information. Simultaneously, based on the unified environmental feature representation, non-self moving point cloud clusters and image regions within the working area are identified and tracked, classified as dynamic obstacles, and the three-dimensional contour, real-time position, and velocity vector of each dynamic obstacle are estimated to form the perception contour and motion trajectory.

[0007] Furthermore, based on the target stacker-reclaimer's own pose information and the real-time control commands, the movement intention and reachable movement space of the target stacker-reclaimer within a preset time window are predicted, including: The real-time control commands are analyzed to obtain the target travel target of the target stacker-reclaimer, the target angle of the slewing mechanism, the target angle of the pitching mechanism, and the expected action of the reclaiming mechanism. Based on the current state in the pose state information and combined with the kinematic model of the stacker-reclaimer, the forward simulation calculates the ideal motion path of the target stacker-reclaimer under the drive of the real-time control command from the current moment to the end of the future preset time window. Considering the response delay, acceleration, and speed limits of each moving joint of the target stacker-reclaimer, a motion uncertainty envelope is extended based on the ideal motion path. The space within the motion uncertainty envelope is the reachable motion space. The reachable motion space is a dynamic three-dimensional space that changes over time.

[0008] Further, dynamic collision detection is performed between the reachable motion space of the target stacker-reclaimer and the perceived contour and motion trajectory of the dynamic obstacle, including: Within each fusion perception cycle, the reachable motion space of the target stacker-reclaimer at the current moment, as well as the latest perception contours and motion trajectories of all dynamic obstacles, are acquired. The motion trajectory of the dynamic obstacle is linearly extrapolated to the future preset time window; Calculate the spatial interference between the three-dimensional volume sequence of the reachable motion space evolving over time and the three-dimensional volume sequence of the perceived contour of each dynamic obstacle evolving over time based on linear extrapolation; If spatial interference exists, it is determined that there is a potential collision risk, and the starting time of the interference is recorded as the expected collision time, and the spatial location where the interference occurs is recorded as the collision risk point.

[0009] Furthermore, if a potential collision risk is detected, the collision risk point, the expected collision time, and the minimum safety margin required for avoidance are calculated, including: After detecting spatial interference, the relative motion direction and velocity of the target stacker-reclaimer and the dynamic obstacle that is interfering with it are analyzed before the collision occurs. Construct a virtual repulsive force field centered on the collision risk point, and calculate the minimum position adjustment required to separate the edge of the reachable motion space of the target stacker-reclaimer from the edge of the perceived contour of the dynamic obstacle before the expected collision time; The minimum position adjustment amount includes at least one of the following: the amount of trolley displacement, the amount of slewing angle adjustment, or the amount of pitch angle adjustment that the target stacker-reclaimer needs to change. The minimum position adjustment amount is converted into a buffer in the time dimension, which is the minimum safety margin.

[0010] Furthermore, based on the collision risk points, the expected collision time, and the minimum safety margin, a hierarchical collision avoidance control strategy is generated, including: The collision avoidance control strategy includes a deceleration strategy, a pause strategy, and a local path replanning strategy. Based on the expected collision time, different response levels are set according to the urgency level; the shorter the expected collision time, the higher the response level. When the expected collision time is greater than a first threshold, the deceleration strategy is generated, the deceleration strategy including instructions to decelerate one or more moving joints of the target stacker-reclaimer; When the expected collision time is less than or equal to the first threshold but greater than the second threshold, the pause strategy is generated. The pause strategy includes instructions to smoothly stop all moving joints of the target stacker-reclaimer within a preset time. When the expected collision time is less than or equal to the second threshold, the local path replanning strategy is generated. The local path replanning strategy calculates a temporary collision-free path that can avoid the collision risk point based on the minimum safety margin, and generates a sequence of instructions to guide the target stacker-reclaimer to move along the temporary collision-free path.

[0011] Furthermore, it also includes: The anti-collision control strategy is converted into a specific motion control command sequence, and the motion control command sequence is sent to the equipment control system of the target stacker-reclaimer; During the execution of the motion control command sequence by the target stacker-reclaimer, the sensing contour and motion trajectory of the dynamic obstacle are continuously monitored for abrupt changes. If the perceived contour and motion trajectory of the dynamic obstacle change abruptly, the online replanning of the collision avoidance control strategy is triggered, and the motion control command sequence is updated. Record the entire process of each collision risk event using multimodal perception data, the generated collision avoidance control strategy and its execution results, and build a collision avoidance case library for subsequent optimization of strategy generation models; The step of converting the collision avoidance control strategy into a specific motion control command sequence includes: The abstract control objective contained in the deceleration strategy, the pause strategy, or the local path replanning strategy is decomposed into specific set values ​​of each moving joint of the target stacker-reclaimer in multiple discrete control cycles. The set values ​​include the target speed of the trolley travel motor, the target angle of the rotary motor, and the target stroke of the pitch hydraulic cylinder. According to the sequence of control cycles, the set values ​​of all moving joints are arranged into a time-synchronized instruction set to form the motion control instruction sequence. If the perceived contour and trajectory of the dynamic obstacle change abruptly, the online replanning of the collision avoidance control strategy is triggered, including: During the monitoring process, the changes in position, velocity, and contour of the dynamic obstacle between adjacent sensing cycles are calculated. If any of the position change, velocity change, or contour change exceeds its corresponding preset mutation threshold, it is determined that the perceived contour and motion trajectory of the dynamic obstacle have undergone a sudden change. Immediately interrupt the currently executing sequence of motion control commands; Based on the latest perceived contour and motion trajectory after the dynamic obstacle mutation, the dynamic collision detection and anti-collision control strategy generation steps are re-executed to obtain the updated anti-collision control strategy and the corresponding motion control command sequence.

[0012] Furthermore, the construction of the anti-collision case library for subsequent optimization strategy generation model includes: Create a case record for each complete collision avoidance intervention process; The case record includes: a snapshot of the original multimodal perception data when a collision risk is triggered, the calculated reachable motion space and dynamic obstacle information, the complete parameters of the generated collision avoidance control strategy, the actual sequence of motion control commands issued, and the success or failure indicator of the actual collision avoidance result after execution. Periodically extract case data from the collision avoidance case library to optimize and train the parameters of the strategy model used to generate the hierarchical collision avoidance control strategy; The goal of the optimization training is to make the collision avoidance control strategy generated by the strategy model require a smaller minimum safety margin or result in a lower loss of production efficiency under similar risk scenarios.

[0013] Furthermore, the present invention also includes a stacker-reclaimer anti-collision system based on multimodal data fusion, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the stacker-reclaimer anti-collision method based on multimodal data fusion described above.

[0014] Compared with the prior art, the beneficial effects of the present invention are: The spatiotemporal synchronous fusion technology of multimodal sensing data eliminates the time deviation of data acquisition from various sensors by aligning the timestamps of LiDAR scan point clouds, ultra-wideband positioning coordinates, inertial measurement unit attitude angles, and high-definition video image streams. Then, it transforms the data from different local coordinate systems to the global coordinate system through a unified spatial coordinate system, forming sensing data that includes stacker-reclaimer pose, obstacle contours, and spatiotemporal correlation information. This technology avoids the limitations of single-sensor systems affected by environmental interference, resolves fusion conflicts caused by spatiotemporal misalignment of multi-source data, and enables the sensing results to more comprehensively reflect the dynamics of the equipment and environment, providing accurate input for subsequent pose determination and obstacle tracking.

[0015] Dynamic collision detection technology based on motion intent prediction utilizes the fused pose state of the stacker-reclaimer and real-time control commands from the equipment control system to calculate its motion trend and mechanically constrained reachable space within a preset future time window. This space is then dynamically compared with the perceived contours of dynamic obstacles and the extrapolated motion paths from historical trajectories. This technology overcomes the instantaneous limitations of static detection by predicting the overlap between all possible locations the equipment might reach and the future locations of obstacles. It identifies risk points where there is currently no overlap but a collision is inevitable, accurately calculating the estimated collision time and the minimum safety margin required for avoidance, thus enabling forward-looking collision avoidance strategies. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a stacker-reclaimer anti-collision method based on multimodal data fusion as described in this invention. Figure 2 A flowchart for generating spatiotemporally synchronized multimodal sensing data; Figure 3 This is a flowchart of multimodal feature extraction and fusion. Figure 4 A comparison chart of minimum position adjustments under different obstacle speeds; Figure 5 This is a diagram for obstacle mutation detection and analysis. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 This method acquires real-time operational status data of the target stacker-reclaimer and related equipment within the working area. This data includes LiDAR scan point clouds, ultra-wideband positioning coordinates, inertial measurement unit attitude angles, high-definition video image streams, and real-time control commands from the equipment control system. The LiDAR scan point clouds, ultra-wideband positioning coordinates, inertial measurement unit attitude angles, and high-definition video image streams are time-stamped and unified with spatial coordinate systems to obtain spatiotemporally synchronized multimodal perception data. Multimodal feature extraction and fusion operations are performed on this spatiotemporally synchronized multimodal perception data to obtain the target stacker-reclaimer's own pose information and the perceived contours and trajectories of dynamic obstacles within the working area. Based on the target stacker-reclaimer's own pose information and real-time control commands, the method predicts the target stacker-reclaimer's intended movement and reachable movement space within a preset time window. Dynamic collision detection is performed between the target stacker-reclaimer's reachable movement space and the perceived contours and trajectories of dynamic obstacles. If a potential collision risk is detected, the collision risk point, the estimated collision time, and the minimum safety margin required for avoidance are calculated. Based on collision risk points, expected collision time, and minimum safety margin, a hierarchical collision avoidance control strategy is generated.

[0019] See Figure 2 In one embodiment of the present invention, the implementation of timestamp alignment and spatial coordinate system one involves the precise synchronization and coordinate transformation of multiple sensor data. A unified high-precision timestamp is applied to each frame of LiDAR scan point cloud, each set of ultra-wideband positioning coordinates, each set of inertial measurement unit attitude angles, and each frame of high-definition video image stream. The high-precision timestamp originates from a master clock source within the system, and its time synchronization accuracy must reach the microsecond level. For example, if the LiDAR completes scanning a frame of point cloud at time t, a timestamp T is immediately applied to that frame of point cloud data. At the same time, the attitude angle data packet output by the inertial measurement unit is also marked with the same timestamp T, thereby ensuring that all data strictly correspond in the time dimension. A global workspace coordinate system is established with a preset reference point on the target stacker-reclaimer body as the origin. The preset reference point is usually selected at the intersection of the stacker-reclaimer's rotation center and the trolley track plane. Starting from this point, the direction along the trolley track is defined as the X-axis, the direction perpendicular to the track in the horizontal plane is defined as the Y-axis, and the direction perpendicular to the horizontal plane upwards is defined as the Z-axis, thus constructing a right-handed rectangular global workspace coordinate system.

[0020] In some embodiments, the lidar scan point cloud is transformed from the lidar's own coordinate system to the global workspace coordinate system. The lidar's own coordinate system has its origin at the lidar's emission center, and the transformation relationship depends on the lidar's mounting matrix on the stacker-reclaimer. Using a fixed rigid transformation matrix obtained through pre-calibration of the lidar to a preset reference point, the coordinates of each lidar point can be transformed to the global workspace coordinate system. The transformation process can be achieved through a matrix multiplication operation. The coordinate transformation formula is: in: This represents the three-dimensional coordinate vector in the global workspace coordinate system after transformation. This represents the original three-dimensional coordinate vector in the lidar's own coordinate system. It is a 3x3 rotation matrix that describes the attitude rotation relationship from the lidar coordinate system to the global workspace coordinate system. It is a 3x1 translation vector describing the translation relationship from the origin of the lidar coordinate system to the origin of the global workspace coordinate system. Rotation matrix. Translation vector The external parameters are determined in advance through the post-installation external parameter calibration process.

[0021] In practical implementation, the ultra-wideband (UWB) positioning coordinates are transformed from the positioning base station coordinate system to the global workspace coordinate system. An UWB positioning system typically consists of multiple base stations deployed at fixed locations and a tag installed on a stacker-reclaimer. Its output coordinates are based on an independent coordinate system established by the positioning base stations. The transformation relationship between the positioning base station coordinate system and the global workspace coordinate system needs to be determined beforehand through calibration measurements. This relationship is also represented by a rotation matrix and a translation vector. After obtaining the real-time coordinates of the UWB tag, this transformation relationship can be applied to obtain the tag's coordinates in the global workspace coordinate system. Since the tag has a fixed positional relationship with a preset reference point on the stacker-reclaimer, further calculations can obtain the global coordinates of the preset reference point itself. The attitude correction of the high-definition video image stream is performed using the attitude angle of the inertial measurement unit. The inertial measurement unit measures the roll angle, pitch angle and yaw angle changes of the stacker-reclaimer body in real time. When the instantaneous shaking of the body caused by the movement or start-up and stop of the trolley is detected, digital anti-shake processing is performed on the high-definition video images acquired in the corresponding time period according to the change of attitude angle, so as to eliminate high-frequency jitter in the image sequence and stabilize the image content on the horizontal reference plane indicated by the inertial measurement unit.

[0022] Optionally, the corrected image pixel coordinates are mapped to a two-dimensional plane in the global workspace coordinate system. This mapping depends on the camera's pinhole imaging model and the camera's extrinsic parameter matrix relative to the global workspace coordinate system. By pre-calibrating the camera's intrinsic and extrinsic parameters, the camera's intrinsic parameter matrix and the transformation matrix from the camera coordinate system to the global workspace coordinate system can be obtained. For any pixel in the corrected image, a ray originating from that pixel in the global workspace coordinate system can be calculated using the above parameters. This ray intersects a preset reference plane, and the intersection point is the projected coordinate of that pixel on the two-dimensional plane in the global workspace coordinate system. The reference plane is typically chosen as the ground plane of the stacker-reclaimer's operating area.

[0023] It is understandable that ensuring all modal data are represented in the global workspace coordinate system under the same high-precision timestamp is the final step in achieving spatiotemporal synchronization. The data processing unit caches and aligns all input data streams according to their timestamps. For example, for timestamp T, the system extracts the LiDAR point cloud frame, ultra-wideband coordinate data packet, inertial measurement unit data packet, and high-definition video image frame with the timestamp closest to T from the cache, and packages these data, which have been uniformly transformed into the global workspace coordinate system, into a synchronization data packet. This data packet represents a consistent description of the same workspace from different sensing dimensions at time T.

[0024] See Figure 3In one embodiment of the present invention, the multimodal feature extraction and fusion operation begins with spatiotemporally synchronized multimodal sensing data. Geometric structural features of the LiDAR point cloud are extracted from the spatiotemporally synchronized multimodal sensing data. These geometric structural features are calculated using a point cloud processing algorithm and include the surface area, volume, principal axis length, plane fitting residual, and local curvature distribution histogram of the point cloud clusters. These features describe the three-dimensional shape and surface properties of objects in the environment. Position stability features of ultra-wideband coordinates are extracted from the spatiotemporally synchronized multimodal sensing data. These position stability features are obtained through statistical analysis of ultra-wideband positioning coordinates over multiple consecutive periods, including the sliding variance of coordinate values, average displacement velocity, and offset relative to the fitted smooth path. These features reflect the noise level of the positioning signal and the smoothness of the target's motion over a short period. Attitude change rate features of the inertial measurement unit (IMU) are extracted from spatiotemporally synchronized multimodal sensing data. These features are derived from the raw angular velocity and linear acceleration data output by the IMU, including the mean and peak values ​​of pitch, roll, and yaw angular velocities, as well as the trend of the magnitude of the synthetic acceleration. These features characterize the instantaneous intensity of the stacker-reclaimer's motion. Visual semantic features of high-definition video images are also extracted from the spatiotemporally synchronized multimodal sensing data. These features are obtained by forward propagation calculation of high-definition video image frames using a pre-trained convolutional neural network model. The model outputs a high-dimensional feature vector, which encodes the category semantic information, texture information, and scene context information of objects in the image.

[0025] In some embodiments, an attention-based feature fusion network is used to weightedly fuse geometric structural features, positional stability features, pose change rate features, and visual semantic features. The attention-based feature fusion network learns a dynamic weight coefficient for each modality's features, the magnitude of which depends on the reliability and information richness of the modality's features in the current scene. The feature fusion network first projects four different types of feature vectors onto a unified feature dimension through a fully connected layer. Then, the projected feature vectors are concatenated and input into an attention scoring module, which outputs a four-dimensional attention weight vector. The final unified environmental feature representation is obtained by weighted summation of the projected feature vectors from each modality and their corresponding attention weights. This weighted fusion process can be expressed by the following formula: in: This represents the unified environmental feature representation vector generated after fusion. These are modal indexes, corresponding to four modes: lidar, ultra-wideband, inertial measurement unit, and high-definition video. This represents the original feature vector of the i-th mode. It is the original feature vector of the i-th mode. Projected onto a learnable weight matrix of uniform dimension, It is the attention weight corresponding to the i-th modal feature, calculated by the attention scoring module, and satisfies the following conditions: .

[0026] Optionally, based on a unified environmental feature representation, an instance segmentation network is used to identify the point cloud clusters and image regions of the target stacker-reclaimer. The instance segmentation network takes the unified environmental feature representation as input. This network architecture is typically based on a fully convolutional network, and its output includes a semantic category label for each 3D pixel or 2D image pixel, as well as an instance identifier. By querying point cloud clusters and image regions belonging to the semantic category "stacker-reclaimer" and whose instance identifiers point to themselves, the components of the target stacker-reclaimer can be segmented from the perceptual data. Combining ultra-wideband coordinates and inertial measurement unit (IMU) attitude angles, the 3D position, orientation, boom pitch angle, and slewing angle of the target stacker-reclaimer are calculated as pose state information. Specifically, the 3D position is mainly derived from the calibrated ultra-wideband coordinates, the orientation is given by the IMU's heading angle, the boom pitch angle is calculated by combining the IMU's pitch angle with the fixed angle between the boom and the machine body, and the slewing angle is calculated from the image offset angle of the boom root relative to the machine body reference plane identified by the instance segmentation network.

[0027] Understandably, based on a unified environmental feature representation, the system identifies and tracks moving point cloud clusters and image regions within the working area that are not of the "Stacker-Reclaimer" itself, classifying them as dynamic obstacles. This unified environmental feature representation is input into the same instance segmentation network. The network assigns additional semantic labels, such as "vehicle," "personnel," and "other equipment," to all point cloud clusters and image regions that do not belong to the "Stacker-Reclaimer" category, and assigns them unique instance identifiers. By associating point cloud clusters and image regions with the same instance identifier across multiple consecutive timestamps and using a multi-object tracking algorithm, the 3D contour, real-time position, and velocity vector of each dynamic obstacle can be estimated, forming a perceived contour and motion trajectory. The 3D contour is defined by the 3D convex hull or bounding box of the point cloud cluster, the real-time position is the coordinate of the geometric center of the contour in the global working space coordinate system, and the velocity vector is calculated by dividing the position difference between two consecutive frames by the time interval.

[0028] In one embodiment of the present invention, the process of predicting the motion intention and reachable motion space based on the target stacker-reclaimer's own pose state information and real-time control commands begins with the parsing of the control commands. Parsing the real-time control commands obtains the target trolley travel target, the target angle of the slewing mechanism, the target angle of the pitching mechanism, and the expected action of the reclaiming mechanism. These commands are typically issued in the form of digital signals from the stacker-reclaimer's programmable logic controller or host computer, containing the target setpoints for each motion axis and the desired action sequence. Based on the current state in the pose state information, combined with the kinematic model of the stacker-reclaimer, a forward simulation is performed to calculate the ideal motion path of the target stacker-reclaimer under the drive of the real-time control commands from the current moment until the end of a future preset time window. The kinematic model describes the geometric relationships and motion constraints of the stacker-reclaimer's trolley, slewing, pitching, and other motion joints. The expected position and attitude of each component of the stacker-reclaimer in the global workspace coordinate system are iteratively calculated at discrete time steps, forming a theoretical motion trajectory that does not consider dynamic response characteristics.

[0029] In some embodiments, considering the response delay, acceleration, and speed limits of each moving joint of the target stacker-reclaimer, a motion uncertainty envelope is extended based on the ideal motion path. The response delay includes the control system delay and the response lag of the actuators, while the acceleration and speed limits are determined jointly by the equipment nameplate parameters and safety regulations. The extension process involves calculating an error ellipsoid surrounding each predicted position point on the ideal motion path, based on the maximum possible lag and tracking error of each joint at that moment. Connecting the error ellipsoids of all time steps in spacetime forms a continuous three-dimensional space, which is the motion uncertainty envelope. The space within this envelope is defined as the reachable motion space, which is a dynamic three-dimensional space that changes over time. The mathematical description of the motion uncertainty envelope can be expressed as the Minkowski sum of the ideal path point and a time-varying error set, one form of which is: in: Let represent the set of points in the reachable motion space at time t. Let represent the position vector of a key point (such as the center of the bucket wheel) on the target stacker-reclaimer at time t on the ideal motion path. This represents a time-varying set of position errors, which is obtained by modeling the joint response delay and limiting parameters. Minkowski sum operations are represented. It is the current moment. It is the length of the preset time window.

[0030] Optionally, within each fusion sensing cycle, the reachable motion space of the target stacker-reclaimer at the current moment, as well as the latest perceived contours and motion trajectories of all dynamic obstacles, are acquired. The fusion sensing cycle is a fixed time interval, such as 100 milliseconds. At the beginning of each cycle, the system reads the latest calculated 3D volumetric sequence description of the reachable motion space for the next few seconds from the prediction module, and reads the latest 3D contour bounding boxes and motion trajectories of all tracked dynamic obstacles from the multimodal sensing fusion module. The motion trajectories include the current position and velocity vector. The motion trajectories of the dynamic obstacles are linearly extrapolated to a preset future time window. Assuming the dynamic obstacles maintain uniform linear motion in the current velocity vector direction, their expected positions at each future moment are calculated based on their current position and velocity vector. Combined with the dimensions of their 3D contours, a sequence of 3D bounding boxes that moves and evolves over time is generated.

[0031] It can be understood that the calculation involves spatial interference between the time-evolving sequence of three-dimensional volumes in the reachable motion space and the time-evolving sequence of three-dimensional volumes of the perceived contours of each dynamic obstacle based on linear extrapolation. Spatial interference calculation is performed at discrete time points. For each sampling time point within a future preset time window, it is calculated whether there is a geometric intersection between the three-dimensional volume of the target stacker-reclaimer's reachable motion space at that moment and the expected three-dimensional bounding box of each dynamic obstacle at that moment. If a geometric intersection exists, spatial interference is determined to exist. Spatial interference calculation can employ a fast collision detection algorithm based on bounding boxes, such as the separating axis theorem, to balance real-time performance and accuracy. If spatial interference exists, it is determined to be a potential collision risk, and the starting time of the interference is recorded as the expected collision time, the spatial location of the interference is recorded as the collision risk point, the expected collision time is the earliest moment on the future timeline when spatial interference is detected, and the collision risk point is the centroid of the spatial region where interference occurs at that moment or the point where contact is most likely to occur.

[0032] In one embodiment of the present invention, after spatial interference is detected, the relative motion direction and velocity of the target stacker-reclaimer and the interfering dynamic obstacle before the collision are analyzed. The relative motion direction is obtained by calculating the vector difference between the current unit vector of the target stacker-reclaimer and the unit vector of the dynamic obstacle. The magnitude of the relative velocity is obtained by calculating the difference in magnitude between the velocity vectors of the two moving objects. A virtual repulsive force field centered on the collision risk point is constructed. The virtual repulsive force field defines a repulsive potential energy in space, which is at its maximum value at the collision risk point and decreases monotonically as the spatial point moves away from the collision risk point. The minimum position adjustment required to separate the edge of the reachable motion space of the target stacker-reclaimer from the edge of the perceived contour of the dynamic obstacle before the expected collision time is calculated. The calculation process of the minimum position adjustment is to find a minimum displacement in the gradient direction of the virtual repulsive force field, so that the reachable motion space after displacement and the contour on the linear extrapolated trajectory of the dynamic obstacle do not intersect within a preset time window.

[0033] In some embodiments, the minimum position adjustment includes at least one of the following: the trolley displacement, slewing angle adjustment, or pitch angle adjustment that the target stacker-reclaimer needs to change. The trolley displacement represents the distance the stacker-reclaimer needs to move along the track direction; the slewing angle adjustment represents the angle the stacker-reclaimer's slewing platform needs to deflect; and the pitch angle adjustment represents the angle the stacker-reclaimer's boom needs to raise or lower. These adjustments can be calculated independently or solved jointly to meet separation conditions. Converting the minimum position adjustment into a time-dimensional buffer yields the minimum safety margin. The conversion process is based on the maximum safe operating speed of each moving joint of the stacker-reclaimer. For example, dividing the trolley displacement by the maximum safe trolley travel speed yields a time value, and dividing the slewing angle adjustment by the maximum safe angular velocity of the slewing mechanism yields another time value. The maximum value among all calculated time values ​​is selected as the final minimum safety margin. Collision avoidance control strategies include deceleration strategies, pause strategies, and local path replanning strategies. Different urgency response levels are set according to the expected collision time; the shorter the expected collision time, the higher the response level. Response levels are typically represented by incrementing integers.

[0034] Optionally, when the expected collision time exceeds a first threshold, a deceleration strategy is generated. This deceleration strategy includes instructions to slow down one or more moving joints of the target stacker-reclaimer. The deceleration instructions are implemented by modifying the control signals sent to the motor driver or hydraulic proportional valve, and the reduction magnitude is inversely proportional to the expected collision time. When the expected collision time is less than or equal to the first threshold but greater than a second threshold, a pause strategy is generated. This pause strategy includes instructions to smoothly stop all moving joints of the target stacker-reclaimer within a preset time. Smooth stopping is achieved by generating an S-shaped speed curve from the current speed to zero speed and sending it to the motion controller. It is understandable that a local path replanning strategy is generated when the expected collision time is less than or equal to the second threshold. The local path replanning strategy calculates a temporary collision-free path that can avoid the collision risk point based on the minimum safety margin. When calculating the temporary collision-free path, in the global workspace coordinate system, the target stacker-reclaimer is used as the starting point and the expected position of the original path after the safety margin time is used as the ending point. The path search algorithm is used to plan a new path under the constraints of avoiding the collision risk point and its surrounding dangerous area, and generates a sequence of instructions to guide the target stacker-reclaimer to move along the temporary collision-free path.

[0035] The potential energy function of a virtual repulsive force field is used to quantify the proximity of a spatial point to a collision risk point; one mathematical expression of this function is a piecewise function. in: Represents a vector in space The repulsive potential energy value at the point it points to, vector This represents the displacement vector from a spatial point to the point of collision risk. Representing vectors The Euclidean norm is the distance. It is a constant representing the maximum radius of influence of the virtual repulsive force field. This is a positive constant representing the intensity coefficient of the repulsive force field. Referring to Table 1, the correspondence between the response level and the collision avoidance control strategy is determined by a preset time threshold.

[0036] Table 1: Correspondence between Estimated Collision Time and Collision Avoidance Strategies In practice, the first threshold and the second threshold are time constants preset based on the inertia and braking performance of the stacker-reclaimer's motion mechanism. The first threshold is greater than the second threshold. For example, the first threshold is set to 5 seconds and the second threshold is set to 2 seconds.

[0037] See Figure 4This chart compares the minimum position adjustments required for different obstacle speeds in a stacker-reclaimer collision avoidance system. It quantifies the operational range for collision avoidance. The trolley displacement adjustment increases linearly with obstacle speed, reaching approximately 1.6m at 2.0m / s. The slewing angle adjustment shows the largest increase, approaching 4.9° at 2.0m / s, making it the most sensitive of the three adjustments. The pitch angle adjustment shows a moderate increase, approximately 2.5° at 2.0m / s. All three minimum adjustments increase monotonically with obstacle speed, indicating that the faster the obstacle moves, the greater the avoidance maneuver required by the stacker-reclaimer. This chart provides a visual assessment of the necessary adjustments for each motion mechanism of the stacker-reclaimer at different obstacle speeds. The slewing angle adjustment is the most sensitive to obstacle speed; in high-speed scenarios, the adjustment range of the slewing mechanism is often a key factor in determining whether a collision can be avoided.

[0038] In one embodiment of the present invention, the process of converting the anti-collision control strategy into a specific motion control command sequence is a step of deconstructing the abstract strategy into underlying execution instructions. The abstract control objectives contained in the deceleration strategy, pause strategy, or local path replanning strategy are decomposed into specific set values ​​for each motion joint of the target stacker-reclaimer within multiple discrete control cycles. These set values ​​include the target speed of the trolley travel motor, the target angle of the slewing motor, and the target stroke of the pitch hydraulic cylinder. For example, for a local path replanning strategy with a future path of 3 seconds, it needs to be converted into a sequence of trolley speed command values, a slewing angle command value sequence, and a pitch stroke command value sequence with a control cycle of 100 milliseconds. The set values ​​of all motion joints are arranged into a time-synchronized instruction set according to the order of the control cycles to form a motion control command sequence. The motion control command sequence is a list sorted by timestamps. Each entry in the list contains a future time point and a combination of command values ​​that should be issued simultaneously to all motion joints at that time point. The motion control command sequence is then sent to the equipment control system of the target stacker-reclaimer. Upon receiving the sequence, the equipment control system executes it line by line according to the timestamps.

[0039] In some embodiments, during the execution of motion control command sequences by the target stacker-reclaimer, the sensing contour and trajectory of the dynamic obstacle are continuously monitored for abrupt changes. During monitoring, the changes in position, velocity, and contour of the dynamic obstacle between adjacent sensing cycles are calculated. The position change is the Euclidean distance between the center of the dynamic obstacle's contour in the coordinate system of adjacent cycles; the velocity change is the difference in the magnitude of the dynamic obstacle's velocity vector; and the contour change is the sum of the absolute values ​​of the changes in the dimensions of the dynamic obstacle's three-dimensional bounding box in the length, width, and height dimensions. These changes can be calculated using the following mathematical expressions: in: This represents the calculated change, which can refer to changes in position, velocity, or profile dimensions. This represents the physical quantity measured in the k-th sensing cycle. This represents the corresponding physical quantity measured in the (k-1)th sensing cycle. This indicates the operation of calculating Euclidean distance or absolute difference; the specific operation form depends on the dimension of the physical quantity.

[0040] Optionally, if any of the changes in position, velocity, or contour exceeds its corresponding preset abrupt change threshold, the perceived contour and motion trajectory of the dynamic obstacle are determined to have undergone abrupt change. The preset abrupt change threshold is pre-set based on the safety standards of the working area and the accuracy of the sensors; for example, the position abrupt change threshold is set to 2 meters, the velocity abrupt change threshold is set to 1.5 meters per second, and the contour abrupt change threshold is set to 0.5 meters. The currently executing motion control command sequence is immediately interrupted. The interruption operation is achieved by sending an emergency stop or pause command to the equipment control system and clearing the queue of unexecuted commands. Based on the latest perceived contour and motion trajectory of the dynamic obstacle after the abrupt change, the dynamic collision detection and anti-collision control strategy generation steps are re-executed. This means that the system needs to start from the changed environmental state, recalculate the reachable motion space of the target stacker-reclaimer, re-perform interference detection, and generate an updated anti-collision control strategy and corresponding motion control command sequence based on the new collision risk calculation results.

[0041] It is understandable that a collision avoidance case library is constructed by recording the entire process of multimodal perception data for each collision risk event, the generated collision avoidance control strategy, and its execution results. A case record is created for each complete collision avoidance intervention process. This record includes a snapshot of the original multimodal perception data when the collision risk is triggered, the calculated reachable motion space and dynamic obstacle information, the complete parameters of the generated collision avoidance control strategy, the actual sequence of motion control commands issued, and a success or failure indicator of the actual collision avoidance result after execution. Case data is periodically extracted from the collision avoidance case library to optimize the parameters of the strategy model used to generate hierarchical collision avoidance control strategies. The optimization training process employs supervised learning or reinforcement learning methods, using the input data and result indicators from the case records as training samples. The goal of the optimization training is to make the collision avoidance control strategy generated by the strategy model require a smaller minimum safety margin or incur less productivity loss in similar risk scenarios. This means that the trained model can make collision avoidance decisions with less interference and greater economy while ensuring safety.

[0042] See Figure 5This is an obstacle abrupt change detection and analysis diagram, showing the changes in the position, velocity, and contour of a dynamic obstacle within adjacent sensing cycles in the stacker-reclaimer collision avoidance system. This is used to determine whether an abrupt change has occurred and trigger a replanning of the collision avoidance strategy. The position change peaks at approximately 5 seconds, reaching about 2.5m, far exceeding the position abrupt change threshold; the velocity change peaks at approximately 5 seconds, reaching about 1.8m / s, exceeding the velocity abrupt change threshold; and the contour change peaks at approximately 5 seconds, reaching about 0.8m, exceeding the contour abrupt change threshold. The three dashed lines represent the abrupt change thresholds for position, velocity, and contour, respectively, used to determine whether a replanning is triggered. At approximately 5 seconds, if the changes in position, velocity, and contour all exceed their respective abrupt change thresholds, the system will determine that the motion state of the dynamic obstacle has abruptly changed. Through multi-dimensional abrupt change detection, the system avoids misjudgments based on a single indicator, improving its robustness under complex operating conditions.

[0043] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A collision avoidance method for stacker-reclaimers based on multimodal data fusion, characterized in that, include: Real-time acquisition of operating status data of the target stacker-reclaimer and related equipment within the working area. The operating status data includes lidar scan point cloud, ultra-wideband positioning coordinates, inertial measurement unit attitude angle, high-definition video image stream, and real-time control commands of the equipment control system. The lidar scan point cloud, the ultra-wideband positioning coordinates, the inertial measurement unit attitude angle, and the high-definition video image stream are time-stamped and aligned with the spatial coordinate system to obtain spatiotemporally synchronized multimodal sensing data. Multimodal feature extraction and fusion operations are performed on the spatiotemporally synchronized multimodal sensing data to obtain the pose state information of the target stacker-reclaimer itself and the sensing contours and motion trajectories of dynamic obstacles in the working area; Based on the target stacker-reclaimer's own pose information and the real-time control commands, the target stacker-reclaimer's movement intention and reachable movement space within a preset time window in the future are predicted; The reachable motion space of the target stacker-reclaimer is dynamically collided with the perceived contour and motion trajectory of the dynamic obstacle. If a potential collision risk is detected, the collision risk point, the estimated collision time, and the minimum safety margin required for avoidance are calculated. Based on the collision risk points, the expected collision time, and the minimum safety margin, a hierarchical collision avoidance control strategy is generated.

2. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 1, characterized in that, The lidar scan point cloud, the ultra-wideband positioning coordinates, the inertial measurement unit attitude angles, and the high-definition video image stream are time-stamped and aligned with the spatial coordinate system to obtain spatiotemporally synchronized multimodal sensing data, including: A unified, high-precision timestamp is applied to each frame of lidar scan point cloud, each set of ultra-wideband positioning coordinates, each set of inertial measurement unit attitude angles, and each frame of high-definition video image stream. A global workspace coordinate system is established with a preset reference point on the target stacker-reclaimer body as the origin; Transform the laser radar scan point cloud from the radar's own coordinate system to the global workspace coordinate system; Transform the ultra-wideband positioning coordinates from the positioning base station coordinate system to the global workspace coordinate system; The attitude angle of the inertial measurement unit is used to perform attitude correction on the high-definition video image stream to eliminate image jitter caused by body shaking, and the corrected image pixel coordinates are mapped to a two-dimensional plane in the global workspace coordinate system; Ensure that all modal data are expressed in the global workspace coordinate system under the same high-precision timestamp, thus achieving spatiotemporal synchronization.

3. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 2, characterized in that, Multimodal feature extraction and fusion operations are performed on the spatiotemporally synchronized multimodal sensing data to obtain the pose state information of the target stacker-reclaimer itself and the perceived contours and motion trajectories of dynamic obstacles within the working area, including: From the spatiotemporally synchronized multimodal sensing data, the geometric structure features of the lidar point cloud, the position stability features of the ultra-wideband coordinates, the attitude change rate features of the inertial measurement unit, and the visual semantic features of the high-definition video image are extracted. An attention-based feature fusion network is used to weight and fuse the geometric structural features, the positional stability features, the pose change rate features, and the visual semantic features to generate a unified environmental feature representation. Based on the unified environmental feature representation, the point cloud clusters and image regions of the target stacker-reclaimer are identified through an instance segmentation network. Combined with the ultra-wideband coordinates and the attitude angle of the inertial measurement unit, the three-dimensional position, orientation, boom pitch angle and rotation angle of the target stacker-reclaimer are calculated as the pose state information. Simultaneously, based on the unified environmental feature representation, non-self moving point cloud clusters and image regions within the working area are identified and tracked, classified as dynamic obstacles, and the three-dimensional contour, real-time position, and velocity vector of each dynamic obstacle are estimated to form the perception contour and motion trajectory.

4. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 3, characterized in that, Based on the target stacker-reclaimer's own pose information and the real-time control commands, the movement intention and reachable movement space of the target stacker-reclaimer within a preset time window are predicted, including: The real-time control commands are analyzed to obtain the target travel target of the target stacker-reclaimer, the target angle of the slewing mechanism, the target angle of the pitching mechanism, and the expected action of the reclaiming mechanism. Based on the current state in the pose state information and combined with the kinematic model of the stacker-reclaimer, the forward simulation calculates the ideal motion path of the target stacker-reclaimer under the drive of the real-time control command from the current moment to the end of the future preset time window. Considering the response delay, acceleration, and speed limits of each moving joint of the target stacker-reclaimer, a motion uncertainty envelope is extended based on the ideal motion path. The space within the motion uncertainty envelope is the reachable motion space. The reachable motion space is a dynamic three-dimensional space that changes over time.

5. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 4, characterized in that, Dynamic collision detection is performed between the reachable motion space of the target stacker-reclaimer and the perceived contours and trajectories of the dynamic obstacles, including: Within each fusion perception cycle, the reachable motion space of the target stacker-reclaimer at the current moment, as well as the latest perception contours and motion trajectories of all dynamic obstacles, are acquired. The motion trajectory of the dynamic obstacle is linearly extrapolated to the future preset time window; Calculate the spatial interference between the three-dimensional volume sequence of the reachable motion space evolving over time and the three-dimensional volume sequence of the perceived contour of each dynamic obstacle evolving over time based on linear extrapolation; If spatial interference exists, it is determined that there is a potential collision risk, and the starting time of the interference is recorded as the expected collision time, and the spatial location where the interference occurs is recorded as the collision risk point.

6. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 5, characterized in that, If a potential collision risk is detected, the collision risk point, the estimated collision time, and the minimum safety margin required for avoidance are calculated, including: After detecting spatial interference, the relative motion direction and velocity of the target stacker-reclaimer and the dynamic obstacle that is interfering with it are analyzed before the collision occurs. Construct a virtual repulsive force field centered on the collision risk point, and calculate the minimum position adjustment required to separate the edge of the reachable motion space of the target stacker-reclaimer from the edge of the perceived contour of the dynamic obstacle before the expected collision time; The minimum position adjustment amount includes at least one of the following: the amount of trolley displacement, the amount of slewing angle adjustment, or the amount of pitch angle adjustment that the target stacker-reclaimer needs to change. The minimum position adjustment amount is converted into a buffer in the time dimension, which is the minimum safety margin.

7. The anti-collision method for stacker-reclaimers based on multimodal data fusion as described in claim 6, characterized in that, Based on the collision risk points, the estimated collision time, and the minimum safety margin, a hierarchical collision avoidance control strategy is generated, including: The collision avoidance control strategy includes a deceleration strategy, a pause strategy, and a local path replanning strategy. Based on the expected collision time, different response levels are set according to the urgency level; the shorter the expected collision time, the higher the response level. When the expected collision time is greater than a first threshold, the deceleration strategy is generated, the deceleration strategy including instructions to decelerate one or more moving joints of the target stacker-reclaimer; When the expected collision time is less than or equal to the first threshold but greater than the second threshold, the pause strategy is generated. The pause strategy includes instructions to smoothly stop all moving joints of the target stacker-reclaimer within a preset time. When the expected collision time is less than or equal to the second threshold, the local path replanning strategy is generated. The local path replanning strategy calculates a temporary collision-free path that can avoid the collision risk point based on the minimum safety margin, and generates a sequence of instructions to guide the target stacker-reclaimer to move along the temporary collision-free path.

8. The collision avoidance method for stacker-reclaimers based on multimodal data fusion as described in claim 7, characterized in that, Also includes: The anti-collision control strategy is converted into a specific motion control command sequence, and the motion control command sequence is sent to the equipment control system of the target stacker-reclaimer; During the execution of the motion control command sequence by the target stacker-reclaimer, the sensing contour and motion trajectory of the dynamic obstacle are continuously monitored for abrupt changes. If the perceived contour and motion trajectory of the dynamic obstacle change abruptly, the online replanning of the collision avoidance control strategy is triggered, and the motion control command sequence is updated. Record the entire process of each collision risk event using multimodal perception data, the generated collision avoidance control strategy and its execution results, and build a collision avoidance case library for subsequent optimization of strategy generation models; The step of converting the collision avoidance control strategy into a specific motion control command sequence includes: The abstract control objective contained in the deceleration strategy, the pause strategy, or the local path replanning strategy is decomposed into specific set values ​​of each moving joint of the target stacker-reclaimer in multiple discrete control cycles. The set values ​​include the target speed of the trolley travel motor, the target angle of the rotary motor, and the target stroke of the pitch hydraulic cylinder. According to the sequence of control cycles, the set values ​​of all moving joints are arranged into a time-synchronized instruction set to form the motion control instruction sequence. If the perceived contour and trajectory of the dynamic obstacle change abruptly, the online replanning of the collision avoidance control strategy is triggered, including: During the monitoring process, the changes in position, velocity, and contour of the dynamic obstacle between adjacent sensing cycles are calculated. If any of the position change, velocity change, or contour change exceeds its corresponding preset mutation threshold, it is determined that the perceived contour and motion trajectory of the dynamic obstacle have undergone a sudden change. Immediately interrupt the currently executing sequence of motion control commands; Based on the latest perceived contour and motion trajectory after the dynamic obstacle mutation, the dynamic collision detection and anti-collision control strategy generation steps are re-executed to obtain the updated anti-collision control strategy and the corresponding motion control command sequence.

9. A collision avoidance method for stacker-reclaimers based on multimodal data fusion as described in claim 8, characterized in that, The construction of the anti-collision case library, used for subsequent optimization strategy generation models, includes: Create a case record for each complete collision avoidance intervention process; The case record includes: a snapshot of the original multimodal perception data when a collision risk is triggered, the calculated reachable motion space and dynamic obstacle information, the complete parameters of the generated collision avoidance control strategy, the actual sequence of motion control commands issued, and the success or failure indicator of the actual collision avoidance result after execution. Periodically extract case data from the collision avoidance case library to optimize and train the parameters of the strategy model used to generate the hierarchical collision avoidance control strategy; The goal of the optimization training is to make the collision avoidance control strategy generated by the strategy model require a smaller minimum safety margin or result in a lower loss of production efficiency under similar risk scenarios.

10. A stacker-reclaimer anti-collision system based on multimodal data fusion, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the stacker-reclaimer anti-collision method based on multimodal data fusion as described in any one of claims 1 to 9.

Images