Multi-sensor fusion hook unhooking robot operation control method and system

Abstract

The application provides a multi-sensor fusion hooking robot operation control method and system, relates to the technical field of traction devices, and comprises the following steps: after triggering a hooking command, controlling a vision sensor and a laser radar to collect scene environment data and reconstruct a static hooking scene; performing scene feature recognition, path fitting of hooking connection, and establishing a target reference path; after executing motion control of the hooking robot, synchronously performing incremental collection of scene data, updating key dynamic information on the static hooking scene; performing path consistency authentication, establishing path following compensation; and after adjusting control parameters, completing hooking connection control of the hooking robot. The application solves the technical problem that the prior art mainly executes hooking and connection operations through a single sensor, the single sensor cannot comprehensively perceive environmental information, thus causing deviation in path planning and action execution, even causing failure or delay, and affecting operation efficiency and safety.

Description

Technical Field

[0001] This invention relates to the field of traction device technology, and more specifically to a multi-sensor fusion-based operation control method and system for unhooking robots. Background Technology

[0002] In railway transportation systems, especially in automation technologies used for carriage connection and traction operations, robotic uncoupling is a key technology for improving operational efficiency and reducing manual labor. However, existing technologies primarily rely on robotic arms and single sensors to perform uncoupling and connection operations. While these systems can complete basic tasks in static environments, they lack sufficient flexibility and accuracy when faced with environmental changes or dynamic factors. Furthermore, a single sensor may not be able to comprehensively perceive environmental information, leading to deviations in path planning and action execution, or even malfunctions or delays, affecting operational efficiency and safety. Therefore, existing technologies still have significant shortcomings in multi-sensor fusion and dynamic adaptation. Summary of the Invention

[0003] This application provides a multi-sensor fusion-based operation control method and system for unhooking robots, aiming to solve the technical problem that existing technologies mainly rely on a single sensor to perform unhooking and connection operations. A single sensor cannot fully perceive environmental information, which leads to deviations in path planning and action execution, or even malfunctions or delays, affecting operational efficiency and safety.

[0004] The first aspect disclosed in this application provides a multi-sensor fusion-based operation control method for a hook-unhooking robot. The method includes: after triggering a hook-unhooking command, using the current time node as the zero point, controlling a vision sensor and a lidar to collect scene environment data and reconstruct a static hook-unhooking scene; performing scene feature recognition on the static hook-unhooking scene, fitting a hook-unhooking connection path based on the track direction features and hook-unhooking coordination features in the scene feature recognition results, establishing a target reference path, and configuring control parameters mapped to the target reference path; after executing motion control of the hook-unhooking robot with the control parameters, timing at the zero point, synchronously activating the vision sensor and lidar to perform incremental scene data collection, updating key dynamic information on the static hook-unhooking scene based on the incremental collection results; performing path consistency authentication based on the updated key dynamic information and the target reference path, establishing path following compensation, wherein the path following compensation is constructed through multiple balancing targets; and adjusting the control parameters using the path following compensation to complete the hook-unhooking connection control of the hook-unhooking robot.

[0005] The second aspect of this application discloses a multi-sensor fusion unhooking robot operation control system. This system is used in the aforementioned multi-sensor fusion unhooking robot operation control method. The system includes: a scene environment data acquisition module, used to, after triggering an unhooking command, use the current time node as the time zero point to control a vision sensor and a lidar to acquire scene environment data and reconstruct a static unhooking scene; and an unhooking connection path fitting module, used to perform scene feature recognition on the static unhooking scene, and based on the track direction features and unhooking coordination features in the scene feature recognition results, to perform unhooking connection path fitting, establish a target reference path, and configure a path with the target... The system includes: control parameters for reference path mapping; a scene data incremental acquisition module, used to execute motion control of the unhooking robot with the control parameters, time the zero point, synchronously activate the vision sensor and lidar to perform incremental scene data acquisition, and update key dynamic information on the static unhooking scene based on the incremental acquisition results; a path consistency authentication module, used to perform path consistency authentication based on the updated key dynamic information and the target reference path, and establish path following compensation, wherein the path following compensation is constructed through multiple balancing targets; and an unhooking connection control module, used to adjust the control parameters using the path following compensation to complete the unhooking connection control of the unhooking robot.

[0006] One or more technical solutions provided in this application have at least the following beneficial effects:

[0007] By fusing multiple sensors, including visual sensors and LiDAR, the robot can autonomously collect environmental data and reconstruct static unhooking scenarios. This provides reliable environmental perception for automated unhooking operations, reducing manual intervention and improving efficiency. Feature recognition of the static unhooking scenario accurately extracts track direction and unhooking coordination features, enabling path fitting for unhooking connections and generating a target reference path. This path fitting precisely considers track features and the geometric information of the target coordination points, providing high-precision path planning for the robot's unhooking actions. During unhooking operations, real-time incremental data acquisition updates key dynamic information in the static unhooking scenario. This update ensures the robot can adapt to environmental changes and adjust its operational path in real time, enhancing its adaptability in dynamic environments. Path consistency authentication and path-following compensation verify the accuracy of the robot's execution path in real time and dynamically compensate for deviations. This compensation is achieved through multiple balancing targets, effectively improving the robot's path stability and accuracy. Control parameter adjustments through path-following compensation ensure the motion control precision of the unhooking robot. The adjusted control parameters help the robot accurately complete the unhooking connection task, improving the success rate and stability of the operation. In summary, this method can effectively improve the accuracy, reliability, and flexibility of automated uncoupling operations, providing efficient and safe technical support for carriage connection and traction operations in railway transportation systems.

[0008] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0009] Figure 1 This is a schematic flowchart of a multi-sensor fusion-based unhooking robot operation control method provided in an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the structure of the multi-sensor fusion unhooking robot operation control system provided in the embodiments of this application.

[0011] Figure labeling: Scene environment data acquisition module 10, hook connection path fitting module 20, scene data incremental acquisition module 30, path consistency authentication module 40, hook connection control module 50. Detailed Implementation

[0012] This application provides a multi-sensor fusion-based unhooking robot operation control method and system, which solves the technical problem that the existing technology mainly uses a single sensor to perform unhooking and connection operations. A single sensor cannot fully perceive environmental information, which leads to deviations in path planning and action execution, or even malfunctions or delays, affecting operation efficiency and safety.

[0013] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0014] Example 1, as Figure 1 As shown in the figure, this application provides a multi-sensor fusion-based operation control method for a hook-unhooking robot, the method comprising:

[0015] After triggering the unhooking command, the current time node is taken as the zero point, and the vision sensor and lidar are controlled to collect scene environment data to reconstruct the static unhooking scene.

[0016] In automated carriage connection or traction operations, once the uncoupling command is triggered, the uncoupling robot begins to execute the corresponding actions. The triggering of the uncoupling command is the starting point of the entire task, marking the robot's entry into the working state. When the uncoupling command is triggered, the current time node is taken as the zero point, providing a reference for subsequent time synchronization and data processing. All subsequent operations are calibrated based on this timestamp to ensure synchronized data processing from different sensors.

[0017] At this point, the visual sensors and LiDAR are used to collect scene environment data. The visual sensors, such as cameras, capture image data and provide two-dimensional information about the environment, helping the robot to identify visual features of the connecting areas of the carriage. The LiDAR collects three-dimensional point cloud data of the environment, which helps to identify objects, structures, obstacles and other important information in the environment.

[0018] By reconstructing a static unhooking scene using environmental data collected by visual sensors and LiDAR, a virtual model reflecting the current environmental state is formed based on the acquired data. This model includes information about static objects such as carriages, tracks, and connectors, providing a foundation for subsequent path planning and control. Specifically, multi-scale pyramid coding processes image and radar data at different resolutions to obtain information at different levels. Inter-scale correlation coding links information from different scales, thus fusing the characteristics of multiple sensors to generate a more accurate environmental model. Finally, an attention mechanism based on dual-channel recognition results prioritizes important environmental features when fusing visual and LiDAR data, thereby improving the effectiveness of static scene reconstruction.

[0019] The static unhooking scenario is subjected to scene feature recognition. Based on the track direction features and unhooking coordination features in the scene feature recognition results, the unhooking connection path is fitted, a target reference path is established, and control parameters mapped to the target reference path are configured.

[0020] The main objective of scene feature recognition in static unhooking scenarios is to identify key features related to the unhooking operation, including track direction features and unhooking coordination features. Track direction features include identifying the direction, orientation, and changes of the track, which are used for precise robot positioning and path planning. Unhooking coordination features include identifying coordination features related to the unhooking operation, such as the position, shape, and orientation of the connector, to ensure that the robot can correctly perform the unhooking action.

[0021] Based on the aforementioned feature recognition results, a path is fitted for the hook-unhooking operation. During path fitting, feature-based algorithms, such as curve fitting and least squares methods, are used to construct an accurate path model. This path is not merely a simple trajectory, but a comprehensive path encompassing position and orientation. After path fitting is complete, a target reference path is generated. This path serves as the standard for subsequent actions, providing an idealized target for the robot's actions. Based on the target reference path, relevant control parameters are configured according to environmental characteristics, path planning, and task requirements. These control parameters include velocity, acceleration, and attitude adjustment parameters, ensuring that the robot can efficiently and safely complete the hook-unhooking task in a dynamic environment.

[0022] After executing the motion control of the unhooking robot with the control parameters, timing is set to zero, and the vision sensor and lidar are activated simultaneously to perform incremental scene data acquisition. Based on the incremental acquisition results, key dynamic information is updated on the static unhooking scene.

[0023] The motion control of the unhooking robot is executed using control parameters. After the robot starts executing motion control, timing is based on the zero point of time. This is to ensure that the robot's motion state is synchronized with the time of subsequent incremental data acquisition. Through time control, the robot's state at a specific time node can be accurately tracked.

[0024] During the robot's task execution, the vision sensors and LiDAR continue to collect data. This data collection differs from the initial environmental modeling data collection; it is incremental collection. That is, the sensors will only collect data related to the robot's movement. Specifically, the vision sensors capture changes in the relative positions of the carriage, the track, and the robot and the target object. The LiDAR provides 3D point cloud data, which further helps the system identify dynamically changing environmental features, such as the relative position between the robot and the target connector and the movement of any obstacles.

[0025] Based on incremental acquisition results from vision and LiDAR, key dynamic information in static unhooking scenarios is updated. This dynamic information includes the robot's current position, posture, and relative changes with the target connection point, as well as any possible obstacles, dynamic objects, or environmental changes (such as the movement of other carriages), thereby helping the robot adjust its motion strategy in real time.

[0026] Path consistency authentication is performed based on the updated key dynamic information and the target reference path, and path following compensation is established, wherein the path following compensation is constructed through multiple balancing objectives.

[0027] When the robot performs the unhooking task, its path is monitored in real time to ensure consistency with the target reference path. This ensures the robot always moves along the set target reference path. Path deviation is generated in real time by comparing the robot's actual position with the deviation of the target path. If path deviation occurs, path-following compensation is established to allow the robot to return to the ideal path. Specifically, a compensation path is calculated based on updated dynamic information and path deviation. This compensation path is constructed based on multiple balance targets, including position compensation, attitude compensation, and velocity compensation. The balance targets comprehensively consider the deviation between the robot's current state and the ideal state to calculate the optimal adjustment scheme. The specific path-following compensation construction process will be detailed in subsequent steps.

[0028] After adjusting the control parameters using the path following compensation, the hook unhooking and connection control of the unhooking robot is completed.

[0029] The robot's control parameters are adjusted based on path-following compensation. These parameters are adjusted to optimize the robot's motion performance and ensure that it can complete the unhooking operation efficiently and accurately. After path compensation is performed, the robot is controlled to accurately perform the unhooking and connection operation according to the adjusted control parameters. At this time, the robot will accurately perform the docking and unhooking operation of the connector based on the position and posture information on the target path. The unhooking and connection control not only includes the robot's motion control, but also involves the contact and status monitoring with the target connector. Through precise control, the robot can ensure the smooth completion of the unhooking action.

[0030] Furthermore, the step of performing path consistency authentication based on the updated key dynamic information and the target reference path, and establishing path following compensation, includes:

[0031] A consistency comparison channel is established, and the evaluation indicators of the consistency comparison channel include position deviation indicators and attitude deviation indicators; the updated key dynamic information and the target reference path are input into the consistency comparison channel to perform consistency deviation comparison and establish a consistency deviation identifier; environmental feature identification is performed based on the updated key dynamic information, and the adjustment weight of the consistency deviation identifier is updated with the environmental feature identification results; path following compensation is established according to the adjustment weight and the consistency deviation identifier.

[0032] To ensure the robot always moves along the target reference path and adjusts the path in a timely manner, a consistency comparison channel is established. This channel compares the robot's current path with the target reference path in real time and detects any deviations. To quantify path deviations and make effective corrections, evaluation metrics for the channel are defined, including position deviation and attitude deviation metrics. The position deviation metric measures the deviation between the robot's current position and the target reference path. Typically, position deviation refers to the distance between the robot's current position and the target position on the target path, reflecting whether the robot moves accurately along the predetermined path. The attitude deviation metric measures the difference between the robot's attitude (e.g., tilt angle, orientation) and the target path. Attitude deviation directly affects the robot's positioning and operational accuracy, especially during unhooking operations, where the accuracy of the attitude alignment between the robot and the target connector is crucial.

[0033] As the robot executes its path, its dynamic information (such as position, posture, and velocity) updates with time and environmental changes. This updated key dynamic information is acquired and compared with a pre-planned target reference path. The updated dynamic information and the target reference path are input into a consistency comparison channel to calculate the deviation between the current position and the target path. The key to this process is using predefined position and posture deviation indices. Based on the difference between the robot's current state and the ideal state of the target path, the deviation value is calculated. If the position deviation is large, it indicates that the robot has deviated from the target path, requiring positional compensation; if the posture deviation is large, it indicates that the robot's posture is not aligned with the target direction, also requiring posture compensation. Based on the comparison results, a consistency deviation identifier is generated. This identifier records the deviation information between the robot's current path and the target path. This deviation identifier is used for subsequent path compensation and control adjustments to ensure the accuracy of the robot's movement.

[0034] When executing a path, the robot will face various dynamic environmental changes, such as track jitter and environmental occlusion. These changes may lead to greater deviations during path execution. Therefore, it is necessary to analyze these influencing factors through environmental feature recognition. For example, sensors can be used to monitor changes in the track to determine whether the track is jittering or changing; visual sensors can be used to identify whether the target is occluded, making it difficult for the robot to adjust its posture.

[0035] Based on the results of environmental feature recognition, the adjustment weights in the consistency deviation identifier are updated. These weights determine the importance of different types of deviations during path compensation. Specifically, if the track jitter is severe, the adjustment weight for position deviation is increased because track jitter may cause significant changes in the robot's position, requiring more attention to position compensation to ensure the robot can return to the target path. If the target is occluded, the adjustment weight for posture deviation is increased because target occlusion may make it difficult for the robot to accurately dock with the target connector, requiring increased attention to posture compensation.

[0036] Based on the updated consistency deviation flags and adjusted weights from the aforementioned steps, path-following compensation is established for the robot. The purpose of compensation is to correct the deviation between the robot and the target path, ensuring that the robot always stays on the ideal path. This includes, for positional deviations, taking positional compensation measures based on track jitter or positional changes, including adjusting the robot's travel route, speed, or acceleration; for attitude deviations, adjusting the robot's attitude based on target occlusion or environmental interference, ensuring that the robot can dock with the target at the correct angle during the unhooking operation.

[0037] Furthermore, the step of establishing path following compensation based on the adjusted weights and the consistency deviation identifier includes:

[0038] Obtain the deviation degree of the consistency deviation identifier, and establish a first balance target based on the deviation degree; create a distance attention level according to the target reference path, and after determining the current position with updated key dynamic information, match the distance attention level to establish a second balance target; establish path following compensation based on the first balance target, the second balance target, the adjusted weight, and the consistency deviation identifier.

[0039] The deviation degree, which is a measure of the difference between the current path and the target reference path, is obtained based on the consistency deviation indicator. This deviation is expressed as positional or orientation deviation. A first balance target is constructed based on the deviation degree. This target is used to adjust the robot's position on the path. The size of the balance target increases with the deviation degree; that is, the larger the deviation, the greater the adjustment required. The adjustment magnitude is calculated using proportional control or gain adjustment based on the deviation degree and serves as the first balance target. This target controls the robot's speed, acceleration, and path adjustment magnitude. For example, if the robot deviates significantly from the path, the adjustment force is increased so that the robot can return to the target path more quickly.

[0040] To optimize path execution, a distance attention level is created on the target reference path. This level represents the distance between the robot's current position and the target path endpoint, reflecting the urgency of path execution. The distance attention level is designed based on the distance between the robot and the target endpoint or the current position on the path. For example, the closer to the target endpoint, the more attention is paid to the accuracy of the path.

[0041] After acquiring the latest dynamic information, the robot's current position is updated, and the distance to the target endpoint is determined. Based on the distance between the robot's current position and the target endpoint, a second balancing target is established. The purpose of this target is to adjust the path more significantly when the distance is closer, ensuring the robot can quickly adjust to the correct path; and to adjust the path less significantly when the distance is greater, allowing for a larger time step in path adjustment and thus enabling the robot to execute the path more smoothly. Specifically, the distance attention level can be calculated using methods such as Euclidean distance and Manhattan distance, considering the distance between the robot and the target endpoint. The construction of the second balancing target is based on changes in distance, and a weighted method can be used to dynamically adjust the strength of the balancing target. Through the second balancing target, the robot's path-following behavior can be dynamically adjusted, ensuring larger adjustments are made when approaching the target and smaller adjustments are made when moving away from the target.

[0042] Combining the first balancing objective (based on deviation) and the second balancing objective (based on distance attention level), the influence of both is integrated through a weighted approach. The goal is to determine the magnitude and frequency of adjustments. Specifically, if the robot deviates significantly from the path, the first balancing objective receives greater weight, prompting the robot to quickly recover its path; if the robot approaches the target endpoint, the second balancing objective receives greater weight, ensuring more precise adjustments. The balancing objectives are also weighted according to updated adjustment weights. This means that if factors such as track jitter or target occlusion occur in the environment, the weight of path compensation is adjusted accordingly. For example, if track jitter is severe, the weight of position deviation adjustment increases, thus placing greater emphasis on position adjustment. Finally, by combining the first balancing objective, the second balancing objective, adjustment weights, and consistency deviation indicators, the final path-following compensation is calculated. This compensation includes adjustments to the robot's path, speed, acceleration, etc., ensuring that the robot can perform unhooking operations efficiently and stably in dynamic environments.

[0043] Furthermore, the simultaneous activation of the visual sensor and LiDAR for incremental scene data acquisition, and the updating of key dynamic information on the static unhooking scene based on the incremental acquisition results, includes:

[0044] The target mating parts and the unhooking mating parts are identified in the static unhooking scenario to establish key mating features; the trajectory of the unhooking mating parts is fitted according to the target reference path, and the trajectory fitting results are used to locate the trajectory-related key features in the static unhooking scenario; after the static unhooking scenario is set as a zero-point scenario, incremental acquisition results are established by focusing on the key mating features and trajectory-related key features.

[0045] In static uncoupling scenarios, the mating parts related to the uncoupling operation are identified. These mating parts can be carriage connectors, traction devices, or other related components, including target mating parts and uncoupling mating parts. Target mating parts, such as carriage connectors or hooks, typically have specific geometric shapes (e.g., circles, squares) or other unique appearance features. Uncoupling mating parts are mating components of the uncoupling device, such as gripping parts, hooks, or buckles, and typically have specific dimensions, shapes, and relative positions.

[0046] Environmental data collected by visual sensors and LiDAR is used to identify key features of mating parts through feature extraction algorithms such as edge detection, corner detection, and image segmentation. These features include geometry, texture, and color, helping the robot locate and identify targets. By recognizing the features of the target mating parts and the hook-and-unhook mating parts, these features are combined into key mating features. These key features are used for subsequent path planning and action execution. For example, the robot can use these features to accurately determine the target positions such as gripping points and connection points.

[0047] Based on the target reference path, trajectory fitting is performed on the unhooking and mating components. The purpose of trajectory fitting is to generate a trajectory consistent with the target reference path. This trajectory guides the robot's movement, ensuring that the robot can accurately reach the target position when performing the unhooking and mating operation. The trajectory fitting uses algorithms such as least squares, spline interpolation, and curve fitting to smoothly fit the path based on the starting point, ending point, and attitude information provided by the target reference path. This ensures that the path does not have sharp turns or abrupt changes during execution, resulting in a smooth path. Simultaneously, this fitted trajectory considers factors such as the position and angle of the mating components to ensure trajectory accuracy. Using the trajectory fitting results, key trajectory-related features are located in a static unhooking and mating scenario. These key features are elements closely related to path fitting, including the precise position and attitude of the vehicle connecting components and their relationship to the path.

[0048] The static unhooking scenario is set as the zero-point scenario. This zero-point scenario is the reference point for the robot to perform operations, representing the robot's starting position and posture. All subsequent operations and data collection will be based on this zero point to ensure the accuracy and consistency of the data.

[0049] In zero-point scenarios, key features for cooperation and trajectory association are collected with attention. This collection involves dynamically tracking and updating these features to adapt to changes in the environment. Incremental collection updates key features in the scene in real time based on sensors (including cameras, LiDAR, etc.). Specifically, for cooperation key feature collection, the positional changes of the target cooperation parts are continuously tracked to ensure accurate identification and grasping of target parts during path execution. For trajectory association key feature collection, the relative positional relationship between the path and the static scene is continuously updated as the robot moves along the trajectory to ensure that the robot's movement remains accurate in dynamic environments.

[0050] Incremental acquisition results refer to the robot's collection of real-time updated environmental data. This data contains key dynamic information related to path execution and operation, and the incremental acquisition results will serve as the basis for the next step of path adjustment and control.

[0051] Furthermore, the in-scene feature recognition of the static unhooking scene includes:

[0052] After performing contour recognition of the static unhooking scene using edge segmentation, a track candidate region is created based on the contour recognition results; after preprocessing the track candidate region, track features are extracted, and track direction features are created based on the track features.

[0053] In static uncoupling scenarios, to effectively identify the track and its surrounding environment, edge segmentation is first performed on the environmental image or point cloud data. Edge segmentation identifies object edges by detecting changes in brightness or color in the image, typically using edge detection algorithms such as Canny edge detection to extract salient boundaries in the image. Specifically, in static uncoupling scenarios, the carriages, tracks, connecting devices, and other important geometric shapes are identified. Edge segmentation extracts the boundary contours (such as the outer contour of the carriages, the edges of the tracks, etc.) from the image or point cloud data as key information to aid subsequent identification and processing.

[0054] After edge segmentation is completed, the region containing the track is extracted based on the segmented edge information. This region is called the track candidate region. The track candidate region is formed by clustering continuous edge regions in the edge detection results to form one or more regions that represent the track. The track candidate region includes the shape, orientation and spatial location of the track.

[0055] After generating candidate orbit regions, these regions undergo preprocessing to improve the accuracy of orbit feature extraction. Preprocessing includes noise removal and geometric restoration. The preprocessed candidate regions are then further analyzed to extract orbit features. These features are key information describing the orbit's shape and position, including its geometry (e.g., straight or curved tracks, radius of curvature, width, and boundaries). Following feature extraction, the orbit's orientation is further analyzed to generate orientation features. Specifically, the orbit's orientation is determined by calculating the normal and tangent directions of the orbit's boundaries or by fitting a curve to the orbit. If the orbit is curved, the orientation information is generated based on its curvature characteristics to ensure the robot can accurately adapt to curves.

[0056] Furthermore, after executing the motion control of the unhooking robot with the control parameters, the process includes:

[0057] The redundant cameras are activated synchronously, and spatial monitoring of the space is carried out using the redundant cameras to establish a spatial monitoring dataset. Spatial intrusion verification is performed on the spatial monitoring dataset to establish a time-series verification result. When any time-series verification result is an abnormal result, an abnormal warning is reported, and the unhooking robot is controlled to stop.

[0058] Previously, to save computing power and improve focus, only key dynamic features were identified, lacking environmental monitoring, such as interference from people or abnormal equipment. In actual unhooking operations, to improve the safety and stability of the operation, redundant cameras are used in addition to the main visual sensors. These redundant cameras provide additional visual information to avoid the blind spots of a single camera. Spatial monitoring is carried out using redundant cameras, especially for monitoring possible interference factors (such as personnel, other equipment, or obstacles). The data collected by the redundant cameras are summarized and stored as a spatial monitoring dataset for subsequent analysis and verification.

[0059] The spatial monitoring data collected by redundant cameras is analyzed to verify whether the space has been subjected to abnormal intrusion. Intrusion verification refers to detecting whether unwanted objects, personnel, or equipment have appeared in the work area, especially intrusion behaviors that may interfere with the operation of the unhooking robot. Specific steps include: static object detection, that is, detecting whether equipment in the work area has moved, or whether other static objects have behaved abnormally; dynamic intrusion detection, that is, detecting whether personnel or moving objects have entered the robot's work area, especially when the robot is performing unhooking tasks, as interference from other objects may affect the safety and efficiency of the operation. Intrusion detection is combined with time information to establish a time-series verification result. This result records whether an intrusion event occurred at a specific time point, the location of the intruded object, the intrusion time, and other information, reflecting the dynamic changes in the working environment and whether there are any conflicts with the expected operation.

[0060] Based on the timing verification results, if any anomaly is detected at any time, such as a person appearing in the robot's operating area or other equipment suddenly entering the workspace, posing a potential collision risk to the robot, an anomaly warning will be issued. To ensure safety, an emergency shutdown procedure will be initiated after the anomaly warning. At this time, the unhooking robot will immediately stop its current operation to avoid collision or equipment damage.

[0061] Furthermore, the control of the visual sensor and lidar to collect scene environment data and reconstruct the static unhooking scene includes:

[0062] A multi-scale pyramid coding system is established, and multi-scale feature extraction is performed on image data and radar signal data using the multi-scale pyramid coding system to establish multi-scale feature extraction results. The multi-scale feature extraction results are equipped with inter-scale correlation coding. The inter-scale correlation coding is used to perform feature recognition of the same signal source to establish dual-channel recognition results. The dual-channel recognition results are then fused using an attention mechanism to reconstruct the static unhooking scene.

[0063] Multi-scale pyramid coding is a technique that extracts features at different scales by processing image and radar signal data at multiple levels. In this process, image data and radar signal data are used to construct multi-level features through filtering and downsampling techniques at different scales. Specifically, the image is downsampled layer by layer through pyramid coding at different scales to obtain image representations at multiple scales. These scale images can capture the features of targets of different sizes and distances. The radar signal is also processed through multi-scale pyramid coding. Radar data at different scales helps to capture target information at different ranges, such as targets at long and short distances.

[0064] For each scale obtained through pyramid encoding, relevant feature information is extracted. These features include edges and textures in images, or object outlines in radar data. In this way, both detailed information and macroscopic structure of the environment can be taken into account, improving the ability to comprehensively perceive the target environment.

[0065] Based on the results of multi-scale feature extraction, features at different scales are associated through methods such as weighted summation and feature fusion. For example, convolutional neural networks are used for automated learning to establish inter-scale association codes. These codes are used to associate features at different scales, such as associating the spatial relationship between distant and near objects, thereby effectively fusing information at different scales and making the robot's perception capabilities more accurate.

[0066] Feature recognition is performed using inter-scale correlation coding, especially to find relevant features of the same signal source in images and radar signals. The purpose of this process is to determine features in image and radar data that are associated with the same object or target. For example, the correlation coding can be used to identify features of carriages or tracks in an image and match them with the corresponding target locations in the radar signal.

[0067] By identifying features from image and radar data, a dual-channel recognition system is established. The image channel uses convolutional neural networks or other deep learning models to process image features, obtaining information such as the object's spatial location and shape. The radar channel processes the echo signals from radar data to obtain spatial information and dimensions related to the target. These two channels process image and radar signals respectively and are effectively integrated through inter-scale correlation coding, ensuring that information about the same object from both data sources can be effectively combined.

[0068] Based on the dual-channel recognition results, an attention mechanism is introduced, enabling the model to dynamically adjust its focus according to the importance of different signal sources. The role of the attention mechanism is to automatically focus on the more important features of the same signal source while ignoring irrelevant or unimportant parts. Specifically, for the image channel, attention is paid to key features in the image data (such as target connection points, track boundaries, etc.) to enhance the feature information of important parts; for the radar channel, key targets in the radar data (such as the outline and position of the target object) are enhanced through the attention mechanism in order to find the most relevant information in multi-sensor data.

[0069] After adjusting the channel weights through the attention mechanism, the features of the image channel and the radar channel are fused. This includes assigning weights to each channel according to the attention mechanism, performing weighted averaging or weighted splicing to obtain more accurate fused features. After feature fusion is completed, the fused feature information is used to reconstruct the static unhooking scene. That is, the relative positions and spatial relationships of the key targets (such as carriages, connectors, etc.) required for the unhooking task are reconstructed in the scene to ensure that the robot can accurately perform the unhooking operation in the scene.

[0070] Furthermore, the process of performing channel fusion under an attention mechanism on the dual-channel recognition results to reconstruct the static de-hooking scene includes:

[0071] An uncertainty impact analysis is performed on the dual-channel recognition results for each feature channel to establish uncertainty-guided attention; the attention mechanism is reconstructed through the uncertainty-guided attention to complete the dual-channel fusion and reconstruct the static unhooking scene.

[0072] In the dual-channel recognition results, there are feature information from the image data channel and the radar signal channel, respectively. The feature information of each channel may have a certain degree of uncertainty. The sources include: image data may be affected by factors such as ambient lighting, occlusion, and resolution, which may lead to errors in the recognition of some features; radar signals may be affected by noise, echo interference, and the shape and distance of objects, which may lead to incomplete or erroneous target information.

[0073] To improve system accuracy, an uncertainty impact analysis is performed on the recognition results of each channel. Specifically, the confidence level of each channel feature is analyzed, identifying which features have high uncertainty and which have high reliability during the recognition process. Indicators such as variance, entropy, and confidence intervals are used to measure the degree of uncertainty for each channel feature. Based on the uncertainty impact analysis results, an uncertainty-guided focus system is established. For features with high uncertainty, their weight in subsequent processing is reduced to avoid over-reliance on unreliable features; for features with low uncertainty, their weight is increased, making subsequent feature fusion more reliant on these high-confidence features. This uncertainty-guided focus system enables the system to effectively handle uncertainty issues when fusing features from different channels, improving the overall system accuracy.

[0074] Based on the uncertainty-guided attention mechanism, different attention intensities are assigned to the features of each channel. For example, when the uncertainty of image data is high, the attention weight of the image channel is reduced, and more reliance is placed on the radar channel data; conversely, the weight of the image channel is increased. Building upon this optimized attention mechanism, dual-channel feature fusion is performed. That is, features from the image and radar are weighted and fused according to the attention intensity of each channel. The fused features combine the advantages of both channels and reduce interference from channels with higher uncertainty. Using the fused feature information, a static unhooking scene is finally reconstructed. In this process, the reconstructed scene accurately reflects the geometry of the robot's environment, the position and pose of the target object, and other key information.

[0075] Furthermore, the process of adjusting control parameters using the path-following compensation to complete the hook-unhooking and connection control of the unhooking robot includes:

[0076] Activate the connection sensing sensor of the unhooking robot to obtain the sensing force dataset of the contact point; use the sensing force dataset to identify the contact point state under time-series analysis and establish a contact point state identifier; use the contact point state identifier to verify the reliability of the unhooking connection.

[0077] During the unhooking operation, the unhooking robot needs to come into contact with the target object (such as a carriage connector or traction device). To ensure the accuracy and reliability of the connection, the connection sensing sensors on the robot are activated, including force sensors, pressure sensors or torque sensors. These sensors can accurately sense changes in the contact parts and provide real-time mechanical feedback.

[0078] When the robot comes into contact with the target connector or unhooking mating part, the connection sensing sensors begin to collect a dataset of sensing forces. This includes recording the force or pressure values ​​at the contact point when in contact with the target connection part; if the contact part requires rotation or twisting to complete the unhooking operation, the sensors will also record torque information related to the rotation. This sensing data will serve as the basis for subsequent operations to evaluate the contact state and the stability and reliability of performing the unhooking connection.

[0079] After acquiring the sensory force dataset of the contact points, time-series analysis is performed on this data. By analyzing the changes in contact force, pressure, and other data over time, the state of the contact points can be determined. For example, when the robot begins to contact the target component, the sensory data will show a sudden increase in force, indicating that the contact point has begun to be established. If the force between the robot and the target connector suddenly decreases, it indicates that the connection has become loose or separated. Analyzing whether the contact force is stable and whether the pressure changes uniformly reflects whether the connection is firm.

[0080] By analyzing the time sequence, the state of the contact points is identified, and contact point status identifiers are established based on this data. The status identifiers describe the current state of the contact points. Common states include no contact established, contact established, contact stable, contact unstable, and contact separated. These identifiers provide a reference for subsequent decisions and operations.

[0081] After the contact point status indicators are completed, these indicators are used to evaluate the reliability of the unhooking connection. Reliability verification includes determining whether the current unhooking connection is secure and whether it can safely complete the task. If the contact point status indicators show that the contact is stable and the mechanical data is within the normal range, the unhooking connection is considered reliable. If the status indicators show that the contact is unstable or separated, the unhooking connection is considered to be at risk and the action needs to be readjusted or the task stopped.

[0082] In summary, the multi-sensor fusion-based unhooking robot operation control method provided in this application has the following technical effects:

[0083] By fusing multiple sensors, including visual sensors and LiDAR, the robot can autonomously collect environmental data and reconstruct static unhooking scenarios. This provides reliable environmental perception for automated unhooking operations, reducing manual intervention and improving efficiency. Feature recognition of the static unhooking scenario accurately extracts track direction and unhooking coordination features, enabling path fitting for unhooking connections and generating a target reference path. This path fitting precisely considers track features and the geometric information of the target coordination points, providing high-precision path planning for the robot's unhooking actions. During unhooking operations, real-time incremental data acquisition updates key dynamic information in the static unhooking scenario. This update ensures the robot can adapt to environmental changes and adjust its operational path in real time, enhancing its adaptability in dynamic environments. Path consistency authentication and path-following compensation verify the accuracy of the robot's execution path in real time and dynamically compensate for deviations. This compensation is achieved through multiple balancing targets, effectively improving the robot's path stability and accuracy. Control parameter adjustments through path-following compensation ensure the motion control precision of the unhooking robot. The adjusted control parameters help the robot accurately complete the unhooking connection task, improving the success rate and stability of the operation. In summary, this method can effectively improve the accuracy, reliability, and flexibility of automated uncoupling operations, providing efficient and safe technical support for carriage connection and traction operations in railway transportation systems.

[0084] Example 2, based on the same inventive concept as the multi-sensor fusion unhooking robot operation control method in the foregoing examples, such as... Figure 2 As shown in the embodiment of this application, a multi-sensor fusion-based unhooking robot operation control system is provided, the system comprising:

[0085] The scene environment data acquisition module 10 is used to acquire scene environment data and reconstruct the static unhooking scene by taking the current time node as the time zero point after triggering the unhooking command; the unhooking connection path fitting module 20 is used to perform scene feature recognition on the static unhooking scene, and perform unhooking connection path fitting based on the track direction features and unhooking coordination features in the scene feature recognition results, establish a target reference path, and configure control parameters mapped to the target reference path; the scene data incremental acquisition module 30 is used to time the time zero point after executing the motion control of the unhooking robot with the control parameters, synchronously activate the vision sensor and lidar to perform scene data incremental acquisition, and update key dynamic information on the static unhooking scene according to the incremental acquisition results; the path consistency authentication module 40 is used to perform path consistency authentication based on the updated key dynamic information and the target reference path, and establish path following compensation, wherein the path following compensation is constructed through multiple balancing targets; the unhooking connection control module 50 is used to complete the unhooking connection control of the unhooking robot after adjusting the control parameters using the path following compensation.

[0086] Furthermore, the path consistency authentication module 40 is used to perform the following operation steps:

[0087] A consistency comparison channel is established, and the evaluation indicators of the consistency comparison channel include position deviation indicators and attitude deviation indicators; the updated key dynamic information and the target reference path are input into the consistency comparison channel to perform consistency deviation comparison and establish a consistency deviation identifier; environmental feature identification is performed based on the updated key dynamic information, and the adjustment weight of the consistency deviation identifier is updated with the environmental feature identification results; path following compensation is established according to the adjustment weight and the consistency deviation identifier.

[0088] Furthermore, the path consistency authentication module 40 is used to perform the following operation steps:

[0089] Obtain the deviation degree of the consistency deviation identifier, and establish a first balance target based on the deviation degree; create a distance attention level according to the target reference path, and after determining the current position with updated key dynamic information, match the distance attention level to establish a second balance target; establish path following compensation based on the first balance target, the second balance target, the adjusted weight, and the consistency deviation identifier.

[0090] Furthermore, the scene data incremental acquisition module 30 is used to perform the following operation steps:

[0091] The target mating parts and the unhooking mating parts are identified in the static unhooking scenario to establish key mating features; the trajectory of the unhooking mating parts is fitted according to the target reference path, and the trajectory fitting results are used to locate the trajectory-related key features in the static unhooking scenario; after the static unhooking scenario is set as a zero-point scenario, incremental acquisition results are established by focusing on the key mating features and trajectory-related key features.

[0092] Furthermore, the hook-unhooking connection path fitting module 20 is used to perform the following operation steps:

[0093] After performing contour recognition of the static unhooking scene using edge segmentation, a track candidate region is created based on the contour recognition results; after preprocessing the track candidate region, track features are extracted, and track direction features are created based on the track features.

[0094] Furthermore, the scene data incremental acquisition module 30 is used to perform the following operation steps:

[0095] The redundant cameras are activated synchronously, and spatial monitoring of the space is carried out using the redundant cameras to establish a spatial monitoring dataset. Spatial intrusion verification is performed on the spatial monitoring dataset to establish a time-series verification result. When any time-series verification result is an abnormal result, an abnormal warning is reported, and the unhooking robot is controlled to stop.

[0096] Furthermore, the scene environment data acquisition module 10 is used to perform the following operation steps:

[0097] A multi-scale pyramid coding system is established, and multi-scale feature extraction is performed on image data and radar signal data using the multi-scale pyramid coding system to establish multi-scale feature extraction results. The multi-scale feature extraction results are equipped with inter-scale correlation coding. The inter-scale correlation coding is used to perform feature recognition of the same signal source to establish dual-channel recognition results. The dual-channel recognition results are then fused using an attention mechanism to reconstruct the static unhooking scene.

[0098] Furthermore, the scene environment data acquisition module 10 is used to perform the following operation steps:

[0099] An uncertainty impact analysis is performed on the dual-channel recognition results for each feature channel to establish uncertainty-guided attention; the attention mechanism is reconstructed through the uncertainty-guided attention to complete the dual-channel fusion and reconstruct the static unhooking scene.

[0100] Furthermore, the hook-off connection control module 50 is used to perform the following operation steps:

[0101] Activate the connection sensing sensor of the unhooking robot to obtain the sensing force dataset of the contact point; use the sensing force dataset to identify the contact point state under time-series analysis and establish a contact point state identifier; use the contact point state identifier to verify the reliability of the unhooking connection.

[0102] Through the foregoing detailed description of the multi-sensor fusion unhooking robot operation control method, those skilled in the art can clearly understand the multi-sensor fusion unhooking robot operation control system in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be referred to the method section.

[0103] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-sensor fusion hook unhooking robot operation control method, characterized by, The method comprises: After triggering the hooking command, taking the current time node as the time zero point, controlling the visual sensor and the laser radar to collect scene environment data, and reconstructing the static hooking scene; Performing in-scene feature recognition on the static hooking scene, fitting the path of hooking connection based on the track direction features and hooking matching features in the in-scene feature recognition result, establishing a target reference path, and configuring control parameters mapped with the target reference path; After performing motion control of the hooking robot by using the control parameters, counting the time zero point, synchronously activating the visual sensor and the laser radar to collect incremental scene data, and updating key dynamic information on the static hooking scene according to the incremental collection result; Performing path consistency authentication according to the updated key dynamic information and the target reference path, and establishing path following compensation, wherein the path following compensation is constructed by multiple balance targets; After adjusting the control parameters by using the path following compensation, completing the hooking connection control of the hooking robot; The path consistency authentication according to the updated key dynamic information and the target reference path, and the establishment of the path following compensation, comprise: Establishing a consistency comparison channel, wherein the evaluation indexes of the consistency comparison channel include a position deviation index and a posture deviation index; Inputting the updated key dynamic information and the target reference path into the consistency comparison channel, performing consistency deviation comparison, and establishing a consistency deviation identifier; Performing environment feature recognition based on the updated key dynamic information, and updating the adjustment weight of the consistency deviation identifier according to the environment feature recognition result; Establishing the path following compensation according to the adjustment weight and the consistency deviation identifier; The establishment of the path following compensation according to the adjustment weight and the consistency deviation identifier, comprises: Obtaining a deviation degree of the consistency deviation identifier, and establishing a first balance target according to the deviation degree; After determining the current position by using the updated key dynamic information, matching the distance attention level to establish a second balance target; Establishing the path following compensation based on the first balance target, the second balance target, the adjustment weight, and the consistency deviation identifier.

2. The multi-sensor fusion unhooking robot work control method according to claim 1, characterized by, The synchronous activation of the visual sensor and the laser radar to collect incremental scene data, and the updating of the key dynamic information on the static hooking scene according to the incremental collection result, comprise: Performing feature recognition on the static hooking scene to establish matching key features; Performing track fitting on the hooking matching part according to the target reference path, and positioning the track-related key features in the static hooking scene by using the track fitting result; After setting the static hooking scene as a zero-point scene, establishing the incremental collection result by focusing on the matching key features and the track-related key features.

3. The multi-sensor fusion dislodging robot operation control method according to claim 1, wherein The in-scene feature recognition on the static hooking scene, comprises: After performing contour recognition on the static hooking scene by using edge segmentation, creating a track candidate area based on the contour recognition result; After preprocessing the track candidate area, extracting track features, and creating track direction features based on the track features.

4. The multi-sensor fusion dislodging robot work control method according to claim 1, characterized by, After the motion control of the hooking robot is performed according to the control parameter, the method comprises: Synchronously activating a redundant camera, and using the redundant camera to perform space monitoring on the matching space to establish a space monitoring data set; Performing space intrusion verification on the space monitoring data set to establish a time sequence verification result; When any time sequence verification result is an abnormal verification result, an abnormal early warning is given, and the hooking robot is controlled to stop.

5. The multi-sensor fusion dislodging robot work control method according to claim 1, wherein The control of the visual sensor and the laser radar to perform scene environment data collection and reconstruct a static hooking scene comprises: A multi-scale pyramid code is established, and multi-scale feature extraction is performed on image data and radar signal data respectively using the multi-scale pyramid code to establish multi-scale feature extraction results, wherein the multi-scale feature extraction results are provided with scale correlation coding; Feature recognition of the same signal source is performed using the scale correlation coding to establish a dual-channel recognition result; Channel fusion under an attention mechanism is performed on the dual-channel recognition result to reconstruct the static hooking scene.

6. The multi-sensor fusion dislodging robot work control method according to claim 5, wherein The channel fusion under the attention mechanism on the dual-channel recognition result to reconstruct the static hooking scene comprises: Uncertainty influence analysis of each feature channel is performed on the dual-channel recognition result to establish uncertainty guidance attention; The attention mechanism is reconstructed through the uncertainty guidance attention to complete dual-channel fusion to reconstruct the static hooking scene.

7. The multi-sensor fusion dislodging robot operation control method according to claim 1, wherein After the control parameter adjustment using the path following compensation, the hooking connection control of the hooking robot is completed, which comprises: The connection perception sensor of the hooking robot is activated to obtain a perception force data set of the contact point; Contact point state recognition under time sequence analysis is performed using the perception force data set to establish a contact point state identifier; Reliability verification of the hooking connection is performed using the contact point state identifier.

8. A multi-sensor fusion bale grab robot operation control system, characterized by, A hooking robot operation control method for implementing the multi-sensor fusion of any one of claims 1-7, the system comprising: A scene environment data collection module, configured to, after triggering a hooking command, take a current time node as a time zero point, control a visual sensor and a laser radar to perform scene environment data collection and reconstruct a static hooking scene; A hooking connection path fitting module, configured to perform scene feature recognition on the static hooking scene, perform path fitting of the hooking connection based on track direction features and hooking matching features in the scene feature recognition result, establish a target reference path, and configure a control parameter mapped with the target reference path; A scene data incremental collection module, configured to, after the motion control of the hooking robot is performed according to the control parameter, count the time zero point, synchronously activate the visual sensor and the laser radar to perform scene data incremental collection, and update key dynamic information on the static hooking scene according to the incremental collection result; A path consistency authentication module, configured to perform path consistency authentication according to the updated key dynamic information and the target reference path to establish a path following compensation, wherein the path following compensation is constructed through multiple balance targets; A hooking connection control module, configured to, after the control parameter adjustment using the path following compensation, complete the hooking connection control of the hooking robot.

Images