Multi-sensor fusion unhooking robot operation control method and system

Abstract

The invention provides a multi-sensor fusion unhooking robot operation control method and system, and relates to the technical field of traction devices.The method comprises the steps that after an unhooking command is triggered, a visual sensor and a laser radar are controlled to conduct scene environment data collection, and a static unhooking scene is reconstructed; feature recognition in the scene is carried out, path fitting of unhooking connection is carried out, and a target reference path is established; after motion control of the unhooking robot is executed, scene data increment collection is synchronously carried out, and key dynamic information is updated on a static unhooking scene; performing path consistency authentication, and establishing path following compensation; and after the control parameters are adjusted, unhooking connection control of the unhooking robot is completed. The technical problems that in the prior art, unhooking and connecting operations are mainly executed through a single sensor, and the single sensor cannot comprehensively sense environment information, so that deviation occurs in path planning and action execution, even faults or delay occurs, and the working efficiency and safety are affected are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traction devices, in particular to a multi-sensor fusion hook unhooking robot operation control method and system. BACKGROUND

[0002] In the railway transportation system, especially in the automation technology for car connection and traction operation, robot hook unhooking operation is a key technology to improve operation efficiency and reduce manual operation. However, the existing technology mainly performs hook unhooking and connection operation through a mechanical arm and a single sensor, which can complete basic tasks in a static environment, but lacks sufficient flexibility and precision when facing environmental changes or dynamic factors. Further, a single sensor may not be able to fully perceive environmental information, resulting in deviations in path planning and action execution, and even failures or delays, affecting operation efficiency and safety. Therefore, the existing technology still has obvious deficiencies in multi-sensor fusion and dynamic adaptation. SUMMARY

[0003] The present application provides a multi-sensor fusion hook unhooking robot operation control method and system, aiming to solve the technical problem that the existing technology mainly performs hook unhooking and connection operation through a single sensor, which cannot fully perceive environmental information, resulting in deviations in path planning and action execution, and even failures or delays, affecting operation efficiency and safety.

[0004] The first aspect of the present application provides a multi-sensor fusion hook unhooking robot operation control method, which comprises: after triggering a hook unhooking command, taking the current time node as a time zero point, controlling a vision sensor and a laser radar to collect scene environment data and reconstruct a static hook unhooking scene; performing in-scene feature recognition on the static hook unhooking scene, fitting a path for hook unhooking and connection based on track direction features and hook unhooking matching features in the in-scene feature recognition result, establishing a target reference path, and configuring control parameters mapped with the target reference path; after performing motion control of the hook unhooking robot with the control parameters, timing the time zero point, synchronously activating the vision sensor and the laser radar to collect incremental scene data, updating key dynamic information on the static hook unhooking scene according to the incremental collection result; performing path consistency authentication according to the updated key dynamic information and the target reference path, establishing path following compensation, wherein the path following compensation is constructed through multiple balance targets; after adjusting the control parameters using the path following compensation, completing hook unhooking and connection control of the hook unhooking robot.

[0005] In a second aspect, a multi-sensor fusion hooking robot operation control system is provided. The system is used in the multi-sensor fusion hooking robot operation control method. The system comprises a scene environment data acquisition module, which is used to acquire scene environment data after a hooking command is triggered, and to reconstruct a static hooking scene by taking a current time node as a time zero point and controlling a vision sensor and a laser radar to acquire scene environment data. A hooking connection path fitting module is used to identify features in the scene, fit a path of hooking connection based on track direction features and hooking matching features in the feature identification results, establish a target reference path, and configure control parameters mapped with the target reference path. A scene data incremental acquisition module is used to time the time zero point after the control parameters are used to perform motion control of the hooking robot, activate the vision sensor and the laser radar to acquire scene data incrementally, and update key dynamic information on the static hooking scene based on the incremental acquisition results. A path consistency authentication module is used to authenticate path consistency based on the updated key dynamic information and the target reference path, and to establish path following compensation, wherein the path following compensation is constructed by multiple balance targets. A hooking connection control module is used to perform control parameter adjustment using the path following compensation, and to complete hooking connection control of the hooking robot.

[0006] One or more technical solutions provided in the present application have at least the following beneficial effects: Through multi-sensor fusion of visual sensors and laser radars, environmental data can be autonomously collected and a static hooking scene can be reconstructed, which provides reliable environmental perception for automated hooking operations, and the automated process reduces manual operation and improves work efficiency; through feature recognition of the static hooking scene, the track direction features and hooking matching features can be accurately extracted, the path fitting of hooking connection is performed, and the target reference path is generated, and this path fitting accurately considers the geometric information of the track features and the target matching points, thereby providing high-precision path planning for the hooking action of the robot; during the execution of the hooking operation by the robot, the key dynamic information in the static hooking scene is updated through real-time incremental data collection, and the incremental data update ensures that the robot can adapt to environmental changes in real time and adjust the operation path, thereby enhancing the adaptability of the robot in a dynamic environment; through path consistency authentication to establish path following compensation, the accuracy of the robot in executing the path can be verified in real time, and dynamic compensation is performed according to the deviation, and this compensation is realized through multiple balance targets, thereby effectively improving the path stability and precision of the robot; through control parameter adjustment of the path following compensation, the motion control precision of the hooking robot is ensured, and the adjusted control parameters help the robot to accurately complete the hooking connection task, thereby improving the success rate and stability of the operation. In summary, this method can effectively improve the accuracy, reliability and flexibility of automated hooking operations, and provides efficient and safe technical support for the car connection and traction operation in the railway transportation system.

[0007] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the following specific embodiments of the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A multi-sensor fusion hooking robot operation control method flowchart is provided for the embodiments of the present application.

[0009] Figure 2 A multi-sensor fusion hooking robot operation control system structure diagram is provided for the embodiments of the present application.

[0010] Explanation of reference signs: scene environment data collection module 10, hooking connection path fitting module 20, scene data incremental collection module 30, path consistency authentication module 40, hooking connection control module 50. DETAILED DESCRIPTION

[0011] The embodiment of the application provides a multi-sensor fusion hooking robot operation control method and system, which solves the technical problem that the prior art mainly performs hooking and connecting operations through a single sensor, the single sensor cannot comprehensively perceive environmental information, thus causing deviation in path planning and action execution, even causing failure or delay, and affecting operation efficiency and safety.

[0012] After introducing the basic principles of the application, various non-limiting embodiments of the application will be specifically introduced below in combination with the drawings of the specification. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0013] Embodiment one, as shown in the figure, the embodiment of the application provides a multi-sensor fusion hooking robot operation control method, which comprises: Figure 1 After triggering the hooking command, the current time node is taken as the time zero point, and the visual sensor and the laser radar are controlled to collect scene environment data and reconstruct the static hooking scene.

[0014] In the automatic carriage connection or traction operation, once the hooking command is triggered, the hooking robot starts to execute the corresponding action, the triggering of the hooking command is the starting point of the whole task, which marks that the robot enters the working state. When the hooking command is triggered, the current time node is taken as the time zero point, which provides a reference for subsequent time synchronization and data processing, and all subsequent operations will be calibrated based on this time stamp to ensure the synchronous processing of data of different sensors.

[0015] At this time, the visual sensor and the laser radar are controlled to collect scene environment data, wherein the visual sensor, such as a camera, captures image data and provides two-dimensional information about the environment, helping the robot to identify the visual features of the carriage connection area; the laser radar (LiDAR) collects three-dimensional point cloud data of the environment, which helps to identify objects, structures, obstacles and other important information in the environment.

[0016] The environment data collected by the visual sensor and the laser radar reconstructs the static hooking scene, which means that based on the obtained data, a virtual model reflecting the current environment state is formed, and the model contains the information of static objects such as carriages, tracks and connectors, and provides a basis for subsequent path planning and control. Specifically, through multi-scale pyramid coding, image and radar data at different resolutions are processed to obtain information at different levels, and through scale correlation coding, information at different scales is associated to fuse the characteristics of multiple sensors to generate a more accurate environment model. Finally, through the attention mechanism of the dual-channel recognition result, important environmental features are preferentially paid attention to when fusing visual and laser radar data, thereby improving the effect of static scene reconstruction. ​

[0017] In the static hooking scene, scene features are identified, and based on the track direction features and hooking cooperation features in the scene feature identification results, the path fitting of hooking connection is performed, the target reference path is established, and the control parameters mapped with the target reference path are configured.

[0018] In the static hooking scene, scene features are identified, and based on the track direction features and hooking cooperation features in the scene feature identification results, the path fitting of hooking connection is performed, the target reference path is established, and the control parameters mapped with the target reference path are configured.

[0019] Based on the above feature identification results, a path is fitted for the hooking connection operation. In the path fitting process, algorithms based on feature identification are used, such as curve fitting and least squares method, to construct an accurate path model. This path is not just a simple trajectory, but a comprehensive path including position and attitude. After fitting the path, the target reference path is generated, which serves as a standard for subsequent actions and provides an ideal target for the robot's subsequent actions. Based on the target reference path, relevant control parameters are configured according to environmental features, path planning and task requirements. These control parameters include speed, acceleration, attitude adjustment parameters, etc., to ensure that the robot can efficiently and safely complete the hooking task in a dynamic environment.

[0020] After executing the motion control of the hooking robot with the control parameters, time zero is counted, and the vision sensor and laser radar are activated synchronously to perform incremental data collection. Based on the incremental collection results, key dynamic information is updated in the static hooking scene.

[0021] After executing the motion control of the hooking robot with the control parameters, time zero is counted, and the vision sensor and laser radar are activated synchronously to perform incremental data collection. Based on the incremental collection results, key dynamic information is updated in the static hooking scene.

[0022] During the execution of the task by the robot, the vision sensor and the laser radar continue to collect data. This time, the data collection is different from the initial environment modeling collection, but is incremental collection, that is, the sensor will only collect part of the data related to the motion process of the robot. Among them, the vision sensor captures the relative position changes between the car, the track and the robot and the target object; the laser radar provides three-dimensional point cloud data, which further helps the system to identify dynamic environmental features, such as the relative position between the robot and the target connector and the movement of any obstacles.

[0023] Based on the incremental acquisition results of vision and laser radar, the key dynamic information on the static hooking scene is updated, including the current position and posture of the robot, the relative changes between the target connection point, and any possible obstacles, dynamic objects or environmental changes (such as the movement of other carriages), so as to help the robot adjust the motion strategy in real time.

[0024] According to the updated key dynamic information and the target reference path, path consistency authentication is performed, and path following compensation is established, wherein the path following compensation is constructed by multiple balance targets.

[0025] When the robot performs the hooking task, it is monitored in real time whether its path is consistent with the target reference path, that is, it is ensured that the robot always moves according to the set target reference path when performing the path, and the path deviation is generated in real time by comparing the actual position of the robot and the target path. If the path deviation occurs, path following compensation needs to be established to make the robot return to the ideal path. Specifically, according to the updated dynamic information and the path deviation, a compensation path is calculated, which is constructed based on multiple balance targets, including position compensation targets, posture compensation targets, and speed compensation targets. The balance targets will integrate the deviation between the current state and the ideal state of the robot to calculate the optimal adjustment scheme. The specific path following compensation construction process is detailed in the subsequent steps.

[0026] After the control parameter adjustment using the path following compensation, the hooking connection control of the hooking robot is completed.

[0027] The control parameters of the robot are adjusted based on the path following compensation. These parameters are adjusted to optimize the motion performance of the robot and ensure that it can efficiently and accurately complete the hooking operation. After the path compensation is executed, the hooking robot is controlled to accurately perform the hooking connection operation according to the adjusted control parameters. At this time, the robot will accurately perform the connector docking and hooking operation according to the position and posture information on the target path. The hooking connection control not only includes the motion control of the robot, but also involves the contact and state monitoring of the target connector. Through accurate control, the robot can ensure the smooth completion of the hooking action.

[0028] Further, the path consistency authentication according to the updated key dynamic information and the target reference path to establish the path following compensation comprises: establish a consistency comparison channel, evaluation indexes of the consistency comparison channel including position deviation indexes and attitude deviation indexes; input the updated key dynamic information and the target reference path into the consistency comparison channel, perform consistency deviation comparison, and establish a consistency deviation identifier; identify environmental features based on the updated key dynamic information, update an adjustment weight of the consistency deviation identifier with an environmental feature identification result; and establish path following compensation according to the adjustment weight and the consistency deviation identifier.

[0029] To ensure that the robot always moves along the target reference path and adjusts the path in time, a consistency comparison channel is established, which compares the current path of the robot with the target reference path in real time and detects whether there is deviation. To quantify the deviation of the path and make effective correction, evaluation indexes of the channel are defined, including position deviation indexes and attitude deviation indexes. The position deviation indexes are used to measure the deviation of the current position of the robot from the target reference path. Generally, the position deviation refers to the distance between the current position of the robot and the target position in the target path, which can reflect whether the robot moves accurately along the predetermined path. The attitude deviation indexes are used to measure the difference between the attitude (such as the inclination angle, orientation, etc.) of the robot and the target path. The attitude deviation directly affects the positioning accuracy and operation accuracy of the robot, and especially in the hook picking operation, whether the attitude of the robot and the target connector is accurately connected is crucial.

[0030] When the robot performs the path, its dynamic information (such as position, attitude, speed, etc.) will be updated with the change of time and environment. At this time, the updated key dynamic information is obtained and compared with the pre-planned target reference path. The updated dynamic information and the target reference path are input into the consistency comparison channel to calculate the deviation between the current position and the target path. The key of this process is to use the defined position deviation indexes and attitude deviation indexes to calculate the deviation value according to the difference between the current state of the robot and the ideal state of the target path. If the position deviation is large, it indicates that the robot deviates from the target path and needs to be compensated in position. If the attitude deviation is large, it indicates that the attitude of the robot is not aligned with the target direction and needs to be compensated in attitude. According to the comparison result, a consistency deviation identifier is generated, which records the deviation information between the current path of the robot and the target path. This deviation identifier is used for subsequent path compensation and control adjustment to ensure the motion accuracy of the robot.

[0031] In the execution of the path, the robot will face various dynamic environmental changes, such as track shaking, environmental occlusion, etc., which may cause greater deviation in the path execution process, so it is necessary to analyze these influencing factors through environmental feature recognition, for example, through sensors to monitor the changes of the track, to judge whether the track is shaking or changing; through visual sensors, to identify whether the target is occluded, causing the robot's pose adjustment to become difficult.

[0032] According to the results of environmental feature recognition, the adjustment weight in the consistency deviation identifier is updated, which determines the importance of different types of deviations in the path compensation process. If the track shakes violently, the adjustment weight of the position deviation is increased, because the track shaking may cause the robot's position to change greatly, and more attention needs to be paid to position compensation to ensure that the robot can recover to the target path. If the target is occluded, the adjustment weight of the pose deviation is increased, because the occlusion of the target makes it difficult for the robot to accurately dock the target connector, and attention needs to be paid to pose compensation.

[0033] According to the consistency deviation identifier and the adjustment weight updated according to the foregoing steps, the path following compensation for the robot is established, and the purpose of the compensation is to correct the deviation between the robot and the target path, to ensure that the robot can always remain on the ideal path. For position deviation, according to the situation of track shaking or position change, position compensation measures are taken, including adjusting the travel route, speed or acceleration of the robot; for pose deviation, according to the situation of target occlusion or environmental interference, the pose of the robot is adjusted to ensure that the robot can dock the target at the correct angle during the hooking operation.

[0034] Further, the path following compensation established according to the adjustment weight and the consistency deviation identifier includes: Obtain the deviation degree of the consistency deviation identifier, and establish a first balance target based on the deviation degree; create a distance attention level based on the target reference path, and after determining the current position based on the updated key dynamic information, match the distance attention level to establish a second balance target; and establish the path following compensation based on the first balance target, the second balance target, the adjustment weight and the consistency deviation identifier.

[0035] The deviation degree of the consistency deviation identifier refers to the difference between the current path and the target reference path, which is represented as a position deviation and an attitude deviation. According to the deviation degree, a first balance target is constructed, which is used to adjust the position of the robot on the path. The size of the balance target increases with the increase of the deviation degree, that is, the greater the deviation degree, the greater the adjustment needs to be made. Based on the deviation degree, the adjustment amplitude is calculated using proportional control or gain adjustment as the first balance target, which is used to control the speed, acceleration and path adjustment amplitude of the robot. For example, if the robot deviates from the path far away, the adjustment intensity is increased so that the robot can return to the target path faster.

[0036] To optimize path execution, a distance attention level is created on the target reference path, which represents the distance between the current position of the robot and the end point of the target path, reflecting the urgency of path execution. The design of the distance attention level is based on the distance between the robot and the target end point or the position of the current path. For example, the closer to the target end point, the more attention is paid to the accuracy of the path.

[0037] After obtaining the latest dynamic information, the current position of the robot is updated, and the distance to the target end point is determined. Based on the distance between the current position of the robot and the target path end point, a second balance target is established. The purpose of this target is to make the path adjustment amplitude larger when the distance is closer, to ensure that the robot can quickly adjust to the correct path, and to make the path adjustment amplitude relatively smaller when the distance is farther away, which can increase the time step of path adjustment, so that the robot can execute the path more smoothly. Specifically, the calculation of the distance attention level can be realized by methods such as Euclidean distance and Manhattan distance, considering the distance between the robot and the target end point. The construction of the second balance target is based on the change of the distance, and a weighting method can be used to dynamically adjust the strength of the balance target. Through the second balance target, the path following behavior of the robot can be dynamically adjusted to ensure that larger adjustments are made when approaching the target, and smaller adjustments are made when moving away from the target.

[0038] In combination with the first balancing target (based on the degree of deviation) and the second balancing target (based on the distance attention level), the influence of both is integrated through weighting, and the goal is to determine the magnitude and frequency of adjustment, i.e., if the robot deviates from the path far, the first balancing target will occupy a larger weight, prompting the robot to quickly restore the path; if the robot is close to the target endpoint, the second balancing target will occupy a larger weight, ensuring that the robot makes more accurate adjustments. Also according to the updated adjustment weight, the balancing target is weighted, which means that if factors such as track shaking or target obstruction appear in the environment, the weight of the path compensation is adjusted accordingly, for example, if the track shakes violently, the adjustment weight of the position deviation will increase, so that more attention is paid to position adjustment. Finally, by combining the first balancing target, the second balancing target, the adjustment weight, and the consistency deviation identifier, the final path following compensation is calculated, which includes adjustments to the robot's path, speed, acceleration, etc., ensuring that the robot can efficiently and stably perform the hook picking operation in a dynamic environment.

[0039] Further, the synchronization activates the vision sensor and the laser radar to incrementally collect scene data, and updates key dynamic information on the static hook picking scene according to the incremental collection results, including: The features of the target mating part and the hook mating part are identified in the static hook picking scene, and the mating key features are established; the trajectory fitting of the hook mating part is performed according to the target reference path, and the trajectory associated key features are located in the static hook picking scene by using the trajectory fitting results; after the static hook picking scene is set as a zero point scene, the incremental collection results are established by attention collection of the mating key features and the trajectory associated key features.

[0040] In the static hook picking scene, the mating parts related to the hook picking operation are identified, which can be the car connector, the traction device, or other related parts, including the target mating part and the hook mating part, wherein the target mating part is like the connector or hook of the car, etc., which usually has a specific geometric shape (such as a circle, a square) or other unique appearance features; the hook mating part is the mating part of the hook picking device, such as the grabbing part, the shackle or the hook, which usually has a specific size, shape and relative position.

[0041] Through the environmental data collected by the vision sensor and the laser radar, the key features of these mating parts are identified through feature extraction algorithms such as edge detection, corner detection, image segmentation, etc., including geometric shape, texture, color, etc., which help the robot to locate and identify the target. Through feature recognition of the target mating part and the hook mating part, these features are combined into mating key features, which are used for subsequent path planning and action execution, for example, the robot can use these features to accurately determine the target position of the grabbing point, the connecting point, etc.

[0042] According to the target reference path, the trajectory fitting of the hooking matching part is performed. The purpose of trajectory fitting is to generate a trajectory consistent with the target reference path. This trajectory is used to guide the movement of the robot, ensuring that the robot can accurately reach the target position when performing the hooking operation. Trajectory fitting uses the starting point, end point, pose and other information provided by the target reference path, and uses least squares method, spline interpolation, curve fitting and other algorithms for smooth fitting of the path, ensuring that the path does not have sharp turns or mutations during execution. Through fitting, a smooth path is obtained. At the same time, this fitted trajectory takes into account the position, angle and other factors of the matching part, ensuring the accuracy of the trajectory. Using the trajectory fitting result, the trajectory associated key features are positioned in the static hooking scene. These key features are closely related elements of path fitting, including the accurate position, pose of the car body connecting part and the relationship with the path.

[0043] The static hooking scene is set as the zero point scene, which is the reference point for the robot to perform the operation, representing the starting position and pose of the robot. All subsequent operations and data collection will be based on this zero point, ensuring the accuracy and consistency of the data.

[0044] In the zero point scene, the matching key features and the trajectory associated key features are collected. Attention collection is a dynamic tracking and updating of these features to cope with changes in the field environment. Incremental collection updates the key features in the scene in real time based on sensors (including cameras, lidar, etc.). For matching key feature collection, the position change of the target matching part is continuously tracked to ensure accurate identification and grasping of the target part during path execution. For trajectory associated key feature collection, the relative position relationship between the path and the static scene is constantly updated as the robot moves along the trajectory to ensure the accuracy of the robot's movement in a dynamic environment.

[0045] Incremental collection result refers to the real-time updated environmental data of the robot through attention collection. These data contain key dynamic information related to path execution and operation. The incremental collection result will be used as the basis for the next path adjustment and control.

[0046] Further, the in-scene feature recognition of the static hooking scene comprises: After the contour recognition of the static hooking scene is performed using edge segmentation, a track candidate area is created based on the contour recognition result. After preprocessing the track candidate area, track features are extracted, and track direction features are created based on the track features.

[0047] In the static hooking scenario, in order to effectively identify the track and its surrounding environment, first, the edge segmentation is performed on the environment image or point cloud data, the edge segmentation is to identify the edge of the object by detecting the change of brightness or color in the image, and the significant boundary in the image is usually extracted by using an edge detection algorithm such as Canny edge detection. Specifically, in the static hooking scenario, the car compartment track, the connecting device and other important geometric shapes are identified, and the boundary contour (such as the outer contour of the car compartment, the edge of the track, etc.) extracted from the image or point cloud data through edge segmentation is taken as key information to help subsequent identification and processing.

[0048] After completing the edge segmentation, according to the segmented edge information, the region containing the track is extracted, which is called the track candidate region. The track candidate region is one or more regions representing the track formed by clustering the continuous edge regions in the edge detection result, and the track candidate region contains the shape, direction and spatial position of the track.

[0049] After generating the track candidate region, the candidate region is preprocessed to improve the extraction accuracy of the track features, and the preprocessing includes noise removal and geometric morphological repair. Further analysis is performed on the preprocessed track candidate region to extract track features, which are key information describing the shape and position of the track, including the geometric shape of the track, such as straight track, curved track, bending radius of the track, width of the track, etc., and the boundary of the track. After extracting the track features, the directionality of the track is further analyzed to generate the track direction feature. Specifically, the orientation of the track is determined by calculating the normal direction, tangent direction of the track boundary or by fitting the curve of the track; if the track is a curve, the direction information of the curve is generated based on the bending characteristics of the track to ensure that the robot can accurately adapt to the curve of the track.

[0050] Further, after executing the motion control of the hooking robot with the control parameters, the method comprises: Synchronously activating the redundant camera, using the redundant camera to perform spatial monitoring of the cooperation space, establishing a spatial monitoring data set; performing spatial intrusion verification on the spatial monitoring data set, establishing a time sequence verification result; when any time sequence verification result is an abnormal verification result, an abnormal early warning is issued, and the hooking robot is controlled to stop.

[0051] In order to save computing power and improve concentration, only key dynamic features are recognized in the foregoing, and the environment is not monitored, such as people, abnormal equipment interference, etc. In actual hook picking operations, in order to improve the safety and stability of the operation, in addition to the main visual sensor, redundant cameras are also used to provide additional visual information, thereby avoiding the visual angle blind area of a single camera. The spatial monitoring data collected by the redundant cameras is summarized and stored as a spatial monitoring data set for subsequent analysis and verification.

[0052] The spatial monitoring data collected by the redundant cameras is analyzed to verify whether the space is invaded abnormally. The invasion verification refers to detecting whether an unexpected object, person or equipment appears in the working area, especially the invasion behavior that may interfere with the hook picking robot operation. The specific steps include: static object detection, that is, detecting whether the equipment in the working area moves or other static objects abnormally; dynamic intrusion detection, that is, detecting whether a person or a moving object enters the working area of the robot, especially when the robot performs the hook picking task, the interference of other objects may affect the safety and efficiency of the operation process. The invasion detection is combined with time information to establish a time sequence verification result. The result records whether an invasion event occurs at a specific time point, the position of the invasion object, the invasion time and other information, reflecting the dynamic changes of the operation environment and whether there is a conflict with the expected operation.

[0053] According to the time sequence verification result, if any abnormality is detected at any time, for example, a person appears in the robot operation area or other equipment suddenly enters the operation space, causing the robot to face potential collision risks, an abnormality warning is given. In order to ensure safety, an emergency shutdown program is started after the abnormality warning, at which time the hook picking robot will immediately stop the current operation to avoid collision or damage to equipment.

[0054] Further, the control of the visual sensor and the laser radar to collect scene environment data and reconstruct the static hook picking scene includes: A multi-scale pyramid code is established, and multi-scale feature extraction of image data and radar signal data is performed using the multi-scale pyramid code to establish a multi-scale feature extraction result, wherein the multi-scale feature extraction result is provided with an inter-scale association code; the inter-scale association code is used for feature recognition of the same signal source to establish a dual-channel recognition result; and the dual-channel recognition result is fused under an attention mechanism to reconstruct the static hook picking scene.

[0055] Multi-scale pyramid coding is a technique that extracts features of different scales by processing image and radar signal data in multiple layers. In this process, image data and radar signal data are processed through different scale filtering and downsampling techniques to construct multi-level features. The image is processed through pyramid coding of different scales, which first downsamples the original image layer by layer to obtain multiple scale image representations. These scale images can capture target features of different sizes and distances. Radar signals are also processed through multi-scale pyramid coding techniques. Different scale radar data helps to capture target information in different ranges, such as distant and close-range objects.

[0056] For each scale obtained through pyramid coding, relevant feature information is extracted, including edges, textures in images, or object outlines in radar data. In this way, both detailed information and macroscopic structure of the environment can be considered, improving the overall perception ability of the target environment.

[0057] Based on the multi-scale feature extraction results, different scale features are associated through weighted summation, feature fusion, etc. For example, a convolutional neural network is used for automated learning to establish scale-related coding. This coding is used to associate features of different scales, such as the spatial relationship between distant objects and close-range objects, effectively fusing information of different scales to make the robot's perception more accurate.

[0058] Feature recognition is performed using scale-related coding, especially finding related features of the same signal source in images and radar signals. The purpose of this process is to determine the features associated with the same object or target in image and radar data. For example, through association coding, the features of the carriages or tracks in the image are identified and matched with the corresponding target positions in the radar signal.

[0059] Through feature recognition of image and radar data, a dual-channel recognition result is established. Image channel recognition processes image features through a convolutional neural network or other deep learning models to obtain spatial position, shape, and other information of objects. Radar channel recognition processes radar data echo signals to obtain spatial information, size, and other features related to targets. The two channels process image and radar signals respectively and are effectively connected through scale-related coding to ensure that information about the same object in the two data sources can be effectively combined.

[0060] On the basis of the dual-channel recognition result, an attention mechanism is introduced, so that the model can dynamically adjust its focus according to the importance of different signal sources. The role of the attention mechanism is to automatically focus on more important features for different features of the same signal source, while ignoring irrelevant or unimportant parts. For the image channel, attention is paid to key features in the image data (such as target connection points, track boundaries, etc.), enhancing the feature information of important parts. For the radar channel, the key targets in the radar data (such as the outline and position of the target object) are strengthened through the attention mechanism to find the most relevant information in the multi-sensor data.

[0061] After adjusting the channel weights through the attention mechanism, the features of the image channel and the radar channel are fused, including assigning weights to each channel according to the attention mechanism, performing weighted averaging or weighted splicing, and obtaining more accurate fused features. After feature fusion, the static hooking scene is reconstructed using the fused feature information, i.e., the relative position and spatial relationship of the key targets (such as the car body and connector) required for the hooking task are reconstructed in the scene, ensuring that the robot can accurately perform the hooking task in the scene.

[0062] Further, the channel fusion of the dual-channel recognition result under the attention mechanism and the reconstruction of the static hooking scene include: Each feature channel of the dual-channel recognition result is analyzed for uncertainty influence, and an uncertainty-guided attention is established. The attention mechanism is reconstructed through the uncertainty-guided attention to complete dual-channel fusion to reconstruct the static hooking scene.

[0063] In the dual-channel recognition result, there are feature information from the image data channel and the radar signal channel respectively. Each channel's feature information may have some uncertainty, which comes from: the image data may be affected by environmental light, shielding, resolution, etc., leading to errors in part of the feature recognition; the radar signal is affected by noise, echo interference, and object shape and distance, which may lead to incomplete or erroneous target information.

[0064] To improve the accuracy of the system, uncertainty influence analysis is performed on the recognition result of each channel. Specifically, the confidence of each channel feature is analyzed, that is, which features have high uncertainty in the recognition process and which features have high reliability. Indicators such as variance, entropy, confidence interval, etc. are used to measure the uncertainty degree of each channel feature. According to the uncertainty influence analysis result, uncertainty guided attention is established, that is, for features with high uncertainty, reduce their weight in subsequent processing to avoid over-reliance on unreliable features; for features with low uncertainty, increase their weight so that subsequent feature fusion relies more on these high-confidence features. This uncertainty guided attention enables the system to effectively handle uncertainty when fusing features from different channels, improving the overall system accuracy.

[0065] According to the results of uncertainty guided attention, different attention strengths are given to the features of each channel. For example, when the uncertainty of image data is high, the attention weight of the image channel is reduced and more reliance is placed on radar channel data; conversely, the weight of the image channel is increased. Based on the optimized attention mechanism, dual-channel feature fusion is performed, that is, according to the attention strength of each channel, the features from the image and radar are weighted and fused. The fused features will integrate the advantages from both channels and reduce the interference from channels with high uncertainty. Using the fused feature information, the static hooking scene is finally reconstructed. In this process, the reconstructed scene can accurately reflect the key information of the robot's environment, such as the geometry, position, and pose of the target object.

[0066] Further, after the control parameter adjustment using the path following compensation, the hooking robot hooking connection control is completed, including: The connection perception sensor of the hooking robot is activated to obtain a set of perception force data of the contact point. The contact point state is identified through time series analysis using the set of perception force data, and a contact point state identifier is established. The reliability of the hooking connection is verified using the contact point state identifier.

[0067] During the hooking operation, the hooking robot needs to contact the target object (such as a car connector or a traction device). To ensure the accuracy and reliability of the connection, the connection perception sensors on the robot are activated, including force sensors, pressure sensors, or torque sensors. These sensors can accurately perceive the changes in the contact components and provide real-time mechanical feedback.

[0068] When the robot contacts the target connector or the hooking fitting, the connection perception sensor starts to collect the perception force data set, including the force or pressure value of the contact point recorded by the sensor when contacting the target connection component; if the contact component needs to be rotated or twisted to complete the hooking operation, the sensor will also record the torque information related to rotation. These perception data will serve as the basis for subsequent operations to assess the contact state, the stability and reliability of the hooking connection.

[0069] After obtaining the perception force data set of the contact point, time series analysis is performed on these data, and by analyzing the changes of contact force, pressure and other data on the time axis, the state of the contact point can be judged, for example, when the robot starts to contact the target component, the perception data will show a sudden increase in force, indicating that the contact point has begun to establish; if the force value between the robot and the target connector suddenly decreases, it indicates that the connection has loosened or separated; analyzing whether the contact force is stable and the pressure changes uniformly reflects whether the connection is firm.

[0070] Through time series analysis, the state of the contact point is identified, and a contact point state identifier is established according to these data. The state identifier describes the current state of the contact point. Common states include contact not established, contact established, contact stable, contact unstable, and contact separated. These identifiers provide a reference for subsequent decision-making and operations.

[0071] After the contact point state identifier is completed, the identifiers are used to evaluate the reliability of the hooking connection. Reliability verification includes determining whether the current hooking connection is firm and whether it can safely complete the task. If the contact point state identifier indicates that the contact is stable and the mechanical data is within the normal range, the hooking connection is considered reliable. If the state identifier indicates that the contact is unstable or separated, the hooking connection is considered risky and needs to be adjusted or stopped.

[0072] In summary, the multi-sensor fusion hooking robot operation control method provided by the embodiments of the present application has the following technical effects: By fusing multiple sensors, including visual sensors and LiDAR, the robot can autonomously collect environmental data and reconstruct static unhooking scenarios. This provides reliable environmental perception for automated unhooking operations, reducing manual intervention and improving efficiency. Feature recognition of the static unhooking scenario accurately extracts track direction and unhooking coordination features, enabling path fitting for unhooking connections and generating a target reference path. This path fitting precisely considers track features and the geometric information of the target coordination points, providing high-precision path planning for the robot's unhooking actions. During unhooking operations, real-time incremental data acquisition updates key dynamic information in the static unhooking scenario. This update ensures the robot can adapt to environmental changes and adjust its operational path in real time, enhancing its adaptability in dynamic environments. Path consistency authentication and path-following compensation verify the accuracy of the robot's execution path in real time and dynamically compensate for deviations. This compensation is achieved through multiple balancing targets, effectively improving the robot's path stability and accuracy. Control parameter adjustments through path-following compensation ensure the motion control precision of the unhooking robot. The adjusted control parameters help the robot accurately complete the unhooking connection task, improving the success rate and stability of the operation. In summary, this method can effectively improve the accuracy, reliability, and flexibility of automated uncoupling operations, providing efficient and safe technical support for carriage connection and traction operations in railway transportation systems.

[0073] Example 2, based on the same inventive concept as the multi-sensor fusion unhooking robot operation control method in the foregoing examples, such as... Figure 2 As shown in the embodiment of this application, a multi-sensor fusion-based unhooking robot operation control system is provided, the system comprising: The scene environment data collection module 10 is configured to, after triggering the hooking command, take a current time node as a time zero point, control a visual sensor and a laser radar to collect scene environment data, and reconstruct a static hooking scene; the hooking connection path fitting module 20 is configured to perform in-scene feature recognition on the static hooking scene, perform path fitting of hooking connection based on track direction features and hooking matching features in the in-scene feature recognition result, establish a target reference path, and configure control parameters mapped with the target reference path; the scene data incremental collection module 30 is configured to, after performing motion control of the hooking robot by using the control parameters, count time from the time zero point, synchronously activate the visual sensor and the laser radar to collect scene data incrementally, and update key dynamic information on the static hooking scene according to an incremental collection result; the path consistency authentication module 40 is configured to perform path consistency authentication according to the updated key dynamic information and the target reference path, and establish path following compensation, wherein the path following compensation is constructed by using a plurality of balance targets; and the hooking connection control module 50 is configured to, after adjusting the control parameters by using the path following compensation, complete hooking connection control of the hooking robot.

[0074] Further, the path consistency authentication module 40 is configured to perform the following operation steps: establish a consistency comparison channel, evaluation indexes of the consistency comparison channel include a position deviation index and a posture deviation index; input the updated key dynamic information and the target reference path into the consistency comparison channel, perform consistency deviation comparison, and establish a consistency deviation identifier; perform environment feature recognition based on the updated key dynamic information, update an adjustment weight of the consistency deviation identifier according to an environment feature recognition result; and establish path following compensation according to the adjustment weight, the consistency deviation identifier.

[0075] Further, the path consistency authentication module 40 is configured to perform the following operation steps: obtain a deviation degree of the consistency deviation identifier, establish a first balance target according to the deviation degree; create a distance attention level according to the target reference path, and after determining a current position by using the updated key dynamic information, match the distance attention level to establish a second balance target; and establish path following compensation based on the first balance target, the second balance target, the adjustment weight, and the consistency deviation identifier.

[0076] Further, the scene data incremental collection module 30 is configured to perform the following operation steps: The feature recognition of the target matching part and the hook matching part is performed on the static hooking scene, and the matching key features are established; the trajectory fitting of the hook matching part is performed according to the target reference path, and the trajectory correlation key features are located in the static hooking scene by using the trajectory fitting result; after the static hooking scene is set as a zero point scene, the incremental collection result is established by collecting the attention of the matching key features and the trajectory correlation key features.

[0077] Further, the hook connection path fitting module 20 is configured to perform the following operation steps: After the contour recognition of the static hooking scene is performed by using edge segmentation, the contour recognition result is used to create a track candidate area; after the track candidate area is preprocessed, the track features are extracted, and the track direction features are created based on the track features.

[0078] Further, the scene data incremental collection module 30 is configured to perform the following operation steps: The redundant camera is activated synchronously, the space monitoring of the matching space is performed by using the redundant camera, and the space monitoring data set is established; the space intrusion verification is performed on the space monitoring data set, and the time sequence verification result is established; when any time sequence verification result is a verification abnormal result, an abnormal early warning is given, and the hooking robot is controlled to stop.

[0079] Further, the scene environment data collection module 10 is configured to perform the following operation steps: A multi-scale pyramid code is established, and multi-scale feature extraction of image data and radar signal data is performed by using the multi-scale pyramid code, and a multi-scale feature extraction result is established, wherein the multi-scale feature extraction result is provided with scale correlation coding; the feature recognition of the same signal source is performed by using the scale correlation coding, and a dual-channel recognition result is established; the dual-channel recognition result is fused under the attention mechanism, and the static hooking scene is reconstructed.

[0080] Further, the scene environment data collection module 10 is configured to perform the following operation steps: The uncertainty influence analysis of each feature channel is performed on the dual-channel recognition result, and the uncertainty guidance attention is established; the attention mechanism is reconstructed by the uncertainty guidance attention, the dual-channel fusion is completed, and the static hooking scene is reconstructed.

[0081] Further, the hooking connection control module 50 is configured to perform the following operation steps: The connection perception sensor of the hooking robot is activated, and the perception force data set of the contact point is acquired; the contact point state recognition is performed by using the perception force data set under time sequence analysis, and the contact point state identification is established; the reliability verification of the hooking connection is performed by using the contact point state identification.

[0082] The multi-sensor fusion hooking robot operation control method is described in detail in the foregoing description, and those skilled in the art can clearly understand the multi-sensor fusion hooking robot operation control system in the embodiment. Since the system corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the method part.

[0083] The above description of disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-sensor fusion-based operation control method for a hook-removing robot, characterized in that, The method includes: After triggering the unhooking command, the current time node is taken as the zero point of time, and the vision sensor and lidar are controlled to collect scene environment data and reconstruct the static unhooking scene. The static unhooking scenario is subjected to scene feature recognition. Based on the track direction features and unhooking coordination features in the scene feature recognition results, the unhooking connection path is fitted, a target reference path is established, and control parameters mapped to the target reference path are configured. After executing the motion control of the unhooking robot with the control parameters, the time is timed at zero point, and the vision sensor and lidar are activated simultaneously to perform incremental scene data acquisition. Based on the incremental acquisition results, key dynamic information is updated on the static unhooking scene. Path consistency authentication is performed based on the updated key dynamic information and the target reference path, and path following compensation is established, wherein the path following compensation is constructed through multiple balancing objectives; After adjusting the control parameters using the path following compensation, the hook unhooking and connection control of the unhooking robot is completed.

2. The multi-sensor fusion-based unhooking robot operation control method as described in claim 1, characterized in that, The step of performing path consistency authentication based on updated key dynamic information and the target reference path, and establishing path following compensation, includes: A consistency comparison channel is established, and the evaluation indicators of the consistency comparison channel include position deviation index and attitude deviation index; The updated key dynamic information and the target reference path are input into the consistency comparison channel to perform consistency deviation comparison and establish a consistency deviation identifier. Environmental feature identification is performed based on updated key dynamic information, and the adjustment weight of consistency deviation identifier is updated based on the environmental feature identification results. Path following compensation is established based on the adjusted weights and the consistency deviation identifier.

3. The multi-sensor fusion-based unhooking robot operation control method as described in claim 2, characterized in that, The step of establishing path following compensation based on the adjusted weights and the consistency deviation identifier includes: Obtain the deviation degree of the consistency deviation indicator, and establish a first balance target based on the deviation degree; A distance attention level is created based on the target reference path. After determining the current position with updated key dynamic information, a second balanced target is established by matching the distance attention level. Path following compensation is established based on the first balance objective, the second balance objective, the adjusted weights, and the consistency deviation identifier.

4. The multi-sensor fusion-based unhooking robot operation control method as described in claim 1, characterized in that, The simultaneous activation of the visual sensor and LiDAR for incremental scene data acquisition, and the updating of key dynamic information on the static unhooking scene based on the incremental acquisition results, including: For the static unhooking scenario, feature identification of the target mating parts and the unhooking mating parts is performed to establish key mating features; The trajectory of the unhooking assembly is fitted according to the target reference path, and the trajectory fitting results are used to locate the key trajectory-related features in the static unhooking scenario. After setting the static unhooking scenario as a zero-point scenario, incremental acquisition results are established by focusing on key features of cooperation and key features of trajectory association.

5. The multi-sensor fusion-based unhooking robot operation control method as described in claim 1, characterized in that, The in-scene feature recognition of the static unhooking scene includes: After performing contour recognition of the static unhooking scene using edge segmentation, a track candidate region is created based on the contour recognition results; After preprocessing the candidate orbit regions, orbit features are extracted, and orbit direction features are created based on the orbit features.

6. The multi-sensor fusion-based unhooking robot operation control method as described in claim 1, characterized in that, After executing the motion control of the unhooking robot with the control parameters, the process includes: Simultaneously activate redundant cameras, utilize the redundant cameras for spatial monitoring of the coordinated space, and establish a spatial monitoring dataset; Spatial intrusion verification is performed on the aforementioned spatial monitoring dataset, and time-series verification results are established; If any timing verification result is an abnormal result, an abnormal warning will be issued, and the unhooking robot will be stopped.

7. The multi-sensor fusion-based operation control method for a hook-unhooking robot as described in claim 1, characterized in that, The control vision sensor and lidar are used to collect scene environment data and reconstruct the static unhooking scene, including: A multi-scale pyramid coding is established, and multi-scale feature extraction is performed on image data and radar signal data using the multi-scale pyramid coding. A multi-scale feature extraction result is established, wherein the multi-scale feature extraction result is equipped with inter-scale correlation coding. The inter-scale correlation coding is used to perform feature recognition of the same signal source, and a dual-channel recognition result is established. The dual-channel recognition results are fused using an attention mechanism to reconstruct the static unhooking scene.

8. The multi-sensor fusion-based unhooking robot operation control method as described in claim 7, characterized in that, The process of performing channel fusion under an attention mechanism on the dual-channel recognition results to reconstruct the static hook removal scene includes: An uncertainty impact analysis is performed on the dual-channel recognition results for each feature channel to establish uncertainty-guided attention. By guiding attention to reconstruct the attention mechanism through the aforementioned uncertainty, dual-channel fusion is completed to reconstruct the static unhooking scenario.

9. The multi-sensor fusion-based operation control method for a hook-unhooking robot as described in claim 1, characterized in that, After adjusting the control parameters using the path following compensation, the unhooking and connection control of the unhooking robot is completed, including: Activate the connection sensing sensors of the unhooking robot to obtain the sensing force dataset of the contact point; Using the aforementioned sensing dataset, contact point status identification is performed under time-series analysis to establish contact point status identifiers. The reliability of the unhooking connection is verified using the contact point status indicator.

10. A multi-sensor fusion-based unhooking robot operation control system, characterized in that, The system is used to implement the multi-sensor fusion operation control method for a hook-removing robot according to any one of claims 1-9, the system comprising: The scene environment data acquisition module is used to control the vision sensor and lidar to acquire scene environment data and reconstruct the static unhooking scene after the unhooking command is triggered, taking the current time node as the time zero point. The hook-and-unhooking path fitting module is used to perform scene feature recognition on the static hook-and-unhooking scene, and to perform hook-and-unhooking path fitting based on the track direction features and hook-and-unhooking coordination features in the scene feature recognition results, to establish a target reference path, and to configure control parameters that are mapped to the target reference path. The scene data incremental acquisition module is used to execute the motion control of the unhooking robot with the control parameters, time the zero point, and simultaneously activate the vision sensor and lidar to perform scene data incremental acquisition, and update key dynamic information on the static unhooking scene according to the incremental acquisition results. The path consistency authentication module is used to perform path consistency authentication based on the updated key dynamic information and the target reference path, and to establish path following compensation, wherein the path following compensation is constructed through multiple balancing targets; The hook-unhooking and connection control module is used to adjust the control parameters using the path following compensation to complete the hook-unhooking and connection control of the hook-unhooking robot.

Images