A method and system for intelligently identifying car couplers by a car unhooking and hooking robot

Abstract

The application discloses a kind of based on unhooking and hooking robot car dumper intelligent identification car hook method and system, it is related to intelligent identification technical field, including, acquisition multi-modal data, pre-processing is carried out, the multi-modal data includes laser radar point cloud data, camera image and infrared data;Multi-modal data after processing is fused, adopts convolutional neural network and pose estimation algorithm to identify car hook information, generates car hook identification result;Optimal capture path is generated by using path planning algorithm, to the capture point path of car hook, avoid interference and optimize capture path, calculate;Robot moves to car hook position according to optimal capture path, utilizes image perception real-time correction capture posture, drives mechanical arm end effector to hold car hook and capture.The application is fused by multi-modal data and path planning, accurately identifies car hook and optimizes capture path, improves the capture precision and stability of robot in complex environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent identification technology, and in particular to an intelligent identification method and system for tippers based on a hook-and-unhook robot. Background Technology

[0002] In railway transportation, tipplers, as key equipment for rapid loading and unloading of vehicles, are widely used in the logistics of bulk materials such as coal and ore. Traditional tippler operations mainly rely on manual verification of the vehicle coupler status. Operators must manually judge and operate the couplers during the coupling and uncoupling process, resulting in low efficiency, high labor intensity, and high operational risks. To improve operational efficiency, some existing technologies have incorporated sensing methods such as camera image recognition and lidar ranging, combined with image processing algorithms to identify and assist in judging the coupler status, achieving preliminary semi-automated operation. However, due to the complex structure of the couplers, drastic changes in ambient lighting, and the similarity between the car body and background color, the recognition accuracy based on single-modal data remains significantly limited, and the lack of dynamic adjustment capabilities makes it difficult to meet the requirements of fully automated coupling and uncoupling operations.

[0003] Existing coupler identification methods mostly employ single-modal sensing approaches, such as visual detection based on camera images, contour recognition using LiDAR, or edge extraction from infrared thermal data. While these methods can be effective under specific conditions, they suffer from poor robustness, incomplete perception information, and susceptibility to environmental interference in complex real-world applications. Furthermore, most current methods lack efficient data fusion strategies and intelligent path planning capabilities, resulting in inflexible adjustments to the coupling / uncoupling actions based on the actual scenario. This leads to limited robotic arm execution trajectories, higher error rates, and an inability to meet the demands for intelligent and highly reliable coupler coupling / uncoupling operations. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent coupler identification method for tippers based on a hook-unhooking robot to solve the problem of inaccurate coupler identification during the hook-unhooking process of tippers in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an intelligent method for identifying car couplers in a tipper based on a hook-and-unhook robot. The method includes: collecting multimodal data and preprocessing it, wherein the multimodal data includes lidar point cloud data, camera images, and infrared data; fusing the processed multimodal data and using a convolutional neural network and attitude estimation algorithm to identify car coupler information and generate a car coupler identification result; employing a path planning algorithm to calculate the path to the car coupler's gripping point, avoiding interference and optimizing the gripping path to generate an optimal gripping path; the robot moving to the car coupler position according to the optimal gripping path, using image perception to correct the gripping posture in real time, and driving the robotic arm end effector to grip the car coupler; the robot moving the car coupler to the docking position along a predetermined trajectory, adjusting the docking posture in real time, and driving the robotic arm end effector to release the car coupler for reconnection.

[0007] As a preferred embodiment of the intelligent coupler identification method for tippers based on a decoupling robot described in this invention, the specific steps for collecting multimodal data and performing preprocessing are as follows: Outlier removal is performed on the lidar point cloud data using statistical filtering to obtain the lidar point cloud data after removal. The camera image is geometrically corrected using a fisheye distortion correction algorithm to obtain the corrected camera image. In addition, the infrared data is smoothed and thermal feature enhanced by combining Gaussian blur and local contrast enhancement methods to obtain the smoothed infrared image. The removed lidar point cloud data is used to perform plane fitting and target extraction using the RANSAC algorithm to separate the ground and coupler-related target areas and generate spatial structure feature data. The Retinex algorithm is used to enhance the brightness and contrast of the corrected camera image, generating a brightness-enhanced image. For the smoothed infrared image, the Canny edge detection algorithm is used to extract the temperature change boundary and generate an infrared thermal boundary feature map; By using a feature-level fusion method, spatial structural feature data, brightness-enhanced images, and infrared thermal boundary feature maps are integrated in a unified manner to construct a multimodal feature dataset.

[0008] As a preferred embodiment of the intelligent coupler identification method for tippers based on a decoupling robot described in this invention, the specific steps for generating the coupler identification result are as follows: Using spatial registration techniques and image registration algorithms, feature-level fusion is performed on a multimodal feature dataset to generate a fused multimodal feature tensor; The fused multimodal feature tensor is used to extract coupler feature information and identify the location using a multi-channel convolutional neural network to obtain candidate regions and preliminary results. Based on the candidate coupler region, the attitude estimation algorithm is applied to extract point coordinates and spatial orientation information, calculate the three-dimensional position and attitude of the coupler, and generate coupler recognition results.

[0009] As a preferred embodiment of the intelligent coupler identification method for tippers based on a decoupling robot described in this invention, the following steps are taken: The fused multimodal feature tensor is used with a multi-channel convolutional neural network to extract coupler feature information and identify its location, thereby obtaining candidate regions and preliminary results. The fused multimodal feature tensor is processed by multi-level convolution using a multi-channel convolutional neural network to extract the edge shape features of the image and obtain the edge contour feature map. Surface texture information is extracted from the edge shape feature map to generate a texture enhancement feature map; By fusing depth channel information, spatial depth distribution is extracted to generate a spatial positioning feature map; Thermal boundary extraction is performed by combining the infrared channel with the spatial positioning feature map to highlight the temperature difference area of ​​the coupler and generate a thermal sensing feature map. The region proposal method is used to analyze the thermal sensing feature map and the spatial positioning feature map, calculate the confidence of the coupler position, and mark the candidate coupler region in the image to obtain the preliminary recognition results.

[0010] As a preferred embodiment of the intelligent identification method for car couplers in a tipper based on a decoupling robot described in this invention, the specific steps for generating the optimal grasping path are as follows: Based on the coupler recognition results, an environmental map is constructed and potential interference objects in the grasping area are identified to obtain a preliminary grasping environment map; Based on the coupler position and interference distribution in the preliminary grasping environment map, a feasible grasping path from the current position to the target grasping point is constructed, and the preliminary grasping path is calculated using the fast random tree algorithm. By combining the joint structure and motion characteristics of the robotic arm, the initial grasping path is optimized, the motion sequence of each joint is adjusted, and the optimal grasping path is generated.

[0011] As a preferred embodiment of the intelligent coupler identification method for tippers based on a decoupling robot described in this invention, the robot moves to the coupler position according to the optimal grasping path, corrects the grasping posture in real time using image perception, and drives the end effector of the robotic arm to grip the coupler for grasping. The specific steps are as follows. The robot moves to the coupler position according to the optimal grasping path and monitors environmental changes in real time to obtain the status information of the robot upon reaching the coupler; Image perception is used to correct the grasping posture of the end effector of the driven robotic arm in real time, so that the end effector of the driven robotic arm is aligned with the hook and a corrected grasping posture is generated. Based on the corrected gripping posture, the end effector of the robotic arm is driven to grip the hook, complete the gripping operation, and obtain the status information of gripping completion.

[0012] As a preferred embodiment of the intelligent coupler identification method for tippers based on a re-coupling robot described in this invention, the robot moves the coupler to the docking position along a predetermined trajectory, adjusts the docking posture in real time, and drives the end effector of the robotic arm to release the coupler for re-coupling. The specific steps are as follows. The robot moves the hook from the gripping position to the docking position along a predetermined trajectory and obtains the status information of reaching the docking position. Based on the relative positional relationship between the coupler and the docking position, the docking posture of the robot's end effector is adjusted in real time to ensure that the coupler is accurately aligned with the docking position, thereby obtaining docking posture adjustment information. Based on the docking posture adjustment information, the position of the end effector of the drive robotic arm is finely adjusted and the coupler is released to complete the reconnection operation of the coupler and obtain the status information of reconnection completion.

[0013] Secondly, this invention provides an intelligent coupler identification system for tippers based on a coupler removal and reconnection robot, comprising a multimodal data acquisition module, a coupler identification result generation module, an optimal grasping path generation module, an actuator grasping module, and a coupler reconnection module; the multimodal data acquisition module is used to acquire multimodal data and perform preprocessing, the multimodal data including lidar point cloud data, camera images, and infrared data; the coupler identification result generation module is used to fuse the processed multimodal data and use a convolutional neural network and attitude estimation algorithm to identify coupler information. The system generates coupler recognition results; an optimal gripping path generation module uses a path planning algorithm to calculate the gripping point path of the coupler, avoid interference, optimize the gripping path, and generate the optimal gripping path; an actuator gripping module is used to move the robot to the coupler position according to the optimal gripping path, use image perception to correct the gripping posture in real time, and drive the end effector of the robotic arm to grip the coupler; a coupler reconnection module is used to move the coupler to the docking position along a predetermined trajectory, adjust the docking posture in real time, and drive the end effector of the robotic arm to release the coupler for reconnection.

[0014] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the intelligent identification method for the tipper hook based on the unhooking robot described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent identification method for the tipper coupler based on the uncoupling robot described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By jointly modeling spatial structure, texture details, and thermal boundary information at the feature level, the coupler recognition process can simultaneously capture the composite features of its geometric shape and thermal profile, effectively reducing the risk of misidentification caused by factors such as material differences, surface contamination, or changes in ambient light. At the same time, the multimodal feature tensor has stronger information redundancy and fault tolerance, and can still maintain high recognition accuracy even when some sensor signals are missing or degraded in quality. This ensures the continuous stability and reliability of the recognition process in extreme or interference environments, providing solid data support for subsequent path planning and end-effector actions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the intelligent identification method for car couplers of a tipper based on a decoupling robot in this invention.

[0019] Figure 2 This is a flowchart of the coupler identification and attitude estimation process in this invention.

[0020] Figure 3 This is a flowchart of path planning and grasping control in this invention.

[0021] Figure 4 This is a flowchart of the coupler movement and reconnection operation in this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This embodiment provides a method for intelligently identifying car couplers in a tipper based on a decoupling robot, including the following steps: S1. Collect multimodal data and perform preprocessing. The multimodal data includes lidar point cloud data, camera images, and infrared data.

[0026] Outlier removal is performed on the lidar point cloud data using statistical filtering to obtain the lidar point cloud data after removal. The camera image is then geometrically corrected using a fisheye distortion correction algorithm to obtain the corrected camera image. Finally, the infrared data is smoothed and its thermal features are enhanced using Gaussian blur and local contrast enhancement methods to obtain the smoothed infrared image.

[0027] It should be noted that in the process of removing outliers from LiDAR point cloud data using statistical filtering, the search radius threshold and the neighbor point threshold are first set to determine outliers. The search radius threshold is set based on the LiDAR's resolution and sampling density. For example, by combining the LiDAR's angular resolution and ranging accuracy, the point cloud distribution density per unit solid angle is calculated, and then a reasonable spherical search radius range is deduced for typical areas on or around the coupler surface. The neighbor point threshold is based on statistical analysis of the point cloud density on the coupler surface. By sampling multiple representative target areas, the number of neighbor points for each sampled point is counted under the set search radius. The average value and standard deviation range are used as references to set the criteria for removing outliers. When performing fisheye distortion correction on camera images, the radial and tangential distortion coefficients of the camera are first calculated based on the calibration board image to establish a mapping relationship between pixel coordinates and real scene coordinates. The position of each pixel is repositioned using the inverse distortion mapping algorithm. Then, the image content is resampled and reconstructed through bilinear interpolation to generate a geometrically corrected image. When smoothing and enhancing the thermal features of infrared data, the Gaussian kernel parameters are first set, for example, using a Gaussian kernel with a kernel size of 5×5 and a standard deviation of 1.0. Convolution operations are performed on the infrared image to suppress noise. Then, the local window size and brightness enhancement threshold are set, for example, the window size is 15×15 and the brightness enhancement threshold is the local average plus 15. Pixels exceeding the brightness threshold are brightness enhanced to enhance the temperature distribution features. Finally, the cleaned and enhanced LiDAR point cloud data, camera images, and infrared data are output.

[0028] The RANSAC algorithm is used to perform plane fitting and target extraction on the removed lidar point cloud data, separating the ground and coupler-related target areas to generate spatial structure feature data.

[0029] It should be noted that, in the process of plane fitting and target extraction using the RANSAC algorithm on the removed LiDAR point cloud data, the maximum number of iterations, distance threshold, and inlier ratio threshold are first set. For example, the maximum number of iterations is 100, the distance threshold is 0.05 meters, and the inlier ratio threshold is 0.8. Then, in each iteration, three points are randomly selected for plane parameter estimation, and the plane equation is calculated using the selected points. The expression is: ; in, Indicates the plane normal vector in Components in the axial direction, Indicates the plane normal vector in Components in the axial direction, Indicates the plane normal vector in Components in the axial direction, This represents the result of multiplying the directed distance from the plane to the origin by the modulus. This indicates that a point in the lidar point cloud is in Coordinate values ​​on the axis This indicates that a point in the lidar point cloud is in Coordinate values ​​on the axis This indicates that a point in the lidar point cloud is in Coordinate values ​​on the axis; Then, iterate through each point in the lidar point cloud data and calculate the distance to the estimated plane, expressed as: ; in, Point to plane vertical distance, Point The absolute value of the algebraic distance obtained after substituting into the plane equation. Represents the magnitude of the plane normal vector; If the distance is less than the set distance threshold, the point is marked as an interior point; the number of all interior points is counted, and when the proportion of interior points exceeds the set threshold, the current plane is recorded as a candidate ground plane; after all iterations, the plane with the most interior points is selected as the ground fitting result; then, the point cloud data on the ground plane is removed, and the remaining LiDAR point cloud data is retained as candidate data containing target areas related to the coupler; finally, the remaining point cloud data is clustered based on spatial location distribution, for example, using the Euclidean clustering method, to generate data representing spatial structure features.

[0030] The Retinex algorithm is used to enhance the brightness and contrast of the corrected camera image, generating a brightness-enhanced image.

[0031] It should be noted that for the cleaned and enhanced camera image, the image is first converted to the logarithmic domain to separate the illumination and reflection components. Then, the local mean brightness of the image is calculated and used as the estimated illumination component. The illumination component is subtracted from the logarithmic image to obtain the reflection component. The reflection component is then normalized so that its value range is between 0 and 255, and the image contrast is enhanced by histogram equalization. After that, the processed image is converted back from the logarithmic domain to the linear domain to restore the natural visual effect of the image. Finally, a bilateral filter is applied to the entire image to smooth noise while maintaining clear edge structure, generating a brightness-enhanced image.

[0032] For the smoothed infrared image, the Canny edge detection algorithm is used to extract the temperature change boundary and generate an infrared thermal boundary feature map.

[0033] It should be noted that for the smoothed infrared image, the Sobel operator is used to calculate the gradient images in the horizontal and vertical directions respectively; the gradient magnitude and gradient direction of each pixel are calculated based on the gradient images; then, non-maximum suppression is performed on the gradient magnitude images to retain edge responses with local maxima along the gradient direction; subsequently, high and low thresholds are set, mainly considering the gray-level distribution of the image and the range of gradient magnitudes to ensure effective differentiation between strong and weak edges. The high threshold is usually set to a larger value (e.g., 80) to identify obvious edges and avoid noise interference; the low threshold is set to a smaller value (e.g., 40) to capture weaker edge information while retaining more details. The selection of high and low thresholds also depends on experimental tuning, adjusting different combinations to achieve the best edge detection effect. Common setting methods include rules of thumb and automated methods, such as the Otsu method, to improve the accuracy of the settings; edge connection algorithms are used to determine the connection of weak edges, retaining only weak edge pixels connected to strong edges, discarding isolated pixels, and generating an infrared thermal boundary feature map.

[0034] By using a feature-level fusion method, spatial structural feature data, brightness-enhanced images, and infrared thermal boundary feature maps are integrated in a unified manner to construct a multimodal feature dataset.

[0035] It should be noted that spatial alignment processing is performed on the spatial structure feature data, brightness enhancement image, and infrared thermal boundary feature map. First, the spatial structure feature data corresponding to the LiDAR point cloud is projected onto the camera image coordinate system using an extrinsic parameter matrix. Then, the infrared thermal boundary feature map and the brightness enhancement image are aligned at the pixel level using an image registration algorithm. After alignment, the depth value of each point in the spatial structure feature data is combined with the brightness value and thermal boundary gradient magnitude of the corresponding pixel according to position to form a pixel fusion feature vector. Subsequently, using image pixels as indices, the pixel fusion feature vector is concatenated by channel to construct a pixel fusion feature tensor, ultimately generating a multimodal feature dataset containing spatial structure feature data, brightness enhancement image, and infrared thermal boundary feature map.

[0036] S2. The processed multimodal data is fused, and a convolutional neural network and attitude estimation algorithm are used to identify the coupler information and generate the coupler identification result.

[0037] By utilizing spatial registration techniques and image registration algorithms, feature-level fusion is performed on multimodal feature datasets to generate fused multimodal feature tensors.

[0038] It should be noted that when performing feature-level fusion on the multimodal feature dataset, firstly, based on the extrinsic parameter matrix between the LiDAR and the camera, the spatial structural feature data is projected from the LiDAR coordinate system to the two-dimensional image coordinate system of the brightness enhancement image to obtain the corresponding position of the three-dimensional image coordinate system in the brightness enhancement image; then, image registration is performed on the brightness enhancement image and the infrared thermal boundary feature map, and the pixel correspondence between the two is calculated through affine transformation parameters to ensure the consistency of the infrared thermal boundary feature map and the brightness enhancement image in spatial position; after completing spatial registration and image registration, the depth value, brightness value and thermal boundary gradient magnitude at the corresponding pixel position are extracted using the image coordinates as a unified index, and concatenated by channel into a unified coordinate fusion feature vector; finally, the fusion feature vectors corresponding to all pixel positions are arranged according to the height and width dimensions of the image to generate the fused multimodal feature tensor.

[0039] The fused multimodal feature tensor is used to extract coupler feature information and identify the location using a multichannel convolutional neural network, thus obtaining candidate regions and preliminary results.

[0040] Based on the candidate coupler region, the attitude estimation algorithm is applied to extract point coordinates and spatial orientation information, calculate the three-dimensional position and attitude of the coupler, and generate coupler recognition results.

[0041] It should be noted that, based on the candidate coupler region, the spatial structure feature data of the corresponding coupler region in the fused multimodal feature tensor are first extracted using a 3D point cloud segmentation method to obtain a dense point set containing the coupler geometry. Then, based on the 3D registration technique in the attitude estimation algorithm, a representative coupler template point cloud is selected, and the coupler template point cloud and the extracted dense point set are rigidly registered using an iterative nearest-point algorithm. The translation vector and rotation matrix are iteratively calculated to minimize the Euclidean distance error between the two. After registration is completed, the center point coordinates, principal axis direction vector, and attitude parameters such as pitch, yaw, and roll angles of the coupler in 3D space are extracted from the registration results. Finally, the 3D position and attitude information of the coupler are output to form the coupler recognition result.

[0042] S3. The fused multimodal feature tensor is used to extract coupler feature information and identify the position using a multi-channel convolutional neural network to obtain candidate regions and preliminary results.

[0043] The fused multimodal feature tensor is processed by multi-level convolution using a multi-channel convolutional neural network to extract the edge shape features of the image and obtain the edge contour feature map.

[0044] It should be noted that the fused multimodal feature tensor is input into a multichannel convolutional neural network for processing. First, multiple parallel input channels are set to receive the modal information of each mode of the fused multimodal feature tensor. In the first convolutional layer, multiple convolutional kernels of size, for example, 3×3, are used to extract features from each channel, and the ReLU activation function is used to enhance the nonlinear expression capability. Subsequently, multi-level convolutional processing is performed to extract low-level texture features, high-level semantic features, and mid-level edge information layer by layer. Batch normalization and max pooling operations are applied after each convolutional layer to stabilize the training process and compress the feature map size. After, for example, three to five levels of convolutional operations, the channel feature map with strong edge response is output in the last layer. Then, the complete edge contour feature map is generated by channel concatenation or weighted fusion.

[0045] Surface texture information is extracted from the edge shape feature map to generate a texture enhancement feature map.

[0046] It should be noted that, based on the edge contour feature map, Gabor filters are used to extract the directional texture information of the local image. First, multiple Gabor kernels with combinations of directions and scales are constructed. For example, the direction angles are set to 0°, 45°, 90°, and 135°, and the scale parameters are set to 2, 4, and 8 pixels, respectively. Two-dimensional convolution operations are performed on the edge contour feature map for each kernel to extract the texture response of the corresponding direction and scale. After normalizing the response maps of each direction, all texture response maps are fused by pixel-level weighted averaging to form a texture fusion map containing multi-directional texture details. Finally, the local binary mode method is combined to perform local texture encoding on the texture fusion map to enhance the ability to express gray-level differences between local image blocks and generate a texture enhancement feature map containing directional texture and local encoding information.

[0047] By fusing depth channel information, spatial depth distribution is extracted to generate a spatial positioning feature map.

[0048] It should be noted that the spatial distance information from the LiDAR point cloud data is combined with the texture enhancement feature map. First, the 3D coordinates of the LiDAR point cloud data are processed to map the spatial distance information to a 2D coordinate space, ensuring that each image pixel corresponds to a depth value, forming a depth feature map. Then, the depth feature map and the texture enhancement feature map are fused pixel-by-pixel using a weighted average method. The weights are adjusted based on the reliability of each channel's information; for example, the weight for depth information can be set to 0.7, and the weight for the texture enhancement feature map to 0.3. Next, a convolutional neural network is used to perform multi-layer convolution operations on the fused depth feature map and texture enhancement feature map to extract the spatial distribution features from the depth information. Finally, max pooling and average pooling operations are used to further extract spatial localization features, generating a spatial localization feature map.

[0049] Thermal boundary extraction is performed by combining the infrared channel with the spatial positioning feature map to highlight the temperature difference area of ​​the coupler and generate a thermal sensing feature map.

[0050] It should be noted that, in the spatial positioning feature map, infrared channel image data related to the coupler is first acquired, and temperature information is extracted from the infrared images. Then, the Canny edge detection algorithm is used to extract thermal boundaries on the infrared images, highlighting areas of temperature difference in the coupler. Next, the extracted thermal boundaries are fused pixel-by-pixel with the spatial positioning feature map using a point-by-point weighted average method. The weight values ​​can be adjusted according to the significance of the temperature difference areas; for example, areas with larger temperature differences have larger weights. Finally, the fused result generates a thermal sensing feature map, which clearly highlights the temperature difference between the coupler and the surrounding environment.

[0051] The region proposal method is used to analyze the thermal sensing feature map and the spatial positioning feature map, calculate the confidence of the coupler position, and mark the candidate coupler region in the image to obtain the preliminary recognition results.

[0052] It should be noted that a sliding window strategy is used to generate multiple candidate region windows of fixed size on the image plane, for example, a window size of 64×64 pixels and a step size of 16 pixels. Then, local statistical analysis is performed on the thermal sensing feature map within each candidate region window to calculate the mean and maximum value of the thermal boundary gradient magnitude, and to determine whether there are continuous regions whose local gradient magnitude is more than twice the mean of the whole image. If so, they are marked as thermal response regions. In parallel, depth gradient calculation is performed on the spatial positioning feature map within each candidate region window to analyze whether there are significant depth changes within the region, such as whether the difference between the maximum and minimum depths within the region exceeds a set confidence threshold for coupler candidate regions, such as 500 mm. Based on the judgment, confidence scores are assigned to candidate regions that meet the thermal response condition and the depth change condition, for example, using the maximum gradient magnitude and the depth difference as confidence sub-items, and the region confidence is calculated by linear weighting. Finally, candidate regions with confidence scores higher than the set confidence threshold for coupler candidate regions are selected, and the positions of these candidate regions are marked on the image plane with rectangular borders to form the preliminary identification results of coupler candidate regions.

[0053] S4. A path planning algorithm is used to calculate the path of the coupler's gripping point, avoid interference and optimize the gripping path to generate the optimal gripping path.

[0054] Based on the coupler recognition results, an environmental map is constructed and potential interference objects in the grabbing area are identified to obtain a preliminary grabbing environment map.

[0055] It should be noted that, based on the coupler recognition results, the three-dimensional position and attitude information of the coupler is obtained to ensure accurate positioning. Next, multimodal sensing data, including LiDAR point cloud data, camera images, and infrared thermal sensing images, are used. Spatial registration technology is employed to align different data sources, ensuring accurate fusion of all sensing data within the same spatial coordinate system. Then, feature extraction algorithms are used to extract the three-dimensional spatial position and attitude information of each object in the environment, and the fused multimodal feature tensor is used to construct an environmental map. This environmental map includes spatial information about the coupler and surrounding objects. Based on the coupler's position coordinates and attitude information, the specific location of the coupler's gripping area and the potential spatial range covered by the gripping action are determined in three-dimensional space. Then, combining the three-dimensional spatial positions of all objects recorded in the environmental map, objects located within the gripping area or intersecting with the gripping path are selected as potential interferences. For each potential interference, the spatial relationship between its three-dimensional coordinates and the gripping path is analyzed to determine whether it will obstruct the robotic arm's gripping action, and the spatial distribution information of the interference is recorded for subsequent gripping path adjustments. During the identification process, factors such as the object's size, relative position, and distance are considered to ensure that interferences do not hinder the coupler's operation during gripping. Through spatial analysis and object feature recognition, the relative position and degree of influence of each interference to the gripping path are calculated; integrating the interference information and the gripping area, a preliminary gripping environment map is generated.

[0056] Based on the coupler position and interference distribution in the preliminary grasping environment map, a feasible grasping path from the current position to the target grasping point is constructed, and the preliminary grasping path is calculated using the fast random tree algorithm.

[0057] It should be noted that, based on the coupler position and interference distribution in the capture environment map, the three-dimensional coordinates of the current position and the target capture point are first determined. Then, the coupler position, target capture point, and the specific locations of potential interference are marked on the environment map. Using the fast random tree algorithm, the current position is taken as the root node of the tree structure and recorded as the starting node. Multiple candidate target nodes are generated through random sampling in space, and the connecting capture paths from the starting node to each candidate target node are calculated. All connecting capture paths are uniformly named candidate extended capture paths. For each candidate extended capture path, its feasibility is determined based on whether there are nodes that overlap or are too close to the three-dimensional coordinates of interference. Feasible candidate extended capture paths are retained for subsequent expansion. During the expansion of the crawling path, the process gradually extends from the current expansion boundary towards the target crawling point, forming a set of connected crawling paths, uniformly named the expanded crawling path segment. For each new node generated, the Euclidean distance from the new node to the target crawling point is calculated, and the minimum spatial distance between the new node and any interfering object is determined. If the distance is less than a set safety threshold, the new node is marked as unusable. The Euclidean distance is used as the crawling path efficiency indicator, and the minimum distance to interfering objects is used as the crawling path safety indicator. The effectiveness of each new node is evaluated by combining these two indicators, and the evaluation results are used to guide the crawling path expansion direction and branch selection. After generating a crawling path that meets the conditions, all extended crawling path segments are connected and crawling path smoothing is performed to obtain a continuous, collision-free overall crawling path, named the preliminary crawling path. The final output is the preliminary crawling path from the current position to the target crawling point, avoiding all interference and satisfying the planning constraints.

[0058] By combining the joint structure and motion characteristics of the robotic arm, the initial grasping path is optimized, the motion sequence of each joint is adjusted, and the optimal grasping path is generated.

[0059] It should be noted that, based on the joint structure and motion characteristics of the robotic arm, a kinematic model of the robotic arm is obtained, including the maximum angle range of each joint, the coupling relationship between joints, and the maximum acceleration and maximum torque limits. Then, the initial grasping path is discretized, decomposing it into trajectory points with multiple time steps to ensure that the trajectory points conform to the maximum angle range and motion limits of the robotic arm joints. Next, based on the kinematic constraints of the robotic arm, the acceleration and torque of each joint are calculated, and the joint speed and acceleration are adjusted according to the actual situation to avoid exceeding the maximum limits. During trajectory optimization, optimization algorithms (e.g., gradient descent or particle swarm optimization) are employed to adjust the smoothness of the grasping path and the motion sequence by minimizing peak acceleration and torque. During optimization, the dynamic performance of each joint is considered, and the acceleration curves of the joints are gradually adjusted to reduce abrupt changes, ensure the smoothness of the grasping path, and balance the load on each joint of the robotic arm. Finally, an optimized optimal grasping path is generated, which satisfies the acceleration and torque constraints of all joints while ensuring the completion of the grasping task.

[0060] S5. The robot moves to the coupler position according to the optimal grasping path, uses image perception to correct the grasping posture in real time, and drives the end effector of the robotic arm to clamp the coupler for grasping.

[0061] The robot moves to the coupler position according to the optimal grasping path and monitors environmental changes in real time to obtain the robot's status information upon reaching the coupler.

[0062] It should be noted that the robot first reads the predetermined trajectory coordinates and speed parameters from the optimal grasping path data, and then drives the robotic arm to move segment by segment to the coupler position according to the predetermined trajectory through the motion controller. During the movement, the robot synchronously collects environmental change information using image perception data, depth map data, and inertial measurement unit data, and updates the robot's current three-dimensional coordinates and attitude angle in real time. By comparing the current position information with the target coordinates in the optimal grasping path data, the relative distance between the robot and the coupler position is calculated, and state information reflecting the relationship between the robot's current position and the target grasping point is output. Finally, the coupler position and the robot's current three-dimensional coordinates are marked on the environmental map to achieve real-time confirmation of the robot's arrival at the coupler position.

[0063] Image perception is used to correct the gripping posture of the end effector of the driven robotic arm in real time, so that the end effector of the driven robotic arm is aligned with the hook and a corrected gripping posture is generated.

[0064] It should be noted that, using image perception, firstly, an image frame sequence containing the coupler is acquired, and the edge contour and feature point coordinates of the coupler in the image are extracted; combined with the depth map information corresponding to the current image frame in the known camera, the feature points of the coupler in the image are converted into three-dimensional coordinates through spatial back projection; the three-dimensional coordinate system corresponding to the current posture of the end effector of the driving robot arm is obtained, and the spatial offset and posture difference between the three-dimensional coordinates of the coupler and the three-dimensional coordinates of the end effector of the driving robot arm are calculated through coordinate transformation; real-time correction instructions are generated based on the spatial offset and posture difference to adjust the position and posture angle of the end effector of the driving robot arm in three-dimensional space, so that the normal vector of the end effector's gripping surface is consistent with the normal vector of the coupler surface; after completing the real-time posture adjustment, the current posture of the end effector is output as the corrected gripping posture.

[0065] Based on the corrected gripping posture, the end effector of the robotic arm is driven to grip the hook, complete the gripping operation, and obtain the status information of gripping completion.

[0066] It should be noted that, based on the calibrated gripping posture, the end effector's driver program is first activated to ensure the actuator is in a precise gripping position. The end effector adjusts its joint angles and motion trajectory according to the calibrated posture information, slowly approaching the coupler and performing the clamping operation. Upon contact with the coupler, the actuator precisely applies an appropriate clamping force according to a preset gripping force control algorithm, ensuring the clamping action does not damage the coupler. After the actuator clamps the coupler, sensors or torque sensors monitor the clamping status in real time to ensure the clamping force meets preset standards. After the gripping operation is completed, the sensors provide feedback on the gripping completion status, including whether the coupler is stably clamped and whether the predetermined gripping force has been reached. During this process, if gripping instability or abnormalities are detected, appropriate adjustments are immediately made to ensure successful gripping operation, ultimately obtaining the gripping completion status information.

[0067] S6. The robot moves the coupler to the docking position along the predetermined trajectory, adjusts the docking posture in real time, and drives the end effector of the robotic arm to release the coupler for reconnection.

[0068] The robot moves the hook from the gripping position to the docking position along a predetermined trajectory and obtains the status information of reaching the docking position.

[0069] It should be noted that, based on the planned robotic arm motion trajectory in the optimal gripping path, each joint of the robotic arm executes control commands sequentially according to the optimized motion sequence, smoothly moving the hook while maintaining the gripping state. During this process, the output data of the robotic arm position encoder, end effector attitude, and gripping status sensor are monitored in real time to determine whether the robotic arm is accurately moving along the predetermined trajectory. During movement, potential interference objects marked on the gripping environment map are avoided, and before approaching the docking position, a vision sensor is used to assist in judging the spatial deviation between the current position and the target position, fine-tuning the end effector attitude. After reaching the target docking position, the position, attitude, and gripping status data of the robotic arm's end effector are read, and a comprehensive judgment is made on whether the robotic arm has stably reached the target position and maintained the gripping state, thereby generating status information indicating arrival at the docking position.

[0070] Based on the relative position of the coupler and the docking position, the docking posture of the robot's end effector is adjusted in real time to ensure that the coupler is accurately aligned with the docking position, thereby obtaining docking posture adjustment information.

[0071] It should be noted that the relative pose information of the coupler and the docking position in three-dimensional space is acquired by a vision sensor set at the docking position. The spatial offset and attitude deviation angle between the center point of the coupler and the center point of the docking position are calculated, and translational and rotational errors are extracted based on Euler angles or quaternion representations, respectively. The translational and rotational errors are input into the end effector control method based on servo control, and a visual servo control algorithm is used to compensate for the errors in real time, adjusting the position of the end effector in the X, Y, and Z directions and the rotation angle around the three axes. During the adjustment process, a new frame image is continuously acquired and the relative pose calculation is updated until the translational and rotational errors are less than the preset error compensation termination threshold, for example, the translational error is less than 2 mm and the attitude error is less than 1°. After the conditions are met, the final end effector attitude parameters are recorded as docking attitude adjustment information.

[0072] Based on the docking posture adjustment information, the position of the end effector of the drive robotic arm is finely adjusted and the coupler is released to complete the reconnection operation of the coupler and obtain the status information of reconnection completion.

[0073] It should be noted that, based on the docking attitude adjustment information, the end effector of the driven robotic arm is first controlled to slowly approach the docking position according to the attitude parameters recorded in the docking attitude adjustment information. During the approach, the real-time positional deviation between the end effector and the coupler docking position is continuously collected, and a visual servo control algorithm is used to dynamically correct minor offsets to ensure that the coupler and the docking surface of the docking position are completely aligned. After reaching the final docking position, the gripper release mechanism of the driven robotic arm end effector is controlled to slowly release the gripper, smoothly releasing the coupler into the target structure. The end effector is kept stationary for a period of time, such as 2 seconds, to confirm that the coupler is stably connected before retracting the end effector. After completing the above operations, the gripping state, release state, and position and attitude parameters of the end effector at the time of coupler release are recorded to generate the reconnection completion status information.

[0074] This embodiment also provides an intelligent coupler identification system for tippers based on a hook-unhooking robot, including: The system comprises a multimodal data acquisition module, a coupler identification result generation module, an optimal gripping path generation module, an actuator gripping module, and a coupler reconnection module. The multimodal data acquisition module collects and preprocesses multimodal data, including LiDAR point cloud data, camera images, and infrared data. The coupler identification result generation module fuses the processed multimodal data, uses a convolutional neural network and attitude estimation algorithm to identify coupler information, and generates a coupler identification result. The optimal gripping path generation module uses a path planning algorithm to calculate the gripping point path of the coupler, avoids interference, and optimizes the gripping path to generate an optimal gripping path. The actuator gripping module moves the robot to the coupler position along the optimal gripping path, uses image perception to correct the gripping posture in real time, and drives the end effector of the robotic arm to grip the coupler. The coupler reconnection module moves the robot along a predetermined trajectory to the docking position, adjusts the docking posture in real time, and drives the end effector of the robotic arm to release the coupler for reconnection.

[0075] This embodiment also provides a computer device applicable to the intelligent identification method of car coupler based on a decoupling and recoupling robot for tippers, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the intelligent identification method of car coupler based on a decoupling and recoupling robot for tippers as proposed in the above embodiment.

[0076] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0077] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the intelligent identification method for the tipper based on the hook-and-unhook robot proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0078] In summary, this invention achieves this by jointly modeling spatial structure, texture details, and thermal boundary information at the feature level. This enables the coupler recognition process to simultaneously capture the composite features of its geometric shape and thermal profile, effectively reducing the risk of misidentification caused by factors such as material differences, surface contamination, or changes in ambient lighting. Furthermore, the multimodal feature tensor possesses stronger information redundancy and fault tolerance, maintaining high recognition accuracy even when some sensor signals are missing or degraded. This ensures the continuous stability and reliability of the recognition process under extreme or interference environments, providing solid data support for subsequent path planning and end-effector actions.

[0079] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligently identifying car couplers in a tipper based on a decoupling robot, characterized in that: include, Collect multimodal data and perform preprocessing. The multimodal data includes lidar point cloud data, camera images, and infrared data. The processed multimodal data is fused, and a convolutional neural network and attitude estimation algorithm are used to identify coupler information and generate coupler identification results. A path planning algorithm is used to calculate the path to the coupler's gripping point, avoid interference, optimize the gripping path, and generate the optimal gripping path. The robot moves to the coupler position according to the optimal grasping path, uses image perception to correct the grasping posture in real time, and drives the end effector of the robotic arm to clamp the coupler for grasping. The robot moves the coupler to the docking position along a predetermined trajectory, adjusts the docking posture in real time, and drives the end effector of the robotic arm to release the coupler for reconnection. The specific steps for generating the coupler recognition result are as follows: Using spatial registration techniques and image registration algorithms, feature-level fusion is performed on a multimodal feature dataset to generate a fused multimodal feature tensor; The fused multimodal feature tensor is used to extract coupler feature information and identify the location using a multi-channel convolutional neural network to obtain candidate regions and preliminary results. Based on the candidate coupler region, the attitude estimation algorithm is applied to extract point coordinates and spatial orientation information, calculate the three-dimensional position and attitude of the coupler, and generate coupler recognition results. The fused multimodal feature tensor is used to extract coupler feature information and identify positions using a multi-channel convolutional neural network, resulting in candidate regions and preliminary results. The specific steps are as follows. The fused multimodal feature tensor is processed by multi-level convolution using a multi-channel convolutional neural network to extract the edge shape features of the image and obtain the edge contour feature map. Surface texture information is extracted from the edge shape feature map to generate a texture enhancement feature map; By fusing depth channel information, spatial depth distribution is extracted to generate a spatial positioning feature map; Thermal boundary extraction is performed by combining the infrared channel with the spatial positioning feature map to highlight the temperature difference area of ​​the coupler and generate a thermal sensing feature map. The region proposal method is used to analyze the thermal sensing feature map and the spatial positioning feature map, calculate the confidence of the coupler position, and mark the candidate coupler region in the image to obtain the preliminary recognition results.

2. The intelligent identification method for car couplers of a tipper based on a decoupling robot as described in claim 1, characterized in that: The process of collecting and preprocessing multimodal data is as follows: Outlier removal is performed on the lidar point cloud data using statistical filtering to obtain the lidar point cloud data after removal. The camera image is geometrically corrected using a fisheye distortion correction algorithm to obtain the corrected camera image. In addition, the infrared data is smoothed and thermal feature enhanced by combining Gaussian blur and local contrast enhancement methods to obtain the smoothed infrared image. The removed lidar point cloud data is used to perform plane fitting and target extraction using the RANSAC algorithm to separate the ground and coupler-related target areas and generate spatial structure feature data. The Retinex algorithm is used to enhance the brightness and contrast of the corrected camera image, generating a brightness-enhanced image. For the smoothed infrared image, the Canny edge detection algorithm is used to extract the temperature change boundary and generate an infrared thermal boundary feature map; By using a feature-level fusion method, spatial structural feature data, brightness-enhanced images, and infrared thermal boundary feature maps are integrated in a unified manner to construct a multimodal feature dataset.

3. The intelligent identification method for car couplers of a tipper based on a decoupling robot as described in claim 1, characterized in that: The specific steps for generating the optimal crawling path are as follows. Based on the coupler recognition results, an environmental map is constructed and potential interference objects in the grasping area are identified to obtain a preliminary grasping environment map; Based on the coupler position and interference distribution in the preliminary grasping environment map, a feasible grasping path from the current position to the target grasping point is constructed, and the preliminary grasping path is calculated using the fast random tree algorithm. By combining the joint structure and motion characteristics of the robotic arm, the initial grasping path is optimized, the motion sequence of each joint is adjusted, and the optimal grasping path is generated.

4. The intelligent coupler identification method for tippers based on a decoupling robot as described in claim 3, characterized in that: The robot moves to the coupler position according to the optimal grasping path, uses image perception to correct the grasping posture in real time, and drives the end effector of the robotic arm to grip the coupler for grasping. The specific steps are as follows. The robot moves to the coupler position according to the optimal grasping path and monitors environmental changes in real time to obtain the status information of the robot upon reaching the coupler; Image perception is used to correct the grasping posture of the end effector of the driving robotic arm in real time, and a corrected grasping posture is generated. Based on the corrected gripping posture, the end effector of the robotic arm is driven to grip the hook, complete the gripping operation, and obtain the status information of gripping completion.

5. The intelligent identification method for car couplers of a tipper based on a decoupling robot as described in claim 4, characterized in that: The robot moves the coupler to the docking position along a predetermined trajectory, adjusts the docking posture in real time, and drives the end effector of the robotic arm to release the coupler for re-engagement. The specific steps are as follows. The robot moves the hook from the gripping position to the docking position along a predetermined trajectory and obtains the status information of reaching the docking position. Based on the relative positional relationship between the coupler and the docking position, the docking posture of the robot's end effector is adjusted in real time to ensure that the coupler is accurately aligned with the docking position, thereby obtaining docking posture adjustment information. Based on the docking posture adjustment information, the position of the end effector of the drive robotic arm is finely adjusted and the coupler is released to complete the reconnection operation of the coupler and obtain the status information of reconnection completion.

6. A smart coupler identification system for tippers based on a decoupling / re-coupling robot, used to implement the smart coupler identification method for tippers based on a decoupling / re-coupling robot as described in any one of claims 1 to 5, characterized in that: It includes a multimodal data acquisition module, a coupler identification result generation module, an optimal grasping path generation module, an actuator grasping module, and a coupler coupling module; A multimodal data acquisition module is used to acquire multimodal data and perform preprocessing. The multimodal data includes lidar point cloud data, camera images, and infrared data. The coupler recognition result generation module is used to fuse the processed multimodal data, and uses convolutional neural networks and attitude estimation algorithms to recognize coupler information and generate coupler recognition results. The optimal grasping path generation module is used to calculate the grasping point path of the coupler using a path planning algorithm, avoid interference and optimize the grasping path to generate the optimal grasping path. The actuator gripping module is used to move the robot to the coupler position according to the optimal gripping path, use image perception to correct the gripping posture in real time, and drive the end effector of the robotic arm to grip the coupler for gripping. The coupler reconnection module is used by the robot to move the coupler to the docking position along a predetermined trajectory, adjust the docking posture in real time, and drive the end effector of the robotic arm to release the coupler for reconnection.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent identification method for the tipper based on the unhooking robot as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent identification method for the tipper based on the unhooking robot as described in any one of claims 1 to 5.

Images