A vision-based robotic unhooking method

Abstract

The application discloses a kind of robot unhooking methods based on vision, it is related to intelligent robot technical field, including, collection environment data, the environment data includes point cloud data, environment color image, environment thermal imaging image and voiceprint data, the environment data is preprocessed, the environment data after pre-processing is fused by multimodal fusion network and is handled, generates environment perception graph, judges car hook state and environmental change, and is labeled;According to the environment perception graph, identify car hook position and mark barrier area, obtain the handle coordinate of car hook under the robot coordinate system by three-dimensional coordinate conversion, and generate smooth motion path in combination with A* algorithm and B spline curve.The application collects multiple types of environmental data, and carries out multimodal fusion, realizes the comprehensive perception to complex environment, improves the accuracy and adaptability of environment perception, solves the problem of insufficient operation accuracy of traditional method in complex environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a vision-based robot unhooking method. Background Technology

[0002] With the development of automation and intelligent technologies, robots are increasingly widely used in industrial production and logistics. Traditional mechanical unhooking operations rely on manual intervention, which is inefficient and poses safety risks. Vision technology collects data through sensors such as RGB cameras and LiDAR, and combines it with image processing algorithms to achieve accurate perception of complex environments. However, single-modal data cannot provide comprehensive information support, which has spurred the development of multimodal data fusion technology.

[0003] Nevertheless, existing technologies still have shortcomings. Vision-based robotic unhooking methods mostly rely on a single type of sensor, resulting in poor adaptability and insufficient operational accuracy in complex environments. Furthermore, path planning based on fixed parameters lacks environmental adaptability, making it difficult to meet real-time operational requirements when obstacles change dynamically. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a vision-based robot unhooking method to solve the problems of insufficient operational accuracy and inflexible path planning in complex environments.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a vision-based robotic de-hooking method, comprising,

[0008] Collect environmental data, including point cloud data, environmental color images, environmental thermal imaging images, and voiceprint data. Preprocess the environmental data, and fuse the preprocessed environmental data through a multimodal fusion network to generate an environmental perception map, determine the coupler status and environmental changes, and annotate them.

[0009] Based on the environmental perception map, the position of the coupler and the marked obstacle area are identified. The coordinates of the coupler handle in the robot coordinate system are obtained through three-dimensional coordinate transformation. A smooth motion path is generated by combining the A* algorithm and B-spline curve.

[0010] Generate state vectors and action vectors, calculate the comprehensive reward value, generate the optimal path and optimal action through the deep Q-network algorithm, and perform the unhooking operation.

[0011] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the preprocessing of the environmental data refers to filtering, smoothing and downsampling the point cloud data, denoising the environmental color image, smoothing the environmental thermal imaging image and performing pseudo-color mapping, performing short-time Fourier transform on the voiceprint data, and extracting the Mel frequency cepstral coefficients of the voiceprint data to obtain the voiceprint feature value.

[0012] The environmental color image, environmental thermal imaging image, and voiceprint feature value are normalized, and the voiceprint feature value is filled and truncated to generate a voiceprint feature vector.

[0013] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the step of fusing the environmental data through a multimodal fusion network to generate an environmental perception map includes the following steps:

[0014] A 3D convolutional neural network is used to extract spatial features from point cloud data, and point cloud feature vectors are generated through max pooling layers and fully connected layers.

[0015] A dual-stream convolutional neural network is used to extract color features from environmental color images and temperature features from environmental thermal imaging images, and then the features are fused through a feature fusion layer to generate an image feature vector.

[0016] A one-dimensional convolutional neural network is used to extract the temporal features of the voiceprint feature vector, and a global average pooling layer is used to generate the voiceprint temporal feature vector.

[0017] Attention scores for point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors are calculated using a fully connected layer. Attention weights for point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors are calculated using a softmax function. The point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors are then weighted and summed to generate a comprehensive feature vector.

[0018] The integrated feature vector is input into the decoder network to obtain the environment perception map.

[0019] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the specific steps for judging the state of the hook and environmental changes and marking them are as follows:

[0020] Support vector machine is used to classify the time-series feature vectors of voiceprints to obtain voiceprint feature analysis results, determine the coupler status, obtain dynamic information based on the environmental perception map, and perform time synchronization analysis on the voiceprint feature analysis results and dynamic information to obtain environmental change information.

[0021] Mark the coupler status and environmental change information on the environmental perception map.

[0022] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the step of generating the optimal action and optimal path through the deep Q-network algorithm refers to inputting the state vector and action vector into the deep Q-network algorithm, calculating the Q value of each action to obtain the optimal action, and calculating the movement position based on the optimal action to obtain the optimal path.

[0023] As a preferred embodiment of the vision-based robot unhooking method described in this invention, the step of combining the A* algorithm and B-spline curve to generate a smooth motion path refers to obtaining the robot's current position through the robot's internal encoder, using the robot's current position as the starting point and the coordinates of the hook handle as the target point, running the A* algorithm to generate a preliminary motion path, and using a B-spline curve to fit the path points of the preliminary path to obtain a smooth motion path.

[0024] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the specific steps of performing the unhooking operation are as follows:

[0025] The dynamic window method is used to calculate the feasible window for the robot's current speed and angular velocity, and the optimal speed and angular velocity are selected within the feasible window.

[0026] The robot moves along the optimal path, at the optimal speed and angular velocity, uses the gripper to grasp the coupler handle according to the optimal action, and lifts the coupler handle according to the optimal path.

[0027] As a preferred embodiment of the vision-based robot unhooking method of the present invention, the steps of identifying the hook position and marking obstacle areas based on the environmental perception map, and obtaining the hook handle coordinates in the robot coordinate system through three-dimensional coordinate transformation, are as follows:

[0028] The position of the coupler handle is detected by a fast region convolutional neural network, the bounding box coordinates of the coupler handle are obtained, and the center coordinates of the bounding box are calculated as the coupler position.

[0029] The three-dimensional structural information in the environmental perception map is projected onto a two-dimensional plane to generate a two-dimensional grid map and mark the obstacle area.

[0030] Multiple sets of calibration data were collected using a checkerboard calibration board, and a calibration algorithm was used to calculate the transformation matrix from the camera coordinate system to the robot coordinate system. The coordinates of the hook handle in the robot coordinate system were then obtained through the transformation matrix.

[0031] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the vision-based robot unhooking method as described in the first aspect of the present invention.

[0032] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the vision-based robot unhooking method as described in the first aspect of the present invention.

[0033] The beneficial effects of this invention are as follows: by collecting various types of environmental data and performing multimodal fusion, comprehensive perception of complex environments is achieved, improving the accuracy and adaptability of environmental perception and solving the problem of insufficient operational accuracy of traditional methods in complex environments; by combining the A* algorithm with B-spline curves, the robot can be ensured to reach the target position quickly and safely while avoiding obstacles in the path, thereby improving the overall success rate and safety of the task; it can also dynamically adjust the path according to real-time environmental changes, improving the flexibility and efficiency of path planning. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the vision-based robot unhooking method in Example 1.

[0036] Figure 2 This is a diagram of the multimodal fusion network structure in Example 1.

[0037] Figure 3 This is a schematic diagram of path planning in Example 1.

[0038] Figure 4 This is a flowchart of the unhooking operation control in Example 1. Detailed Implementation

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0042] Example 1, referring to Figure 1 , Figure 2 , Figure 3 and Figure 4 This is the first embodiment of the present invention, which provides a vision-based robot unhooking method, including the following steps:

[0043] S1: Collect environmental data, including point cloud data, environmental color images, environmental thermal imaging images, and voiceprint data. Preprocess the environmental data, and fuse the preprocessed environmental data through a multimodal fusion network to generate an environmental perception map, determine the coupler status and environmental changes, and label them.

[0044] The specific steps are as follows:

[0045] The LiDAR device is set to a scanning frequency of 10Hz, a scanning angle of 360 degrees, and a resolution of 0.1 degrees. The LiDAR device is used to scan the environment and collect point cloud data. Each point contains three-dimensional coordinates and reflection intensity information.

[0046] Set the RGB camera to a resolution of 1920x1080 and a frame rate of 30fps to capture environmental color images.

[0047] Set the infrared camera to a resolution of 640x480 and a frame rate of 25fps to capture environmental thermal images.

[0048] Set the sampling rate of the voiceprint acquisition device to 44.1kHz and the bit depth to 16bit, and use the voiceprint acquisition device to record voiceprint data in the environment.

[0049] Statistical filtering was used to filter the point cloud data, with a neighborhood size of 50 and a standard deviation factor of 1.0. Gaussian filtering was used to smooth the point cloud data, with a Gaussian kernel size of 3x3 and a standard deviation of 1.0. Voxel filtering was used to downsample the point cloud data, with a voxel size of 0.05m.

[0050] The color image is denoised using a nonlocal means denoising algorithm, with the search window size set to 21x21 and the similarity window size set to 7x7.

[0051] The thermal imaging image is smoothed using a Gaussian filtering algorithm with a Gaussian kernel size of 3x3 and a standard deviation of 1.0. Pseudo-color mapping is then applied to the thermal imaging image to convert the temperature information into a visualized color image.

[0052] Short-time Fourier transform was performed on the voiceprint data, with the window function set to Hanning window, window length set to 1024, and overlap rate set to 50%. Mel frequency cepstral coefficients of the voiceprint data were extracted, and the number of Mel filter banks was set to 26. The first 13 coefficients were extracted as voiceprint feature values.

[0053] The environmental color image was adjusted to a resolution of 640x480, and linear normalization was used to normalize the pixel values ​​of the environmental color image to the range of [0,1]. The environmental thermal imaging image was adjusted to a resolution of 640x480, and min-max normalization was used to normalize the temperature values ​​of the environmental thermal imaging image to the range of [0,1]. The min-max normalization method was used to normalize the voiceprint data, and the voiceprint feature values ​​were scaled to the range of [0,1] to ensure uniform data scale and facilitate subsequent model processing. Zero-padding or truncation was performed on the voiceprint feature values ​​to generate a voiceprint feature vector with a fixed length of 128.

[0054] A 3D convolutional neural network (3D CNN) was used to extract spatial features from point cloud data. The convolutional kernel size was 3x3x3 and the stride was 1. A 128-dimensional point cloud feature vector was generated through a max pooling layer and a fully connected layer.

[0055] Two-Stream CNN was used to extract color features from environmental color images and temperature features from environmental thermal images, respectively. The kernel size was 3x3 and the stride was 1. The features were then fused through the feature fusion layer of a multimodal fusion network (MFN) to generate a 128-dimensional image feature vector.

[0056] A one-dimensional convolutional neural network (1D CNN) was used to extract the temporal features of the voiceprint feature vector. The kernel size was 3 and the stride was 1. A 128-dimensional voiceprint temporal feature vector was generated through a global average pooling layer.

[0057] Point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors are input into the attention mechanism module. A fully connected layer is used to calculate the attention scores of the point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors. The softmax function is used to calculate the attention weights of the point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors. The point cloud feature vectors, image feature vectors, and audioprint temporal feature vectors are then weighted and summed to generate a comprehensive feature vector. The comprehensive feature vector is input into the decoder network to obtain an environmental perception map, which includes 3D structural information, color information, temperature information, and audioprint feature information.

[0058] The support vector machine classification model is used to classify the temporal feature vector of the voiceprint to obtain the voiceprint feature analysis results and determine the status of the coupler. If a high-frequency friction sound is detected, that is, the frequency is greater than 5kHz and the duration is more than 1 second, it is determined that there is abnormal friction in the coupler. If a short-term collision sound is detected, that is, the duration is less than 0.1 seconds, it is determined that there is a collision risk in the coupler. Otherwise, the coupler status is determined to be normal.

[0059] The status of the coupler is marked on the environmental perception map, including normal and abnormal friction and collision risk.

[0060] Three-dimensional structural information is extracted from the environmental perception map. The location of obstacles in the environment is identified using the Euclidean clustering algorithm. Color information is extracted from the environmental perception map to identify the type and attributes of obstacles. Temperature information is extracted from the environmental perception map to identify high-temperature and low-temperature regions. Based on the three-dimensional structural and color information, the obstacle is identified using the target detection algorithm, and the movement trajectory of the obstacles is tracked. The temperature change trend is analyzed using environmental thermal imaging images.

[0061] The location, type, attributes, movement trajectory, and temperature change trends of obstacles are integrated into dynamic information.

[0062] The results of voiceprint feature analysis are synchronized with dynamic information over time to obtain environmental change information. If the voiceprint feature is detected to be related to the movement trajectory of an obstacle, that is, the voiceprint feature is synchronized with the movement time of the obstacle, then it is determined that there is a dynamic obstacle in the environment. If the voiceprint feature is detected to be related to a temperature change area, that is, the voiceprint feature is synchronized with the high temperature area, then it is determined that there is a temperature anomaly in the environment.

[0063] Environmental change information, including dynamic obstacles and temperature anomalies, is marked on the environmental perception map.

[0064] It should also be noted that by using statistical filtering, Gaussian filtering, and voxel grid filtering algorithms to preprocess point cloud data, noise can be effectively removed and the amount of computation reduced, improving the accuracy and efficiency of subsequent feature extraction. Compared with existing technologies, this not only improves the accuracy of the environmental perception map but also more accurately identifies the coupler status and environmental changes, especially the judgment of abnormal friction and collision risks, thereby improving the safety and reliability of operation.

[0065] S2: Identify the position of the coupler and mark the obstacle area based on the environmental perception map, obtain the coordinates of the coupler handle in the robot coordinate system through three-dimensional coordinate transformation, and generate a smooth motion path by combining the A* algorithm and B-spline curve.

[0066] The specific steps are as follows:

[0067] The environment-aware map is input into the Faster R-CNN model to detect the position of the coupler handle and obtain the bounding box coordinates of the coupler handle, including the coordinates of the top left and bottom right corners of the bounding box. The coordinates of the top left and bottom right corners of the bounding box are added together and divided by 2 to obtain the center coordinates of the bounding box, which are used as the coupler position. The Faster R-CNN is used as the target detection network because it has high detection accuracy and real-time performance in complex scenes.

[0068] The 3D structural information in the environmental perception map is projected onto a 2D plane (the projection method is to directly map the X and Y coordinates of the 3D structural information onto the 2D plane, ignoring the Z coordinate) to generate a 2D grid map. The resolution of the grid map is set according to the task requirements, for example, each pixel represents 0.05 meters. Obstacle areas are marked as impassable, and free areas are marked as passable.

[0069] Multiple sets of calibration data were collected using a checkerboard calibration board, including images of the calibration board and coordinates and attitude information of the robot end effector in three-dimensional space. The correspondence between the calibration board image coordinate system and the robot coordinate system was established. Based on the calibration data, the transformation matrix from the camera coordinate system to the robot coordinate system was calculated using a calibration algorithm. This matrix includes rotation matrix and translation vector, which are used to describe the relative position and attitude relationship between the camera coordinate system and the robot coordinate system. The center coordinates of the bounding box of the coupler handle were transformed from the image coordinate system to the camera coordinate system, and then transformed to the robot coordinate system using the transformation matrix to obtain the coordinates of the coupler handle in the robot coordinate system.

[0070] The robot's current position is obtained through its internal encoder. Starting from the robot's current position and with the coordinates of the coupler handle as the target point, the A* algorithm is run to generate a preliminary motion path. The A* algorithm is used for preliminary path planning because it has high efficiency and reliability in known environments. B-spline curves are used to fit the path points of the preliminary path to obtain a smooth motion path, ensuring the continuity and executability of the path.

[0071] It should also be noted that the method of using a fast region convolutional neural network model for target detection, combined with the A* algorithm and B-spline curves to generate smooth motion paths, significantly improves real-time performance and accuracy in complex scenarios compared to traditional methods. Furthermore, obtaining the hook handle coordinates in the robot's coordinate system through 3D coordinate transformation enables more precise positioning and path planning, ensuring the robot performs tasks more efficiently and safely, and avoiding task failures due to inaccurate or infeasible paths.

[0072] S3: Generate state vectors and action vectors, calculate the comprehensive reward value, generate the optimal path and optimal action through the deep Q-network algorithm, and perform the unhooking operation.

[0073] The specific steps are as follows:

[0074] The robot begins to move along a smooth path, and a PID controller is used to adjust the robot's speed and direction in real time during the movement to ensure path tracking accuracy.

[0075] Ambient soundprint data was collected using a high-sensitivity microphone at a sampling frequency of 44.1 kHz. The soundprint data was denoised and normalized. Short-time Fourier transform (STFT) was used to convert the ambient soundprint data into time-frequency domain features to obtain ambient soundprint features. Support vector machine (SVM) classification model was used to classify the ambient soundprint features and determine whether there are anomalies in the current environment (such as friction sounds and collision sounds), generating anomaly types and anomaly locations.

[0076] The robot's current position, the hook handle position, the environmental perception map, the anomaly type, the anomaly location, and the force sensor data are integrated into a state vector.

[0077] The robot's direction of movement, speed of movement, and gripper control commands (opening, closing, and adjusting force) are defined and integrated into a motion vector. The robot's direction of movement ranges from 0 to 360 degrees, and the robot's speed ranges from 0 to 1 meter per second.

[0078] The Euclidean distance between the robot's current position and the hook handle position is calculated. An inverse proportional relationship is used to design the distance reward, ensuring that the closer the distance, the greater the reward. An exponential decay relationship is used to design the voiceprint reward, with a high reward when there are no anomalies and a low reward when there are anomalies, ensuring that the impact of anomalies on the reward is significant. A hyperbolic tangent relationship is used to design the grasping reward, with a positive reward when the grasp is successful and a negative reward when the grasp fails, ensuring that the reward is relatively high when the grasping force is moderate. Through experimental optimization and task requirement analysis, the weight coefficients are determined, and the distance reward, voiceprint reward, and grasping reward are weighted and summed to obtain the comprehensive reward value.

[0079] For example, if the Euclidean distance between the robot's current position and the hook handle position is 0.2m, the distance reward is 1 / (0.2+0.1)=3.33; if there are no abnormalities, the voiceprint reward is exp(-0.5*0)=1; if the grab is successful, the grab reward is tanh(8-5)=0.995, and the comprehensive reward is 0.5*3.33+0.3*1+0.2*0.995=2.15.

[0080] The state vector, action vector, and total reward value are input into the Deep Q-Network (DQN) algorithm to calculate the Q-value for each possible action. The Q-value represents the expected cumulative reward of performing the action in a given state. The Q-value is obtained by adding a discount factor to the total reward value obtained after performing the action in the current state and multiplying it by the maximum Q-value of the next state. The action with the largest Q-value is selected as the optimal action. For example, the combination of movement direction and speed with the largest Q-value is selected as the optimal action.

[0081] Based on the direction and speed of movement in the optimal action, adjust the robot's movement. For example, if the optimal action is "forward" and "medium speed", the robot moves forward at a medium speed. Based on the current robot position and speed, combined with the direction and speed of movement in the optimal action, calculate the next movement position. For example, if the direction of movement is "forward" and the speed is "medium speed", the next movement position is the current position plus the speed multiplied by the time step. Check whether the next movement position is within the free area of ​​the environmental perception map to avoid collisions with obstacles. If the next movement position is not within the free area of ​​the environmental perception map, adjust the direction or speed and recalculate the next movement position.

[0082] Create an empty path list, add the current robot position and the next move position to the path list, and continuously add new path points to the path list during the robot's movement to form a complete motion path.

[0083] B-spline curves are used to smooth the path points in the path list. The smoothed path is then optimized using the Dynamic Window Method (DWA) to avoid dynamic obstacles and abnormal areas, thus obtaining the optimal path.

[0084] The Dynamic Window Method (DWA) is used to calculate the feasible window for the robot's current speed and angular velocity. Within the feasible window, the optimal speed and angular velocity are selected (by evaluating path safety and efficiency through an objective function, the speed and angular velocity that maximizes the objective function are selected as the optimal speed and angular velocity). This allows the robot to avoid abnormal areas and quickly approach the target point. The Dynamic Window Method can calculate the robot's motion trajectory in real time, ensuring that the robot can complete tasks efficiently in complex environments.

[0085] The robot moves along the optimal path, at the optimal speed and angular velocity. When the robot reaches the position of the coupler handle, the gripper motor is started to control the gripper to move towards the coupler handle. During the movement of the gripper, the distance between the gripper and the coupler handle is monitored in real time to ensure that the gripper is accurately aligned.

[0086] The robot uses a gripper to grasp the coupler handle according to the optimal action. A force sensor detects the contact force between the gripper and the coupler handle to ensure that the gripping force is appropriate. An RGB camera captures the contact image between the gripper and the coupler handle. The image processing algorithm determines whether the gripper has fully grasped the handle. If the grasping fails, the position or force of the gripper is readjusted.

[0087] The robot lifts the coupler handle along the optimal path. The lifting angle is set according to the coupler type and task requirements (e.g., 30 degrees). During the lifting process, the robot continuously monitors the voiceprint data to ensure that the movement is smooth and without abnormalities. If an abnormality is detected, the robot immediately stops the movement and readjusts the path.

[0088] It should also be noted that applying Deep Q-Networks (DQN) to generate optimal paths and action decisions, and using PID controllers to adjust the robot's speed and direction, results in stronger adaptability and higher operational accuracy in dynamic environments. Compared to existing technologies, it not only responds quickly to environmental changes, such as obstacle movement and abnormal temperatures, but also significantly improves the success rate and efficiency of unhooking operations while ensuring operational safety. Furthermore, optimizing action selection by calculating a comprehensive reward value further ensures the rationality and effectiveness of action execution.

[0089] This embodiment also provides a computer device applicable to vision-based robot unhooking methods, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the vision-based robot unhooking method proposed in the above embodiment.

[0090] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0091] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the vision-based robot unhooking method as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0092] In summary, this invention achieves comprehensive perception of complex environments by collecting various types of environmental data and performing multimodal fusion, thereby improving the accuracy and adaptability of environmental perception and solving the problem of insufficient operational accuracy of traditional methods in complex environments. By combining the A* algorithm with B-spline curves, it ensures that the robot can reach the target position quickly and safely while avoiding obstacles in the path, thus improving the overall success rate and safety of the task. Furthermore, it can dynamically adjust the path according to real-time environmental changes, improving the flexibility and efficiency of path planning.

[0093] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A vision-based robotic hook unhooking method, characterized by: The application relates to a method for realizing automatic uncoupling of a train, and belongs to the field of train automatic uncoupling. The method comprises the following steps: collecting environment data, including point cloud data, environment color images, environment thermal imaging images and voiceprint data, preprocessing the environment data, fusing the preprocessed environment data through a multi-modal fusion network, generating an environment perception graph, judging the coupler state and environment changes, and marking; The voiceprint time sequence feature vector is classified using a support vector machine classification model to obtain a voiceprint feature analysis result, and the coupler state is judged; if high-frequency friction sound, i.e. a frequency greater than 5 kHz and a duration longer than 1 second, is detected, it is judged that the coupler has abnormal friction; if short-time collision sound, i.e. a duration shorter than 0.1 second, is detected, it is judged that the coupler has a collision risk; otherwise, it is judged that the coupler state is normal; The coupler state is marked in the environment perception graph, including normal, abnormal friction and collision risk; Three-dimensional structure information is extracted from the environment perception graph, the position of an obstacle in the environment is recognized using a Euclidean clustering algorithm, color information is extracted from the environment perception graph, the type and attribute of the obstacle are recognized, temperature information is extracted from the environment perception graph, high-temperature and low-temperature areas are recognized, the obstacle is recognized using a target detection algorithm based on the three-dimensional structure information and the color information, the moving track of the obstacle is tracked, and temperature change trends are analyzed using the environment thermal imaging images; The position, type, attribute, moving track and temperature change trend of the obstacle are integrated into dynamic information; The voiceprint feature analysis result and the dynamic information are analyzed in time synchronization to obtain environment change information; if the voiceprint feature is related to the moving track of the obstacle, i.e. the voiceprint feature is synchronized with the moving time of the obstacle, it is judged that there is a dynamic obstacle in the environment; if the voiceprint feature is related to the temperature change area, i.e. the voiceprint feature is synchronized with the high-temperature area in time, it is judged that there is a temperature anomaly in the environment; The environment change information is marked in the environment perception graph, including dynamic obstacles and temperature anomalies; The coupler position is recognized according to the environment perception graph, and an obstacle area is marked; the coupler handle coordinates in a robot coordinate system are obtained through three-dimensional coordinate conversion; a smooth motion path is generated by combining an A* algorithm and a B-spline curve; A state vector and an action vector are generated, a comprehensive reward value is calculated, an optimal action and an optimal path are generated through a deep Q network algorithm, and a coupler uncoupling operation is performed.

2. The vision-based robotic unhooking method of claim 1, wherein: The preprocessing of the environment data refers to filtering, smoothing and down-sampling processing of the point cloud data, denoising processing of the environment color images, smoothing processing of the environment thermal imaging images, pseudo-color mapping, short-time Fourier transform of the voiceprint data, extraction of the mel-frequency cepstrum coefficients of the voiceprint data and obtaining of voiceprint feature values; The environment color images, the environment thermal imaging images and the voiceprint feature values are normalized, the voiceprint feature values are filled and truncated to generate a voiceprint feature vector.

3. The vision-based robotic unhooking method of claim 2, wherein: The environment data is fused through a multi-modal fusion network to generate an environment perception graph, and the specific steps are as follows: Three-dimensional convolutional neural networks are used to extract the spatial features of the point cloud data, and a point cloud feature vector is generated through a maximum pooling layer and a full connection layer; Color features of the environment color image and temperature features of the environment thermal image are extracted by using a double-stream convolutional neural network, and are fused by a feature fusion layer to generate an image feature vector; A one-dimensional convolutional neural network is used to extract the time sequence features of the voiceprint feature vector, and a global average pooling layer is used to generate a voiceprint time sequence feature vector; A fully connected layer is used to calculate the attention scores of the point cloud feature vector, the image feature vector and the voiceprint time sequence feature vector, a softmax function is used to calculate the attention weights of the point cloud feature vector, the image feature vector and the voiceprint time sequence feature vector, and the point cloud feature vector, the image feature vector and the voiceprint time sequence feature vector are weighted and summed to generate a comprehensive feature vector; The comprehensive feature vector is input into a decoder network to obtain an environment perception map.

4. The vision-based robotic unhooking method of claim 3, wherein: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The computer program is executed by the processor to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

5. The vision-based robotic unhooking method of claim 1, wherein: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

6. The vision-based robotic unhooking method of claim 1, wherein: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

7. The vision-based robotic unhooking method of claim 5, wherein: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

8. The vision-based robotic unhooking method of claim 1, wherein: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8. The processor executes the computer program to realize the steps of the vision-based robot uncoupling method of any one of claims 1-8.

Images