Multi-degree-of-freedom dual-arm robot filling path planning method, medium and equipment

Abstract

The application discloses a multi-degree-of-freedom dual-arm robot filling path planning method, which comprises the following steps: inputting a 2D image of a current workpiece surface into a trained recognition network model to obtain a labeled image; obtaining the center coordinates of a filling port area and an exhaust port area and the normal vectors of the center coordinates in a collecting device coordinate system and a robot base coordinate system; obtaining the corresponding poses of an arbitrary joint angle of a robot end in the robot base coordinate system and the poses between the center coordinates in the robot base coordinate system; obtaining the coordinates of each joint space path point in the joint space path and the corresponding motion speed; substituting the coordinates of each joint space path point in the joint space path and the corresponding motion speed into a preset model to obtain the positions and speeds of all joints when the robot is located at each joint space path point; and completing path planning. The application solves the problems that the prior art cannot fully utilize the redundancy of a multi-degree-of-freedom dual-arm robot when performing path planning, and the efficiency is too low and the intelligentization is insufficient.

Description

Technical Field

[0001] This invention relates to the field of path planning, and in particular to a method, medium, and device for planning the refueling path of a multi-degree-of-freedom dual-arm robot. Background Technology

[0002] Robots are a type of high-tech automated production equipment that has developed over the past thirty years. The robotic arm is an important branch of robotics, characterized by its ability to perform various pre-defined tasks through programming. It combines the advantages of both humans and machines in its construction and performance, particularly in terms of human intelligence and adaptability, as well as the accuracy and ability of machines to perform tasks in various environments.

[0003] Dual-arm robots are a new highlight in the industrial robot market. They offer solutions for reducing labor costs, improving production efficiency, and filling labor shortages. Collaborative operation of multiple robots is an inevitable requirement for manufacturing development, and dual-arm robots are a new type of robot developed to meet this need. Compared to single-arm robots, they significantly enhance the robot's adaptability to complex assembly tasks, improve workspace utilization, and their increased redundancy and coordinated operation greatly improve their operational performance, making them more practical than bi-arm robots. Their high autonomy and learning capabilities enable them to adapt to many complex environments, leading to their widespread application in industrial production, hazardous material handling, national defense, and aerospace. However, because the redundancy of dual-arm robots is much higher than that of single-arm robots, the complexity of their path planning methods is also far greater.

[0004] Therefore, how to improve the existing technology for path planning of multi-degree-of-freedom dual-arm robots, which cannot make full use of robot redundancy and has low efficiency and insufficient intelligence, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] Based on this, the purpose of this application is to provide a method, medium, and device for planning the dispensing path of a multi-degree-of-freedom dual-arm robot, so as to solve at least one of the technical problems mentioned in the background art.

[0006] Firstly, this application provides a method for planning the dispensing path of a multi-degree-of-freedom dual-arm robot, including:

[0007] Construct and train a recognition network model that takes 2D images as input and labeled images as output;

[0008] The robot is controlled to move sequentially according to a set step size, acquire a 2D image of the current workpiece surface, and input it into the trained recognition network model to obtain an annotated image; the annotated image includes the filling port area and the venting port area;

[0009] Acquire the point cloud of the current workpiece surface with the current robot end position as the center and a set distance as the radius, and obtain the center coordinates of the filling port area and the exhaust port area in the coordinate system of the acquisition device, as well as the normal vector of each center coordinate, based on the current workpiece surface point cloud.

[0010] Obtain the center coordinates of the filling port area and the exhaust port area, as well as the normal vector of each center coordinate, in the robot's base coordinate system;

[0011] Based on the current coordinates in the robot's base coordinate system and the center coordinates and corresponding normal vectors of the filling port area and the exhaust port area, the pose of each center coordinate in the robot's base coordinate system is obtained;

[0012] Obtain the angle of any joint of the robot in space to get the corresponding pose of the robot end effector in the robot base coordinate system;

[0013] Obtain the joint space path between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtain the coordinates of each joint space path point and the corresponding motion velocity in the joint space path.

[0014] By substituting the coordinates of each joint space path point and its corresponding motion velocity into the preset model, the position and velocity of all joints when the robot is located at each joint space path point are obtained, thus completing the path planning.

[0015] Furthermore, identifying the specific structure of the network model includes:

[0016] Input layer, used to input the acquired 2D image;

[0017] Convolutional layers, connected to the input layer, are used to accept the input 2D image and extract the features of the 2D image to obtain an initial feature map;

[0018] Pooling layers, connected to convolutional layers, are used to downsample the initial feature map to obtain an optimized initial feature map.

[0019] Activation layers, connected to pooling layers, are used to add non-linear activation functions after convolutional and fully connected layers, enhancing the non-linear expressive power of the network.

[0020] Fully connected layers, connected to activation layers, are used to fuse and process extracted features, and to annotate the feature parts to generate an annotated image;

[0021] The output layer, connected to the fully connected layer, is used to output the generated labeled image.

[0022] Further, the steps of constructing and training a recognition network model that takes 2D images as input and labeled images as output include:

[0023] Construct an initial recognition network model;

[0024] Select several 2D images containing the filling port and annotate the area covered by the filling port to obtain an annotated image;

[0025] Each labeled image is rotated several times by a predetermined angle to obtain a subset of labeled images at different angles;

[0026] Construct a training dataset based on labeled images and subsets of labeled images;

[0027] The training dataset is input into the initial recognition network model to optimize its parameters, resulting in the final recognition network model.

[0028] Furthermore, the steps of controlling the robot to move sequentially according to a set step size, acquiring a 2D image of the current workpiece surface, and inputting it into the trained recognition network model to obtain an annotated image include:

[0029] S21: Control the robot to move at the first set step size and obtain a 2D image of the current workpiece surface;

[0030] S22: Input the 2D image of the current workpiece surface into the recognition network model, determine whether the filling port area appears in the 2D image. If it appears, mark the filling port area to obtain the initial marked image and execute step S23. If it does not appear, return to step S21.

[0031] S23: Obtain the center point of the filling port area and the initial annotation image, and determine whether the distance between each center point is greater than the set threshold. If it is greater, move towards the center point of the filling port area according to the second set step size until the distance between each center point is not greater than the set threshold. If it is not greater, execute step S24.

[0032] S24: Obtain a 2D image of the current workpiece surface, which will be the final labeled image.

[0033] Furthermore, the lengths of the first and second steps are as follows:

[0034] The first step length is the diameter of the scanning range of the scanning device; the second step length is the radius of the prior filling port area; the prior filling port area is pre-marked by a person skilled in the art or / and obtained according to the workpiece model or / and pre-set by a person skilled in the art.

[0035] Furthermore, the steps of obtaining the center coordinates of the filling port area and the exhaust port area, as well as the normal vectors of each center coordinate, in the coordinate system of the acquisition device include:

[0036] Obtain the prior region of interest on the workpiece surface and determine the point cloud of the filling port region and the venting port region;

[0037] The coordinates of all points in the point cloud of each region are obtained and calculated in the coordinate system of the acquisition device to obtain the coordinates of the center point of each region.

[0038] Obtain the normal vector corresponding to the coordinates of the center point of each region.

[0039] Further steps to obtain the pose of each center coordinate in the robot's base coordinate system include:

[0040] Construct a right-handed coordinate system with the center coordinate as the origin;

[0041] Obtain the x-axis direction vector and y-axis direction vector in the right-handed coordinate system;

[0042] Construct the rotation matrix between the right-handed coordinate system and the robot's base coordinate system based on the x-axis direction vector and the y-axis direction vector;

[0043] The angle of the center coordinates in the robot's base coordinate system is obtained by using the rotation matrix, thus obtaining the pose of the center coordinates in the robot's base coordinate system.

[0044] Furthermore, the steps of obtaining the joint space path between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtaining the coordinates of each joint space path point and the corresponding motion velocity in the joint space path, include:

[0045] A smooth joint angle trajectory is generated between the arbitrary pose of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system.

[0046] Obtain the joint angle trajectory of each joint at each time point, and obtain the corresponding robot end-effector pose based on the corresponding joint angle trajectory;

[0047] By combining the joint angles, robot end-effector poses, joint angular velocities, and joint angular accelerations at each time point, the joint space path points are obtained.

[0048] Secondly, this application also provides a computer storage medium storing executable program code; the executable program code is used to execute the multi-degree-of-freedom dual-arm robot refueling path planning method described in any one of the first aspects.

[0049] Thirdly, this application also provides a terminal device, including a memory and a processor; the memory stores program code executable by the processor; the program code is used to execute the multi-degree-of-freedom dual-arm robot refueling path planning method described in any one of the first aspects. Attached Figure Description

[0050] Figure 1 This is a flowchart of a method for planning the injection path of a multi-degree-of-freedom dual-arm robot according to an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept and is within the scope of the present invention should be included in the protection scope of the present invention.

[0053] like Figure 1 As shown, this invention provides a method for planning the injection path of a multi-degree-of-freedom dual-arm robot, including:

[0054] S1: Construct and train a recognition network model that takes 2D images as input and labeled images as output;

[0055] Specifically, the recognition network model can be any existing image recognition model to construct the initial recognition network model. Then, by batch acquiring 2D images of different workpieces of the same model, the filling ports in the 2D images are labeled to construct a training dataset. Finally, based on the training dataset, the initial recognition network model is input to train its parameters, resulting in the trained recognition network model. It is worth noting that this step is preparatory work; it can be pre-constructed and trained, or it can be performed simultaneously with step S2, as long as it is constructed and trained before step S3, when the recognition network model is needed for prediction.

[0056] Preferably, the recognition network model can be, but is not limited to, based on the YOLO model, to recognize 2D images and annotate the filling port area. For the application of the present invention, since the filling port and the exhaust port always appear side by side, the recognition accuracy of the filling port area can be improved based on similar features. Therefore, the annotated image preferably includes the filling port area and the exhaust port area.

[0057] More preferably, the specific structure of the recognition network model may include, but is not limited to, the following:

[0058] Input layer, used to input the acquired 2D image;

[0059] A convolutional layer, connected to the input layer, is used to accept the input 2D image and extract the features of the 2D image to obtain an initial feature map.

[0060] Pooling layers, connected to convolutional layers, are used to downsample the initial feature map, reduce the feature dimension, improve computational efficiency, and obtain an optimized initial feature map.

[0061] The activation layer, connected to the pooling layer, is used to add a non-linear activation function after the convolutional and fully connected layers to enhance the non-linear expressive power of the network. The non-linear activation function can be selected, but is not limited to, commonly used functions such as ReLU and Sigmoid.

[0062] A fully connected layer, connected to an activation layer, is used to fuse and process extracted features, and to annotate the feature parts to generate an annotated image;

[0063] The output layer, connected to the fully connected layer, is used to output the generated labeled image.

[0064] Further preferably, the process of constructing and training the recognition network model may optionally include, but is not limited to, the following:

[0065] S11: Construct the initial recognition network model; preferably based on the YOLO model;

[0066] S12: Select several 2D images containing the filling port, and annotate the area covered by the filling port to obtain an annotated image;

[0067] S13: Rotate each labeled image several times by a predetermined angle to obtain a subset of labeled images at different angles;

[0068] S14: Construct a training dataset based on the labeled images and subsets of labeled images;

[0069] S15: Input the training dataset into the initial recognition network model to optimize the parameters of the initial recognition network model and obtain the final recognition network model.

[0070] Specifically, the number of rotations, angles, and directions mentioned above can be arbitrarily set by those skilled in the art.

[0071] Preferably, the labeled images are rotated 60 degrees each time, and rotated 5 times to obtain five new subsets with different angles; then the labeled images and the subsets of labeled images are used together to form the training dataset, or a portion of each set is selected to form the training dataset.

[0072] For example, a person skilled in the art can annotate along the edge of the injection port area in the selected annotated image, and the annotation range is the injection port area, thus obtaining an annotated image. Each annotated image is rotated clockwise to obtain a subset of annotated images with angles differing from the original annotated image by 60°, 120°, 180°, 240°, and 300°, respectively.

[0073] More preferably, step S16 may optionally include, but is not limited to:

[0074] The relevant loss function and optimization algorithm are set by those skilled in the art to obtain the required recognition network model. The training dataset is input into the recognition network model and iterated continuously to finally obtain a recognition network model with 2D images as input and labeled images as output. Specifically, the optimization algorithm, including stochastic gradient descent, Adam, RMSprop, etc., is used to update the recognition network parameters to minimize the loss function.

[0075] S2: Control the robot to move sequentially according to the set step size, acquire the 2D image of the current workpiece surface, and input it into the trained recognition network model to obtain the labeled image; it is worth noting that since the filling port and the venting port always appear side by side at the same time, the labeled image includes the images of the filling port area and the venting port area.

[0076] Specifically, but not limited to, controlling the robot to use an image acquisition device to scan the current workpiece surface with a set step size each time it moves, until the entire workpiece surface is scanned to complete the acquisition of 2D images. Each acquired 2D image is input into the recognition network model. If the 2D image contains a filling port, the recognition network model marks the filling port area to obtain an annotated image. The step size can be set arbitrarily by those skilled in the art based on factors such as the workpiece surface area, the scanning range of the image acquisition device, the influence of the on-site environment, and the process conditions.

[0077] Preferably, the steps of controlling the robot to move sequentially according to a set step size, acquiring a 2D image of the current workpiece surface, and inputting it into the trained recognition network model to obtain an annotated image are optional, but not limited to:

[0078] S21: Control the robot to move at the first set step size and obtain a 2D image of the current workpiece surface;

[0079] S22: Input the 2D image of the current workpiece surface into the recognition network model, determine whether the filling port area appears in the 2D image. If it appears, mark the filling port area to obtain the initial marked image and execute step S23. If it does not appear, return to step S21.

[0080] S23: Obtain the center point of the filling port area and the initial annotation image, and determine whether the distance between each center point is greater than the set threshold. If it is greater, move towards the center point of the filling port area according to the second set step size until the distance between each center point is not greater than the set threshold. If it is not greater, execute step S24.

[0081] S24: Obtain a 2D image of the current workpiece surface, which will be the final labeled image.

[0082] Preferably, the first step length, the second step length, and the set threshold can be arbitrarily set by those skilled in the art.

[0083] More preferably, the first step length is the diameter of the scanning range of the scanning device, and the second step length is the radius of the prior filling port area; the prior filling port area may be selected, but is not limited to, pre-marked by a person skilled in the art or / and obtained according to the workpiece model or / and pre-set by a person skilled in the art.

[0084] For example, a first step length and a second step length are set. The robot's end effector, carrying an image acquisition device, acquires the first 2D image from the lower left corner of the workpiece surface. This image is then input into the recognition network model to determine if a filling port area appears in the 2D image. If it does, the filling port area is marked to obtain an initial marked image. The center point between the filling port area and the initial marked image is then acquired. The distance between each center point is determined to be greater than a set threshold. If it is, the robot moves towards the center point of the filling port area by the second set step length until the distance between each center point is less than the set threshold. If the distance is not greater, the 2D image of the current workpiece surface is acquired, which is the final marked image. If the filling port area does not appear, the robot moves to the right side of the workpiece surface by one step length, continues to acquire 2D images and input them into the recognition network model. When the robot moves to the far right of the workpiece surface, it moves upward by one step length, and then moves to the left side of the workpiece surface by one step length, until the entire workpiece surface has been traversed. Each movement acquires a 2D image of the workpiece surface.

[0085] S3: Obtain the point cloud of the current workpiece surface with the current robot end position as the center and a set distance as the radius, and obtain the center coordinates of the filling port area and the exhaust port area in the coordinate system of the acquisition device, as well as the normal vector of each center coordinate, based on the current workpiece surface point cloud.

[0086] Specifically, a radius is set, and the point cloud of the current workpiece surface within the set radius is obtained. Then, the point cloud of the filling port area and the point cloud of the exhaust port area are obtained based on the obtained workpiece surface point cloud. The center coordinates of the filling port area and the center coordinates of the exhaust port area are obtained based on the point cloud of each area. Finally, the normal vector corresponding to each center point is calculated based on the specific coordinates of the center point of each area. It is worth noting that the radius distance must ensure that the scanning range of the acquisition device includes the filling port area and the exhaust port area.

[0087] Preferably, the step of obtaining the center coordinates of the filling port area and the exhaust port area, as well as the normal vector of each center coordinate, in the coordinate system of the acquisition device may optionally include, but is not limited to, the following:

[0088] S31: Obtain the prior region of interest on the workpiece surface and determine the point cloud of the filling port region and the point cloud of the venting port region;

[0089] S32: Obtain and calculate the average coordinates of all points in the point cloud of each region in the coordinate system of the acquisition device to obtain the coordinates of the center point of each region;

[0090] S33: Obtain the normal vector corresponding to the coordinates of the center point of each region.

[0091] Specifically, optional but not limited to obtaining the prior regions of interest on the workpiece surface, including the filling port region and the vent region, by pre-annotation by those skilled in the art or by pre-setting based on the workpiece model, is possible. Then, preferably, the point cloud on the workpiece surface is filtered by a pass-through filtering method to determine the point cloud of the filling port region and the point cloud of the vent region. The values ​​of all points in the region point cloud on the x-axis and y-axis are obtained, and divided by the number of points in the corresponding region point cloud to obtain the coordinates of the center point of the corresponding region point cloud. Finally, the normal vector corresponding to each center point coordinate is calculated by the point cloud normal vector estimation method. The point cloud normal vector estimation method includes commonly used algorithms such as the Nearest Neighbor Method, the Least Squares Plane Fitting Method, and the Normal Estimation Method.

[0092] S4: Obtain the center coordinates of the filling port area and the exhaust port area, as well as the normal vector of each center coordinate, in the robot's base coordinate system;

[0093] Specifically, optionally, but not limited to, using the prior hand-eye calibration matrix between the acquisition device coordinate system and the robot base coordinate system, the center coordinates of the inlet and outlet regions in step S3, as well as the normal vectors of each center coordinate, can be transformed to the robot base coordinate system to obtain the center coordinates of the inlet and outlet regions and the normal vectors of each center coordinate in the robot base coordinate system. It is worth noting that the prior hand-eye calibration matrix can be preset according to the relationship between the image acquisition device and the robot base, or it can be calculated by those skilled in the art after fixing the image acquisition device to the robot end effector and before the coordinate transformation step.

[0094] S5: Based on the current coordinates in the robot's base coordinate system and the center coordinates and corresponding normal vectors of the filling port area and the exhaust port area, obtain the pose of each center coordinate in the robot's base coordinate system;

[0095] For example, taking the center coordinates (x, y, z) and the corresponding normal vector (nx, ny, nz) as an example, the specific steps to obtain the pose of the center coordinates in the robot's base coordinate system can be, but are not limited to, including:

[0096] S51: Construct a right-handed coordinate system with the center coordinate as the origin;

[0097] S52: Obtain the x-axis direction vector and y-axis direction vector in the right-handed coordinate system;

[0098] S53: Construct the rotation matrix between the right-handed coordinate system and the robot's base coordinate system based on the x-axis direction vector and the y-axis direction vector;

[0099] S54: Obtain the angle of the center coordinates in the robot base coordinate system based on the rotation matrix, and obtain the pose of the center coordinates in the robot base coordinate system.

[0100] Specifically, a right-handed coordinate system can be constructed, but is not limited to, with the center coordinates as the origin and the normal vector (nx, ny, nz) as the z-axis unit vector. Then, preferably, the X-axis and Y-axis direction vectors are obtained by cross product calculation, and a rotation matrix is ​​constructed with the X-axis direction vector as the first column and the Y-axis direction vector as the second column. Preferably, the rotation angle of the center coordinates in the robot base coordinate system is calculated by inverse trigonometric functions, and the pose of the center coordinates in the robot base coordinate system can be obtained by combining the position of the center coordinates.

[0101] S6: Obtain the angle of any joint of the robot in space, and get the corresponding pose of the robot end effector in the robot base coordinate system;

[0102] Specifically, the angles of the joints can be obtained by reading the feedback signals from the servo motors carried by the robot itself. By substituting the joint angles of the robot into the existing positive kinematics equations, the corresponding pose of the robot end effector in the robot base coordinate system can be obtained.

[0103] S7: Obtain the joint space path between the robot end effector's pose at any joint angle in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtain the coordinates of each joint space path point and its corresponding motion speed.

[0104] Specifically, optional but not limited to using joint space interpolation methods, a smooth joint angle trajectory is generated between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system. Boundary conditions for joint angles and angular velocities are set to calculate the angle change trajectory of each joint at each time point. Then, the joint angle corresponding to the angle change trajectory at each time point is calculated, and the joint angle is substituted into the existing forward kinematics equations to obtain the corresponding robot end effector pose. Finally, the angular velocity of each joint is calculated. By combining the joint angles, robot end effector poses, and joint angular velocities at each time point, the joint space path points from the robot end effector pose corresponding to any joint angle to the poses of each center coordinate can be obtained.

[0105] Preferably, the steps of obtaining the joint space path between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtaining the coordinates of each joint space path point and the corresponding motion velocity in the joint space path, may include, but are not limited to:

[0106] S71: Generate a smooth joint angle trajectory between the robot end effector in any pose in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system.

[0107] Specifically, a smooth joint angle trajectory can be generated between any pose of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, which can be selected according to the joint space interpolation method, but is not limited to. The joint space interpolation method can be selected according to common methods such as linear interpolation, polynomial interpolation, and spline interpolation.

[0108] S72: Obtain the joint angle trajectory of each joint at each time point, and obtain the corresponding robot end pose based on the corresponding joint angle trajectory;

[0109] Specifically, depending on the chosen interpolation method, the angle change of each joint in the time series can be calculated to obtain the angle trajectory of each joint over time. Then, the calculated joint angle trajectory is substituted into the forward kinematics equation to obtain the corresponding robot end-effector pose.

[0110] S73: By combining the joint angles, robot end-effector poses, joint angular velocities, and joint angular accelerations at each time point, the joint space path points are obtained.

[0111] Specifically, the corresponding joint space path points can be obtained, but are not limited to, based on the joint angles, robot end-effector poses, and joint angular velocities at each time point.

[0112] S8: Substitute the coordinates of each joint space path point and the corresponding motion velocity into the preset model to obtain the position and velocity of all joints when the robot is located at each joint space path point, thus completing the path planning.

[0113] Specifically, by inputting all joint space path points into the existing inverse kinematics model of the robot system, the joint position and angular velocity of the robot at each time point can be obtained, thereby completing the path planning.

[0114] In this embodiment, a multi-degree-of-freedom dual-arm robot refueling path planning method of the present invention is presented. A recognition network model is constructed and trained with 2D images as input and labeled images as output. The robot is then controlled to move sequentially at a set step size to acquire a 2D image of the current workpiece surface. This image is then input into the trained recognition network model to obtain a labeled image, improving the quality of the acquired image and ensuring the refueling area is centered on the image. This facilitates direct point cloud acquisition in subsequent steps, improving processing efficiency and intelligence. The point cloud of the current workpiece surface is then acquired within a radius centered on the current robot end-effector position. Based on the current workpiece surface point cloud, the center coordinates of the refueling port and vent regions in the acquisition device coordinate system, along with the normal vectors of each center coordinate, are obtained. Finally, the center coordinates of the refueling port and vent regions in the robot base coordinate system are also obtained. The system obtains the pose of each center coordinate in the robot's base coordinate system based on the current coordinates and the center coordinates and corresponding normal vectors of the filling and exhaust port areas. Then, it acquires arbitrary joint angles of the robot to obtain the corresponding pose of the robot's end effector in the robot's base coordinate system. This allows for the acquisition of the joint space path between the pose of the robot's end effector at any joint angle in the robot's base coordinate system and the poses of each center coordinate in the robot's base coordinate system. The system also acquires the coordinates and corresponding velocity of each joint path point in the joint space path. Substituting these coordinates and velocities into a preset model, the system obtains the position and velocity of all robot joints when the robot is located at each joint path point. This fully utilizes the robot's redundancy to complete path planning. This approach solves the problems of existing technologies for path planning of multi-degree-of-freedom dual-arm robots, such as the inability to fully utilize robot redundancy, low efficiency, and insufficient intelligence.

[0115] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute the above-mentioned multi-degree-of-freedom dual-arm robot injection path planning method.

[0116] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned multi-degree-of-freedom dual-arm robot refueling path planning methods.

[0117] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0118] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0119] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0120] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0121] The aforementioned computer storage medium and terminal device are created based on the multi-degree-of-freedom dual-arm robot injection path planning method. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for planning the injection path of a multi-degree-of-freedom dual-arm robot, characterized in that, include: Construct and train a recognition network model that takes 2D images as input and labeled images as output; The robot is controlled to move sequentially according to a set step size, acquire a 2D image of the current workpiece surface, and input it into the trained recognition network model to obtain an annotated image; the annotated image includes the filling port area and the venting port area; Acquire the point cloud of the current workpiece surface with the current robot end position as the center and a set distance as the radius, and obtain the center coordinates of the filling port area and the exhaust port area in the coordinate system of the acquisition device, as well as the normal vector of each center coordinate, based on the current workpiece surface point cloud. Obtain the center coordinates of the filling port area and the exhaust port area, as well as the normal vector of each center coordinate, in the robot's base coordinate system; Based on the current coordinates in the robot's base coordinate system and the center coordinates and corresponding normal vectors of the filling port area and the exhaust port area, the pose of each center coordinate in the robot's base coordinate system is obtained; Obtain the angle of any joint of the robot in space to get the corresponding pose of the robot end effector in the robot base coordinate system; Obtain the joint space path between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtain the coordinates of each joint space path point and the corresponding motion velocity in the joint space path. By substituting the coordinates of each joint space path point and its corresponding motion velocity into the preset model, the position and velocity of all joints when the robot is located at each joint space path point are obtained, thus completing the path planning.

2. The method according to claim 1, characterized in that, Identify the specific structure of the network model, including: Input layer, used to input the acquired 2D image; Convolutional layers, connected to the input layer, are used to accept the input 2D image and extract the features of the 2D image to obtain an initial feature map; Pooling layers, connected to convolutional layers, are used to downsample the initial feature map to obtain an optimized initial feature map. Activation layers, connected to pooling layers, are used to add non-linear activation functions after convolutional and fully connected layers, enhancing the non-linear expressive power of the network. Fully connected layers, connected to activation layers, are used to fuse and process extracted features, and to annotate the feature parts to generate an annotated image; The output layer, connected to the fully connected layer, is used to output the generated labeled image.

3. The method according to claim 1, characterized in that, The steps for constructing and training a recognition network model that takes 2D images as input and labeled images as output include: Construct an initial recognition network model; Select several 2D images containing the filling port and annotate the area covered by the filling port to obtain an annotated image; Each labeled image is rotated several times by a predetermined angle to obtain a subset of labeled images at different angles; Construct a training dataset based on labeled images and subsets of labeled images; The training dataset is input into the initial recognition network model to optimize its parameters, resulting in the final recognition network model.

4. The method according to claim 1, characterized in that, The steps of controlling the robot to move sequentially according to a set step size, acquiring a 2D image of the current workpiece surface, and inputting it into the trained recognition network model to obtain an annotated image include: S21: Control the robot to move at the first set step size and obtain a 2D image of the current workpiece surface; S22: Input the 2D image of the current workpiece surface into the recognition network model, determine whether the filling port area appears in the 2D image. If it appears, mark the filling port area to obtain the initial marked image and execute step S23. If it does not appear, return to step S21. S23: Obtain the center point of the filling port area and the initial annotation image, and determine whether the distance between each center point is greater than the set threshold. If it is greater, move towards the center point of the filling port area according to the second set step size until the distance between each center point is not greater than the set threshold. If it is not greater, execute step S24. S24: Obtain a 2D image of the current workpiece surface, which will be the final labeled image.

5. The method according to claim 4, characterized in that, The lengths of the first and second steps are as follows: The first step length is the diameter of the scanning range of the scanning device; the second step length is the radius of the prior filling port area; the prior filling port area is pre-marked by a person skilled in the art or / and obtained according to the workpiece model or / and pre-set by a person skilled in the art.

6. The method according to claim 1, characterized in that, The steps for obtaining the center coordinates of the filling port area and the venting port area, as well as the normal vectors of each center coordinate, in the coordinate system of the acquisition device include: Obtain the prior region of interest on the workpiece surface and determine the point cloud of the filling port region and the venting port region; The coordinates of all points in the point cloud of each region are obtained and calculated in the coordinate system of the acquisition device to obtain the coordinates of the center point of each region. Obtain the normal vector corresponding to the coordinates of the center point of each region.

7. The method according to claim 1, characterized in that, The steps to obtain the pose of each center coordinate in the robot's base coordinate system include: Construct a right-handed coordinate system with the center coordinate as the origin; Obtain the x-axis direction vector and y-axis direction vector in the right-handed coordinate system; Construct the rotation matrix between the right-handed coordinate system and the robot's base coordinate system based on the x-axis direction vector and the y-axis direction vector; The angle of the center coordinates in the robot's base coordinate system is obtained by using the rotation matrix, thus obtaining the pose of the center coordinates in the robot's base coordinate system.

8. The method according to claim 1, characterized in that, The steps for obtaining the joint space path between the pose of any joint angle of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system, and obtaining the coordinates and corresponding motion velocity of each joint space path point in the joint space path, include: A smooth joint angle trajectory is generated between the arbitrary pose of the robot end effector in the robot base coordinate system and the pose of each center coordinate in the robot base coordinate system. Obtain the joint angle trajectory of each joint at each time point, and obtain the corresponding robot end-effector pose based on the corresponding joint angle trajectory; By combining the joint angles, robot end-effector poses, joint angular velocities, and joint angular accelerations at each time point, the joint space path points are obtained.

9. A computer storage medium, characterized in that, It stores executable program code; the executable program code is used to execute the multi-degree-of-freedom dual-arm robot dispensing path planning method according to any one of claims 1-8.

10. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the multi-degree-of-freedom dual-arm robot dispensing path planning method according to any one of claims 1-8.

Images