Robotic arm teleoperation method, device, medium, program, and robot

Abstract

Embodiments of the present application disclose a kind of mechanical arm teleoperation method, equipment, medium, program and robot, wherein, method includes: the real-time visual information of target operation scene is collected;According to the real-time visual information of target operation scene, the motion planning is carried out to target mechanical arm, determine the target path and target end pose of the target mechanical arm;According to the target path and target end pose of the target mechanical arm, the target mechanical arm is controlled and target interactive object is interacted.This embodiment of the present application can simplify the operation process of mechanical arm, enhance the accuracy and reliability of mechanical arm operation, and then improve the use efficiency of mechanical arm.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of robotics, and more particularly to a method, device, medium, program, and robot for remote operation of a robotic arm. Background Technology

[0002] In many areas of human production and life, robotic arm technology has become a key supporting technology. It is widely used in industrial automation and environmental cleaning, enabling operators to complete complex tasks without having to be in dangerous environments or perform high-risk tasks, thus ensuring operational efficiency and personnel safety.

[0003] In existing technologies, teleoperation is the primary method for controlling robotic arms. Operators use specific control devices or interactive interfaces to send commands to the robotic arm and receive feedback environmental perception data and execution information, thereby constructing a closed-loop control system of "perception-decision-execution".

[0004] In the process of realizing this invention, the inventors discovered the following defects in the prior art: (1) Operators usually need to use complex interactive interfaces and cumbersome operation methods to control the robotic arm, which makes the operation interaction process complicated and seriously reduces the efficiency of the robotic arm; (2) In cluttered scenes, the target object is easily obscured or interfered with, making it difficult for the operator to accurately recognize and precisely locate it, which in turn causes a deviation between the operation intention and the actual action of the robotic arm, resulting in misoperation; (3) The prior art lacks understanding of some of the operator's operation postures, making it difficult to accurately capture subtle movement intentions, which further affects the accuracy and reliability of the robotic arm in performing complex operations. Summary of the Invention

[0005] This invention provides a method, device, medium, program, and robot for remotely operating a robotic arm, which simplifies the operation process of the robotic arm, enhances the accuracy and reliability of the robotic arm operation, and thus improves the efficiency of the robotic arm.

[0006] According to one aspect of the present invention, a method for teleoperating a robotic arm is provided, applied to a robotic arm control device, comprising:

[0007] Collect real-time visual information of the target operation scene;

[0008] Based on the real-time visual information of the target operation scenario, motion planning is performed on the target robotic arm to determine the target path and target end-effector pose of the target robotic arm;

[0009] The target robotic arm is controlled to interact with the target interactive object based on the target path and the target end-effector pose.

[0010] According to another aspect of the present invention, a robotic arm teleoperation device is provided, configured in a robotic arm control device, comprising:

[0011] The information acquisition module is used to collect real-time visual information of the target operation scene;

[0012] The motion planning module is used to perform motion planning for the target robotic arm based on real-time visual information of the target operation scenario, and to determine the target path and target end-effector pose of the target robotic arm.

[0013] The interaction module is used to control the target robotic arm to interact with the target interactive object based on the target path and target end pose of the target robotic arm.

[0014] According to another aspect of the present invention, a robotic arm control device is provided, the robotic arm control device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the robotic arm teleoperation method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the robotic arm teleoperation method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the robotic arm teleoperation method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a robot is also provided, comprising a robotic arm control device and a robotic arm, wherein the robotic arm control device controls the robotic arm using the robotic arm teleoperation method described in any one of the present invention.

[0021] This invention collects real-time visual information of the target operation scene and performs motion planning for the target robotic arm based on this information. This determines the target path and end-effector pose of the robotic arm, allowing for interaction between the robotic arm and the target interactive object. By planning the motion path and end-effector pose using real-time visual information of the target operation scene, this solution provides precise navigation for the robotic arm's operation, reducing operational errors caused by path deviations or improper postures. It solves the problems of low operational accuracy in existing technologies due to complex and chaotic operation processes, insufficient understanding of user intent, and simplifies the robotic arm's operation process, enhancing its accuracy and reliability, and ultimately improving its utilization efficiency.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the robotic arm teleoperation method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a robotic arm teleoperation method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a robotic arm teleoperation system provided in Embodiment 2 of the present invention;

[0027] Figure 4 This is a specific flowchart of a robotic arm teleoperation provided in Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of a robotic arm remote operation device provided in Embodiment 3 of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of a robotic arm control device provided in Embodiment 4 of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of a robot provided in Embodiment 5 of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," and "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a robotic arm teleoperation method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the motion path and end-effector pose of the robotic arm are determined based on real-time visual information of the operation scenario. The method can be executed by a robotic arm teleoperation device, which can be implemented by software and / or hardware and is generally integrated into an electronic device, which can be a robotic arm control device.

[0035] Correspondingly, such as Figure 1 As shown, the method includes the following operations:

[0036] S110: Collect real-time visual information of the target operation scene.

[0037] The real-time visual information of the target operation scene can be visual data about the operation environment acquired in real time by an image acquisition device when the robotic arm performs the operation task. For example, the real-time visual information of the target operation scene may include, but is not limited to, the spatial geometric information of the target interactive object and the interactive posture information of the target user. This embodiment of the invention does not limit the specific content of the real-time visual information of the target operation scene.

[0038] In this embodiment of the invention, the robotic arm performing the operation task can be designated as the target robotic arm, and the operation scene in which the target robotic arm performs the operation task can be designated as the target operation scene. Before controlling the target robotic arm to perform the operation task, the robotic arm control device can first collect real-time visual information of the target operation scene based on a camera or a visual sensor such as an RGB-D (Red-Green-Blue-Depth) camera, so as to perform motion planning for the target robotic arm based on the real-time visual information of the target operation scene.

[0039] S120. Based on the real-time visual information of the target operation scenario, perform motion planning on the target robotic arm to determine the target path and target end-effector pose of the target robotic arm.

[0040] The target path can be the motion path of the target robotic arm when performing the operation task. The target end effector pose can be the position and orientation of the end effector in three-dimensional space when the target robotic arm performs the operation task.

[0041] Correspondingly, after collecting real-time visual information of the target operation scene, the robotic arm control device can analyze the real-time visual information of the target operation scene to determine the target path and target end pose of the target robotic arm when performing the operation task, thereby ensuring that the target robotic arm completes the operation task along the optimal path at the correct angle and position.

[0042] S130. Control the target robotic arm to interact with the target interactive object according to the target path and target end pose of the target robotic arm.

[0043] The target interaction object can be any object that the target robotic arm needs to physically contact, manipulate, or interact with during the execution of its tasks. For example, in the field of industrial automation, the target interaction object may include, but is not limited to, workpieces, tools, and materials; in the field of medical robotics, the target interaction object may include, but is not limited to, medical equipment and tools; in the field of construction, the target interaction object may include, but is not limited to, building materials such as bricks and beams. This embodiment of the invention does not limit the specific field in which the target robotic arm performs its tasks or the specific type of the target interaction object.

[0044] Accordingly, after performing motion planning on the target robotic arm based on the real-time visual information of the target operation scenario, and determining the target path and end-effector pose of the target robotic arm, the robotic arm control device can convert the target path and end-effector pose of the target robotic arm into corresponding control commands, and send the control commands to the target robotic arm to control the target robotic arm to interact with the target interactive object.

[0045] This invention collects real-time visual information of the target operation scene and performs motion planning for the target robotic arm based on this information. This determines the target path and end-effector pose of the robotic arm, allowing for interaction between the robotic arm and the target interactive object. By planning the motion path and end-effector pose using real-time visual information of the target operation scene, this solution provides precise navigation for the robotic arm's operation, reducing operational errors caused by path deviations or improper postures. It solves the problems of low operational accuracy in existing technologies due to complex and chaotic operation processes, insufficient understanding of user intent, and simplifies the robotic arm's operation process, enhancing its accuracy and reliability, and ultimately improving its utilization efficiency.

[0046] Example 2

[0047] Figure 2 This is a flowchart of a robotic arm teleoperation method provided in Embodiment 2 of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, various specific optional implementation methods are given for motion planning of the target robotic arm based on real-time visual information of the target operation scene, determining the target path and target end-effector pose of the target robotic arm. Correspondingly, as... Figure 2 As shown, the method in this embodiment may include:

[0048] S210. Collect real-time visual information of the target operation scene.

[0049] S220. Generate a virtual model of the target interactive object based on the spatial geometric information of the target interactive object.

[0050] The spatial geometric information of the target interactive object can be data describing the geometric features of the target object in three-dimensional space, such as its shape, size, position, and depth. The virtual model of the target interactive object can be a digital representation used to describe the geometric features and physical properties of the target interactive object.

[0051] Specifically, after acquiring real-time visual information of the target operation scene, the robotic arm control device can accurately construct a virtual model of the target interactive object based on the spatial geometric information of the target interactive object using computer modeling technology. This virtual model can not only accurately reflect the appearance characteristics of the target interactive object, but also include its physical attributes and dynamic behavior information, thus providing a comprehensive and accurate reference for the subsequent operation of the robotic arm.

[0052] In an optional embodiment of the present invention, generating a virtual model of the target interactive object based on the spatial geometric information of the target interactive object may include: classifying the target interactive object based on the spatial geometric information of the target interactive object to obtain a classification result of the target interactive object; and generating a virtual model of the target interactive object based on the classification result of the target interactive object and the spatial geometric information.

[0053] The classification result of the target interactive object can be the final output of classifying the target interactive object into a specific category through a certain classification method or algorithm.

[0054] In this embodiment of the invention, during the process of generating a virtual model of the target interactive object based on its spatial geometric information, the spatial geometric information of the target interactive object can first be input into a machine learning model to classify the target interactive object, thereby obtaining a classification result. Further, the spatial geometric information, including the depth information of the target interactive object, can be converted into three-dimensional coordinates of the target interactive object in virtual space. Combined with the classification result of the target interactive object, a virtual model belonging to the same category and having the same physical properties as the target interactive object is generated in virtual space. This allows the target robotic arm to plan and adjust its operation strategy in advance based on the characteristics of the virtual model. Simultaneously, this virtual model can also provide a more accurate reference for the motion planning and collision detection of the target robotic arm.

[0055] Figure 3 This is a schematic diagram of the structure of a robotic arm teleoperation system provided in Embodiment 2 of the present invention. Figure 4 This is a specific flowchart of a robotic arm teleoperation process provided in Embodiment 2 of the present invention. In a specific example, the robotic arm teleoperation system can include two parts: a physical space and a digital space. The visual detection module in the physical space can use an RGB-D camera to acquire real-time visual information of the target operation scene. Furthermore, the 3D reconstruction module in the digital space can include an object recognition module and a model generation module. Specifically, the object recognition module can perform object recognition on the target interactive object based on the real-time visual information of the target operation scene to determine the classification of the target interactive object. Further, the model generation module can convert the depth information in the real-time visual information of the target operation scene into the 3D coordinates of the target interactive object in virtual space, and use the object recognition result to generate a virtual object in virtual space with the same category and physical properties as the target interactive object, thereby restoring the real 3D interactive scene.

[0056] S230. Generate the operation task of the target robotic arm based on the interaction posture information of the target user and the virtual model of the target interaction object.

[0057] The target user can be the user who remotely operates the target robotic arm. The target user's interaction posture information can include their body posture, gestures, body language, and other interaction-related bodily state information during human-computer interaction. For example, the target user's interaction posture information may include, but is not limited to, the posture of their arm and the degree of finger bending. The target robotic arm's operation task can be the specific work or action that the target robotic arm needs to perform in the target operation scenario.

[0058] Specifically, after generating a virtual model of the target interactive object based on its spatial geometric information, the robotic arm control device can comprehensively analyze the interactive posture information of the target user and the virtual model of the target interactive object to generate the target robotic arm's operation task.

[0059] In an optional embodiment of the present invention, the step of generating the operation task of the target robotic arm based on the interaction posture information of the target user and the virtual model of the target interaction object may include: performing intent recognition on the interaction posture information of the target user to determine the interaction operation intent of the target user; and generating the operation task of the target robotic arm based on the interaction operation intent of the target user and the virtual model of the target interaction object.

[0060] Among them, the interactive operation intent can be the target user's target or desired behavior conveyed to the target robotic arm through interactive posture information.

[0061] Specifically, when generating the target robotic arm's operational tasks based on the target user's interactive posture information and the virtual model of the target interaction object, the robotic arm control device first uses artificial intelligence to monitor and analyze the target user's interactive posture information in real time, understanding the target user's intentions and needs, thereby determining the target user's interactive operation intentions. Furthermore, the robotic arm control device can generate the target robotic arm's operational tasks based on the target user's interactive operation intentions and the virtual model of the target interaction object.

[0062] like Figure 3As shown, in a specific example, the digital space of the robotic arm teleoperation system can also include an intent understanding module. This intent understanding module includes a hand recognition module, a gesture understanding module, and a layout understanding module. Specifically, the hand recognition module can use a machine learning model to extract features such as the shape, contour, and key points of the hand from the target user's interactive posture information. Further, the gesture understanding module can classify the hand based on the features extracted by the hand recognition module to determine whether the target user is facing the RGB-D camera with their palm or the back of their hand. Further, combining the hand orientation, the extracted key points of the finger joints, finger direction, and curvature can be mapped to corresponding actions based on preset rules. For example, a palm facing outwards and fingers pointing downwards represents a pushing action; fingers pointing downwards and bent together represent a grasping action; fingers rotating left and right while maintaining the same curvature represents a twisting action. After obtaining the target user's corresponding actions based on the gesture understanding module, the layout understanding module can use the machine learning module to classify the real-time visual information of the RGB-D camera image input, i.e., the target operation scene, to determine the target user's interactive operation intent. The gesture-based control method described above makes operation more intuitive and natural, eliminating the need for complex control interfaces and cumbersome operation steps. Compared to traditional robotic arm teleoperation methods, it simplifies the operation process, reduces the learning curve, and improves the accuracy and efficiency of robotic arm operation.

[0063] Optionally, the layout understanding module can also embed prior knowledge of interactions into the classification model by introducing CAM (Class Activation Map) activation constraints. Specifically, firstly, a CNN (Convolutional Neural Network) classification model can be trained using training samples to enable the model to classify image inputs from an RGB-D camera. Further, the trained classification model can be used to perform global average pooling on the feature maps of the last convolutional layer to obtain channel weights, which can then be used to obtain CAM maps for specific categories through weighting operations. During training, for images from an RGB-D camera, i.e., real-time visual information of the target operation scene:

[0064] F k =∑ x，y f k (x, y)

[0065] Among them, F k f is the global average pooling result of the k-th channel on the last convolutional layer. k (x, y) is the activation value of the feature map at position (x, y) on the k-th channel of the last convolutional layer.

[0066] Therefore, for image category c, the input to the softmax activation layer is:

[0067]

[0068] Among them, S c For image category c, the input to the softmax activation layer, For F k For the weights of image category c. If F k As scores for different image categories, then:

[0069]

[0070] Furthermore, M c (x, y) is a CAM defined as class c:

[0071]

[0072] Sc=∑ x，y M c (x,y)

[0073] Among them, M c (x, y) represents the importance of the activation value at (x, y) to the image category c, M c The larger (x, y) is, the more important the feature at that position is to category c.

[0074] Optionally, activation constraints can be set based on prior knowledge of the interactions. These constraints can be defined based on features such as the distribution, size, and shape of the CAM. For example, the classification model can be set to give higher attention to specific regions of a certain category. In grasping operations, this typically means the target robotic arm should grasp object A placed on object B, rather than object B itself, or it should place the object in a region of the same category as the object. Furthermore, activation constraints can be incorporated into the training of the classification model. By adding corresponding penalty terms to the loss function, the classification model can be optimized, allowing it to consider the influence of prior knowledge during training and try to satisfy the activation constraints. This controls the target robotic arm to perform specific actions and prevents collisions or erroneous operations.

[0075] S240. Based on the operation task of the target robotic arm, perform motion planning on the target robotic arm to determine the target path and target end-effector pose of the target robotic arm.

[0076] Specifically, after generating the target robotic arm's operation task based on the target user's interaction posture information and the virtual model of the target interactive object, the robotic arm control device can analyze the operation task to determine its specific content. Furthermore, based on the specific content of the operation task, the robotic arm control device can perform motion planning to determine the optimal path for the target robotic arm from its current position to the target position, and use this path as the target path. Simultaneously, based on the shape, size, and required operation type of the target interactive object, the robotic arm control device can accurately calculate the target end effector pose of the target robotic arm to ensure that the end effector can interact with the target interactive object in the correct posture and position.

[0077] In an optional embodiment of the present invention, the step of performing motion planning on the target robotic arm according to the operation task of the target robotic arm to determine the target path and target end-effector pose of the target robotic arm may include: determining the target path and a first end-effector pose of the target robotic arm according to the operation task of the target robotic arm; matching the operation task of the target robotic arm with the end-effector pose database of the target robotic arm to obtain a second end-effector pose of the target robotic arm; and determining the target end-effector pose of the target robotic arm according to the first end-effector pose and the second end-effector pose.

[0078] The first end-effector pose can be the pose of the end effector of the target robotic arm determined during the path planning process. The end-effector pose database can be a collection of data storing the poses required by the end effector of the robotic arm in different operational tasks. The second end-effector pose can be the pose of the end effector of the target robotic arm that matches the operational task of the target robotic arm in the end-effector pose database.

[0079] Specifically, when performing motion planning for the target robotic arm based on its operational tasks, and determining the target path and target end-effector pose, the target path and first end-effector pose of the target robotic arm can first be determined based on its operational tasks.

[0080] In a specific example, such as Figure 3 and Figure 4 As shown, the digital space of the robotic arm teleoperation system can also include a motion planning module. This motion planning module can include a trajectory planning module and an end-effector pose fusion module. Specifically, the trajectory planning module can first determine the current and target positions of the robotic arm based on real-time visual information of the target operation scene or by the target user. Furthermore, it can use heuristic methods based on Riemannian motion strategies to search for the optimal path from the current position to the target position, thereby improving the robotic arm's motion accuracy and efficiency, and enhancing its ability to adapt to complex motion planning problems.

[0081] For example, if the path of the target robotic arm from its current position to its target position is represented as q(t), then the Riemann metric of the path can be expressed as:

[0082]

[0083] Where g(i,j) is the Riemann metric matrix of the path. Let be the local metric function for the path, and q be the position vector of the end effector joint of the target robotic arm, i.e., the path of the target robotic arm from its current position to its target position. Let be the derivative of the joint velocity of the end effector of the target robotic arm, i.e., the joint acceleration.

[0084] Furthermore, the end-effector pose matching module can match the target robotic arm's operation task with the target robotic arm's end-effector pose database, thereby filtering out a second end-effector pose that matches the target robotic arm's operation task from the end-effector pose database. For example, if the target robotic arm's operation task is to unscrew a horizontal bottle cap, then the second end-effector pose could be horizontally holding a cylindrical object.

[0085] Optionally, an end-effector pose database can be pre-established based on the target interaction object obtained by the intent understanding module and the recognized gestures. Understandably, if the gestures recognized by the intent understanding module do not match the target user's interaction posture information, the data can be manually removed.

[0086] The size of the end-effector pose database can be:

[0087] S = N ob *N op

[0088] Where S is the capacity of the end-effector pose data, and N ob N represents the number of target interaction objects. op The number of target end poses.

[0089] After obtaining the first and second end-effector poses, the end-effector pose fusion module can use these poses as input to a DQN (Deep Q Network) to obtain the fused target end-effector pose. This approach can effectively improve the success rate of the robotic arm in grasping objects while correctly understanding the target user's operational intent, and reduce the possibility of misoperation.

[0090] In an optional embodiment of the present invention, after performing motion planning on the target robotic arm according to the operation task of the target robotic arm and determining the target path and target end-effector pose of the target robotic arm, the method may further include: performing interactive operation simulation based on real-time visual information of the target operation scene, the target path of the target robotic arm, and the target end-effector pose to obtain interactive operation simulation results; if it is determined that there is a collision conflict in the interactive operation simulation results, the operation of performing motion planning on the target robotic arm according to the operation task of the target robotic arm and determining the target path and target end-effector pose of the target robotic arm is re-executed until it is determined that there is no collision conflict in the operation simulation results.

[0091] The interactive operation simulation results can be the output results obtained by simulating the execution process of the target robotic arm's operation task. Collision conflicts can be the situations where the target robotic arm collides with other objects while performing the operation task along the target path in the target end-effector pose.

[0092] During the execution of the target robotic arm's task, objects in the target operation scene may shift or new objects may be added. Therefore, after determining the target path and end-effector pose of the target robotic arm, interactive operation simulations can be performed based on real-time visual information of the target operation scene, the target path of the target robotic arm, and the end-effector pose to ensure that the target robotic arm does not collide with obstacles in the target operation environment while performing the task along the target path. If collision conflicts exist in the interactive operation simulation results, the motion planning of the target robotic arm can be re-executed according to the target robotic arm's task until no collision conflicts are found in the operation simulation results.

[0093] Optionally, after determining the target path for the robotic arm, optimization algorithms such as gradient descent can be used to optimize the searched path to improve its quality. Furthermore, paths can be evaluated and compared using metrics such as path length, time efficiency, and stability to select the optimal path as the target path for the robotic arm.

[0094] S250. Control the target robotic arm to interact with the target interactive object according to the target path and target end pose of the target robotic arm.

[0095] In a specific example, such as Figure 3 and Figure 4As shown, the digital space of the robotic arm teleoperation system can also include a virtual communication module, and the physical space can also include a robot operation module. Specifically, the virtual communication module can include, but is not limited to, wireless communication interfaces and wired communication interfaces, mainly including WIFI (Wireless Fidelity), Bluetooth, TCP / IP (Transmission Control Protocol / Internet Protocol), and UDP / IP (User Datagram Protocol / Internet Protocol), etc., which can ensure reliable real-time data communication between the digital space and the physical operation space, and convert the processed gestures and intentions into control commands and send them to the robot operation module to make the target robotic arm perform corresponding actions.

[0096] This invention collects real-time visual information of a target operation scene, including the spatial geometry of the target interactive object and the interactive posture information of the target user, and generates a virtual model of the target interactive object based on its spatial geometry. Further, it generates an operation task for the target robotic arm based on the interactive posture information of the target user and the virtual model of the target interactive object. Then, it performs motion planning on the target robotic arm based on the operation task, determining the target path and end-effector pose, and controlling the interaction between the target robotic arm and the target interactive object based on the target path and end-effector pose. This solution plans the motion path and end-effector pose of the target robotic arm using real-time visual information of the target operation scene, providing precise navigation for the robotic arm's operation. This reduces operational errors caused by path deviations or improper postures, solving the problems of low operational accuracy in existing technologies due to complex and chaotic operation processes, insufficient understanding of user intent, and other issues. It simplifies the robotic arm's operation process, enhances the accuracy and reliability of robotic arm operation, and ultimately improves the efficiency of robotic arm use.

[0097] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information (such as facial information, image information, etc.) involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0098] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0099] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0100] Example 3

[0101] Figure 5 This is a schematic diagram of a robotic arm teleoperation device provided in Embodiment 3 of the present invention, as shown below. Figure 5 As shown, the device is configured in a robotic arm control system and includes: an information acquisition module 310, a motion planning module 320, and an interaction module 330, wherein:

[0102] The information acquisition module 310 is used to acquire real-time visual information of the target operation scene.

[0103] The motion planning module 320 is used to perform motion planning for the target robotic arm based on the real-time visual information of the target operation scenario, and to determine the target path and target end-effector pose of the target robotic arm.

[0104] The interaction module 330 is used to control the target robotic arm to interact with the target interactive object according to the target path and target end pose of the target robotic arm.

[0105] This invention collects real-time visual information of the target operation scene and performs motion planning for the target robotic arm based on this information. This determines the target path and end-effector pose of the robotic arm, allowing for interaction between the robotic arm and the target interactive object. By planning the motion path and end-effector pose using real-time visual information of the target operation scene, this solution provides precise navigation for the robotic arm's operation, reducing operational errors caused by path deviations or improper postures. It solves the problems of low operational accuracy in existing technologies due to complex and chaotic operation processes, insufficient understanding of user intent, and simplifies the robotic arm's operation process, enhancing its accuracy and reliability, and ultimately improving its utilization efficiency.

[0106] Optionally, the real-time visual information of the target operation scene includes the spatial geometric information of the target interactive object and the interactive posture information of the target user; the motion planning module 320 is specifically used to: generate a virtual model of the target interactive object based on the spatial geometric information of the target interactive object; generate the operation task of the target robotic arm based on the interactive posture information of the target user and the virtual model of the target interactive object; perform motion planning on the target robotic arm based on the operation task of the target robotic arm, and determine the target path and target end-effector pose of the target robotic arm.

[0107] Optionally, the motion planning module 320 is further configured to: classify the target interactive object according to the spatial geometric information of the target interactive object, and obtain the classification result of the target interactive object; and generate a virtual model of the target interactive object according to the classification result of the target interactive object and the spatial geometric information.

[0108] Optionally, the motion planning module 320 is further configured to: perform intent recognition on the interactive posture information of the target user to determine the interactive operation intent of the target user; and generate the operation task of the target robotic arm based on the interactive operation intent of the target user and the virtual model of the target interactive object.

[0109] Optionally, the motion planning module 320 is further configured to: determine the target path of the target robotic arm and the first end-effector pose of the target robotic arm based on the operation task of the target robotic arm; match the operation task of the target robotic arm with the end-effector pose database of the target robotic arm to obtain the second end-effector pose of the target robotic arm; and determine the target end-effector pose of the target robotic arm based on the first end-effector pose and the second end-effector pose.

[0110] Optionally, the above device may further include an interactive operation simulation module, used for: performing interactive operation simulation based on the real-time visual information of the target operation scene, the target path of the target robotic arm, and the target end-effector pose, to obtain interactive operation simulation results; if it is determined that there is a collision conflict in the operation simulation results, re-execute the operation of motion planning for the target robotic arm based on the operation task of the target robotic arm, and determining the target path and target end-effector pose of the target robotic arm, until it is determined that there is no collision conflict in the operation simulation results.

[0111] The above-described robotic arm teleoperation device can execute the robotic arm teleoperation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the robotic arm teleoperation method provided in any embodiment of the present invention.

[0112] Since the robotic arm teleoperation device described above is a device capable of executing the robotic arm teleoperation method in the embodiments of the present invention, those skilled in the art can understand the specific implementation methods and various variations of the robotic arm teleoperation device in this embodiment based on the robotic arm teleoperation method described in the embodiments of the present invention. Therefore, how the robotic arm teleoperation device implements the robotic arm teleoperation method in the embodiments of the present invention will not be described in detail here. Any device used by those skilled in the art to implement the robotic arm teleoperation method in the embodiments of the present invention falls within the scope of protection of this application.

[0113] Example 4

[0114] Figure 6 A schematic diagram of a robotic arm control device 10, which can be used to implement embodiments of the present invention, is shown. The robotic arm control device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The robotic arm control device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0115] like Figure 6 As shown, the robotic arm control device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the robotic arm control device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] Multiple components in the robotic arm control device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the robotic arm control device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as robotic arm teleoperation methods.

[0118] In some embodiments, the robotic arm teleoperation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the robotic arm control device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the robotic arm teleoperation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the robotic arm teleoperation method by any other suitable means (e.g., by means of firmware).

[0119] Optionally, the robotic arm teleoperation method, applied to a robotic arm control device, may include: acquiring real-time visual information of the target operation scene; performing motion planning on the target robotic arm based on the real-time visual information of the target operation scene to determine the target path and target end-effector pose of the target robotic arm; and controlling the target robotic arm to interact with a target interactive object based on the target path and target end-effector pose of the target robotic arm.

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] To provide interaction with the user, the systems and techniques described herein can be implemented on a robotic arm control device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the robotic arm control device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0125] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0126] Example 5

[0127] Figure 7 This is a schematic diagram of the structure of a robot provided in Embodiment 5 of the present invention, as shown below. Figure 7 As shown, this embodiment of the invention provides a robot 500, which includes a robotic arm control device 510 and a robotic arm 520. The robotic arm control device 510 controls the robotic arm 520 using the robotic arm teleoperation method provided in any embodiment of the invention.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for remotely operating a robotic arm, characterized in that, Applications in robotic arm control equipment, including: Collect real-time visual information of the target operation scene; Based on the real-time visual information of the target operation scenario, motion planning is performed on the target robotic arm to determine the target path and target end-effector pose of the target robotic arm; The target robotic arm is controlled to interact with the target interactive object based on the target path and target end-effector pose; The real-time visual information of the target operation scene includes the spatial geometric information of the target interactive object and the interactive posture information of the target user; the step of performing motion planning for the target robotic arm based on the real-time visual information of the target operation scene to determine the target path and target end-effector pose of the target robotic arm includes: A virtual model of the target interactive object is generated based on the spatial geometric information of the target interactive object; The target robotic arm's operation task is generated based on the target user's interaction posture information and the virtual model of the target interaction object; Based on the target robotic arm's operational task, motion planning is performed on the target robotic arm to determine its target path and target end-effector pose. The step of generating a virtual model of the target interactive object based on the spatial geometric information of the target interactive object includes: The target interactive object is classified according to its spatial geometric information to obtain the classification result of the target interactive object; A virtual model of the target interactive object is generated based on the classification result of the target interactive object and the spatial geometric information.

2. The method according to claim 1, characterized in that, The step of generating the operation task of the target robotic arm based on the interaction posture information of the target user and the virtual model of the target interaction object includes: The interaction posture information of the target user is used to identify the user's interaction intention; The target robotic arm's operation tasks are generated based on the target user's interactive operation intentions and the virtual model of the target interactive object.

3. The method according to claim 1, characterized in that, The step of performing motion planning for the target robotic arm based on its operational task, and determining the target path and end-effector pose of the target robotic arm, includes: The target path and the first end effector pose of the target robotic arm are determined based on the operation task of the target robotic arm. The operation task of the target robotic arm is matched with the end-effector pose database of the target robotic arm to obtain the second end-effector pose of the target robotic arm; The target end-effector pose is determined based on the first end-effector pose and the second end-effector pose.

4. The method according to claim 1, characterized in that, After performing motion planning on the target robotic arm according to its operational task and determining the target path and end-effector pose, the method further includes: Based on the real-time visual information of the target operation scenario, the target path of the target robotic arm, and the pose of the target end effector, an interactive operation simulation is performed to obtain the interactive operation simulation result. If a collision conflict is found in the simulation results of the interactive operation, the operation of motion planning for the target robotic arm based on the operation task of the target robotic arm is re-executed to determine the target path and target end pose of the target robotic arm until it is determined that there is no collision conflict in the simulation results of the operation.

5. A robotic arm control device, characterized in that, The robotic arm control device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the robotic arm teleoperation method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the teleoperation method of the robotic arm as described in any one of claims 1-4.

7. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the robotic arm teleoperation method as described in any one of claims 1-4.

8. A robot comprising the robotic arm control device and robotic arm as described in claim 5, wherein, The robotic arm control device uses the robotic arm teleoperation method according to any one of claims 1-4 to control the robotic arm.

Images