Control method of mechanical arm and mechanical arm system
By obtaining the modal information and task recognition model of the robotic arm's work area, appropriate neural network models are selected, which solves the problem of insufficient operation efficiency and reliability of the robotic arm in different task scenarios, and realizes autonomous and reliable automatic operation.
Patent Information
- Application Number
- CN202411380768.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-09-30
AI Technical Summary
It is difficult for robotic arms to independently select appropriate neural network models in different task scenarios, resulting in insufficient operational efficiency and reliability.
By obtaining the modal information of the robotic arm work area, the task recognition model is used to determine the task scenario, and the target neural network model is filtered out from the preset set of neural network models, and the robotic arm is controlled to perform operations based on the model.
It realizes autonomous and reliable automatic operation of the robotic arm in different task scenarios, improving operation efficiency and reliability.
Smart Images

Figure CN118893635A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robotic arms, and in particular to a robotic arm control method and a robotic arm system. Background Art
[0002] In recent years, with breakthroughs in large-scale models and AI technology, the application of general artificial intelligence (AGI) in robotics (or robotic arms) has become increasingly widespread. By training robots (or robotic arms) with extensive models, robots (or robotic arms) in real-world environments can be imbued with intelligence, enabling them to interact with their environment like humans and perform a variety of tasks. Summary of the Invention
[0003] An embodiment of the present application provides a method for controlling a robotic arm, including: Acquire at least one modal information of a working area where the robot arm is located, where the working area allows placement of at least one target object; Using a task identification model to identify the at least one modal information to determine a task scenario; According to the task scenario, a target neural network model is selected from a preset neural network model set; Based on the target neural network model, the robotic arm is controlled to perform an operation on the at least one target object.
[0004] An embodiment of the present application provides a robotic arm system applying a robotic arm control method, including: A robotic arm includes M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to manipulate a target object; An image acquisition device configured to acquire image information containing a target object; The control device is configured to obtain operation instructions for the robotic arm, determine the task scenario through the operation instructions, filter out the target neural network model corresponding to the task scenario from the preset neural network model set, and control the robotic arm to operate on the target object based on the target neural network model.
[0005] The beneficial effect of the present application is: after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task recognition model to identify the at least one modal information to determine the task scenario, and then screens out a target neural network model from a preset neural network model set according to the task scenario, and controls the robotic arm to operate on at least one target object in the working area based on the target neural network model, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0007] Figure 1 1 is a schematic structural diagram of an embodiment of a robotic arm system provided in an embodiment of the present application; Figure 2 This is a flow chart of an embodiment of a method for controlling a robotic arm provided in an embodiment of the present application; Figure 3 1 is a schematic structural diagram of an embodiment of a control device for a robotic arm provided in an embodiment of the present application; Figure 4 1 is a schematic structural diagram of an embodiment of a control device for a robotic arm provided in an embodiment of the present application; Figure 5 It is a structural diagram of an embodiment of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0008] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.
[0009] Combine Figure 1 The robotic arm system 100 may include a robotic arm 110, an image acquisition device 120, and a control device 130. The robotic arm 110 described in this application may refer to a traditional mechanical structure fixed at a certain workstation to perform certain functions, such as a collaborative robotic arm, or may refer to other emerging mechanical structures (or parts thereof), such as a humanoid robot (or its arm).
[0010] The robotic arm 110 may include M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to operate a target object.
[0011] Among them, in this application, joint motors are generally used to enable the joints of the robot (or robotic arm) to have the ability to move. Joint motors can include brushless DC motors, stepper motors, DC motors, servo motors, steering gears, etc. The selection depends on application requirements, such as the required torque, speed, accuracy, and power efficiency. Furthermore, the degree of freedom is the minimum number of coordinates required to determine the position of a system in space. Among them, m can be a natural number greater than zero, such as m is 4, 6 or other values. For example: the end of the robotic arm can move completely freely in the air, and at least six quantities (such as rectangular coordinates x, y, z and three angles θ, φ, ρ) are required to determine its position, that is, the end has 6 degrees of freedom in the air.
[0012] The actuator can be a gripper, a suction cup, or other mechanism that can perform the target operation. The actuator's drive or control type can include electric, pneumatic, hydraulic, etc.
[0013] The target object can be understood as the object to be operated by the robotic arm system 100 when performing the target task. For example, if the target task is folding clothes, the target object can include at least one piece of clothing to be folded, or if the target task is washing dishes, the target object can include at least one dish to be washed, and so on.
[0014] The image acquisition device 120 may acquire image information containing the target object. The image information may be of various types, such as RGB images, depth images, infrared images, 3D point cloud information, or images in other color spaces or types.
[0015] The control device 130 may be a server or a terminal. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud storage, and cloud computing services such as big data and artificial intelligence platforms. The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.
[0016] Furthermore, the control device 130 is configured to obtain operating instructions for the robotic arm, determine the task scenario through the operating instructions, and filter out the target neural network model corresponding to the task scenario from the preset neural network model set, so as to control the robotic arm to perform operations on the target object based on the target neural network model. For details, see the relevant description below.
[0017] The operation instruction can be understood as an instruction that triggers the robotic arm to perform an operation on the target object, and the operation instruction can be generated through image information.
[0018] In some embodiments, the robotic arm system 100 may further include a voice acquisition device, which may include a microphone or other device capable of acquiring audio or voice information. The voice acquisition device may be configured to acquire the user's voice information. After the user's voice information is acquired by the voice acquisition device, operation instructions may be generated based on the voice information.
[0019] In some embodiments, combined Figure 1 , the embodiment of the present application can provide a control method for a robotic arm of the robotic arm system 100, such as Figure 2 As shown, the specific process of the robot arm control method can be as follows: 210. Obtain at least one modal information of the working area where the robotic arm is located.
[0020] Among them, the working area allows the placement of at least one target object. The target object can be understood as an object placed in the working area to be operated by the robotic arm. For example, if an apple is placed in the working area and the robotic arm can cut or peel the apple, then the apple is the target object. Furthermore, the modal information can include various types of images, videos, audio or text, etc. Accordingly, the image information of the area where the robotic arm is located can be collected by an image acquisition device to obtain modal information under at least one image modality, the audio information of the area where the robotic arm is located can be collected by an audio acquisition device to obtain modal information under an audio modality, and various types of sensors can be used to collect information of the area where the robotic arm is located to obtain at least one modal information, and so on.
[0021] The method of triggering the acquisition of at least one modal information of the robot arm in the working area may include: acquiring an operating instruction for the robot arm, and acquiring at least one modal information of the area where the robot arm is located in response to the operating instruction. The operating instruction may be understood as instruction information that triggers the robot arm to start working in the working area.
[0022] In some embodiments, a method for obtaining an operation instruction for a robotic arm may include: displaying a task list, the task list including at least one preset task; determining task information in response to a selection operation on the task list; and generating an operation instruction based on the task information. Preset tasks may be pre-set tasks that the robotic arm can perform, such as "cooking tasks," "cleaning tasks," and "carrying tasks." Furthermore, there are various ways to display the task list, such as displaying a robotic arm startup page including a task selection control; and displaying a task selection interface including the task list in response to a trigger operation on the task selection control.
[0023] After the task list is displayed, the user can select the task list by clicking or selecting it with the mouse. Then, in response to the selection operation on the task list, the task information is determined. The task information can be understood as information containing the selected task, for example, it can include the task identifier, task attribute information, task start time, task stop time, task preset result information, or other task-related information.
[0024] In some embodiments, the method of obtaining the operation instruction for the robotic arm may further include: receiving a control request for the robotic arm, the control request carrying task information corresponding to the robotic arm, and generating the operation instruction based on the task information. The control request may be understood as a request to control the robotic arm.
[0025] Among them, there can be many ways to trigger a control request for the robotic arm in the robotic arm system. For example, corresponding trigger controls such as buttons can be set on the robotic arm system. For example, a button corresponding to the "cooking task", a button corresponding to the "cleaning task", and a button corresponding to the "carrying task" can be set. Then, when it is detected that the user presses the button corresponding to the "cleaning task", a corresponding robotic arm control request is generated, and the control request carries the task information corresponding to the cleaning task, and so on.
[0026] In some embodiments, the method of obtaining operation instructions for the robotic arm may also include: obtaining voice information for the robotic arm, performing voice recognition on the voice information to obtain task information, and generating operation instructions based on the task information.
[0027] The robotic arm system can be equipped with an audio acquisition device to collect voice information about the robotic arm. After obtaining the voice information about the robotic arm, the voice information can be recognized using a voice recognition model or other methods to obtain task information.
[0028] 220. Use a task identification model to identify at least one modal information to determine a task scenario.
[0029] The task recognition model can understand and identify the task or scenario corresponding to the target object, such as a CNN, RNN, DNN, LSTM, transformer, or other models capable of task or scenario recognition. Furthermore, the task scenario can be a task or scenario that requires working on the target object. For example, if the target object is a bowl, the task scenario can include washing the bowl, moving the bowl, or other tasks or scenarios involving working on the bowl.
[0030] There are many ways to use the task identification model to identify at least one modal information to determine the task scenario, which can be as follows: For example, a task recognition model can be used to identify the object information of the target object in at least one modal information. The object information can be understood as information indicating the object type, object identification or other attributes of the target object. For example, taking the target object as a bowl, the object information can include the type of bowl, the identification of the bowl, the color of the bowl, the size of the bowl, the position of the bowl in the work area or other attribute information, etc. Specifically, the task recognition model can be used to identify each modal information separately to obtain at least one initial object information of the target object, and the initial object information can be fused to obtain the object information of the target object; or, the task recognition model corresponding to each modal information can be screened out from the preset task recognition model set, and the task recognition model can be used to perform object recognition on the corresponding modal information to obtain the initial object information corresponding to each modal information, and the initial object information can be fused to obtain the object information of the target object, and so on.
[0031] (1) When there is only one target object, the object information can be used as the first object information, and task scenarios can be screened out from a preset task scenario set based on the first object information. There are various ways to screen out task scenarios from a preset task scenario set based on the first object information. For example, at least one candidate task scenario corresponding to the first object information can be screened out from the preset task scenario set, and then, based on the operation instruction, the task scenario corresponding to the target object can be screened out from the candidate task scenarios. Alternatively, the object identifier or type of the target object can be extracted from the first object information, and then the task scenario corresponding to the object identifier or type can be screened out from the preset task scenario set as the task scenario corresponding to the target object, and so on.
[0032] Among them, there can be many ways to filter out at least one candidate task scene corresponding to the first object information in the preset task scene set. For example, taking the first object information indicating that the target object is a bowl as an example, at least one candidate task scene related to the bowl can be filtered out in the preset task scene set. For example, it can include cleaning tasks (washing dishes), carrying tasks (placing / carrying / moving the bowl to a designated location), or placing specific items in the bowl and other candidate task scenes.
[0033] After filtering out at least one candidate task scenario, the task scenario corresponding to the target object can be filtered out from the candidate task scenarios based on the task instructions. There are many ways to filter out the task scenario corresponding to the target object from the candidate task scenarios based on the task instructions. For example, the job type of the job instruction can be determined, and the task scenario corresponding to the job type can be filtered out from the candidate task scenarios. For example, if the target object is a bowl, if the job type is cleaning, then washing dishes can be filtered out from the candidate task scenarios as a task scenario. If the job type is carrying, then carrying bowls can be filtered out from the candidate task scenarios as a task scenario, and so on.
[0034] (2) When there are multiple target objects, the object information can be used as the second object information. Based on the second object information, the target objects are compared, and based on the comparison results and the operation instructions, the task scenarios are screened out from the preset task scenario set.
[0035] Among them, there can be multiple ways to compare the target objects based on the second object information. For example, the current object information corresponding to each target object can be identified in the second object information, and the current object information can be compared to obtain a comparison result. The comparison result indicates whether the target objects are the same type of objects. For example, the target objects include large bowls, small bowls and plates. At this time, the large bowls and small bowls can belong to the same type of objects, but the large bowls and plates or the small bowls and plates do not belong to the same type of objects.
[0036] After comparing the target objects based on the second object information, task scenarios can be screened out from a preset set of task scenarios based on the comparison results and the work instructions. There are multiple ways to screen out task scenarios from a preset set of task scenarios based on the comparison results and the work instructions. For example, when the target objects are objects of the same category, task scenarios are screened out from a preset set of task scenarios based on the second object information. When the target objects are objects of different categories, candidate task scenarios corresponding to each target object are screened out from a preset set of task scenarios based on the second object information, and the task scenario is determined from the candidate task scenarios based on the work instructions.
[0037] When the target objects are objects of the same category, a candidate task scene of any target object is screened out from a preset task scene set according to the second object information, and the candidate task scene is used as the task scene.
[0038] Specifically, when the target objects are of different categories, candidate task scenarios corresponding to each target object can be screened from a preset set of task scenarios based on the second object information. When the work instruction is generated based on the task information, the task scenario is screened from the candidate task scenarios based on the task information. When the work instruction is generated based on the object detection results in the work area, attribute information of the candidate task scenarios is obtained and, based on the attribute information, the candidate task scenarios are sorted to obtain the task scenario.
[0039] Among them, there can be many ways to filter out task scenarios from candidate task scenarios based on task information. For example, the task scenario identifier or type can be identified in the task information, and the candidate task scenarios corresponding to the task scenario identifier or type can be filtered out from the candidate task scenarios to obtain the task scenario. Alternatively, the execution priority of the candidate task scenarios can be determined based on the task information, and the candidate task scenario with the highest execution priority can be filtered out from the candidate task scenarios as the task scenario. Alternatively, the operation order of the robot arm for each target object can be determined based on the task information, and the candidate task scenarios corresponding to the target object can be sorted based on the operation order. Based on the sorting result, the task scenario can be filtered out from the candidate task scenarios, and so on.
[0040] The execution priority can be understood as the priority of the target task within the execution task scenario. Based on the task information, there are multiple ways to determine the execution priority of candidate task scenarios. For example, if the candidate task scenarios include washing vegetables, cutting vegetables, and plating, when the task information indicates that vegetables need to be washed, the execution priority of washing vegetables can be set to the highest. When the task information indicates that vegetables need to be cut, the execution priority of cutting vegetables can be set to the highest, and so on.
[0041] Among them, based on the task information, there can be multiple ways to determine the operation order of the robot arm for the target objects. For example, taking the target objects including objects A, B and C as an example, the task information instructs the robot arm to operate on object A first, then object B, and finally object C. At this time, the operation order can include objects ABC.
[0042] After determining the robot arm's operating sequence for the target object, the candidate task scenarios corresponding to the target object can be sorted based on the operating sequence. Based on the sorting results, task scenarios can be selected from the candidate task scenarios. For example, if the operating sequence is objects ABC, the sorting results may include the candidate task scenario corresponding to object A, the candidate task scenario corresponding to object B, and the candidate task scenario corresponding to object C. In this case, the candidate task scenario corresponding to object A can be selected as the task scenario to be executed.
[0043] Among them, the attribute information of the candidate task scenario can be understood as information indicating at least one attribute of the candidate task scenario, for example, it can include the identification, type, execution condition, execution authority, execution priority or other attribute-related information of the candidate task scenario. Based on the attribute information, there can be multiple processes for sorting the candidate task scenarios. For example, the execution priority can be extracted from the attribute information, and the candidate task scenarios can be sorted based on the execution priority, and based on the sorting result, the task scenarios can be screened out from the candidate task scenarios. Alternatively, the execution condition of the candidate scenario task can be determined from the attribute information, and based on the execution condition, the execution priority of the candidate scenario task can be determined, and the candidate task scenarios can be sorted based on the execution priority, and based on the sorting result, the task scenarios can be screened out from the candidate task scenarios, and so on.
[0044] Among them, there are many ways to determine the execution priority of candidate task scenarios based on the execution conditions. For example, taking the execution condition of candidate task scenario A (cooking) as starting execution after candidate task scenario B (washing vegetables) is completed, at this time, it can be determined that the execution priority of candidate task scenario B is greater than that of candidate task scenario A, and so on, thereby determining the execution priority of each candidate task scenario, and so on.
[0045] 230. According to the task scenario, select the target neural network model from the preset neural network model set.
[0046] The preset neural network model set can be understood as a pre-set trained neural network model. The trained neural network model can be obtained by training using the above model training method.
[0047] There are multiple ways to select a target neural network model from a preset neural network model set according to the task scenario, which can be as follows: For example, the corresponding target task can be determined according to the task scenario, and the neural network model corresponding to the target task can be screened out from the preset neural network model set to obtain the target neural network model.
[0048] Among them, there are many ways to determine the corresponding target task according to the task scenario. For example, a task set corresponding to a preset neural network model set can be obtained, and the tasks corresponding to the task scenario can be filtered out from the task set to obtain the target task.
[0049] The target task can be understood as the task that the robot arm needs to perform in the task scenario. For example, if the task scenario is washing dishes, the target task for that task scenario can be washing dishes. Or, if the task scenario is carrying objects, the target task for that task scenario can be carrying objects, and so on.
[0050] After determining the corresponding target task, the neural network model corresponding to the target task can be screened out from the preset neural network model set. There are many ways to screen out the neural network model corresponding to the target task from the preset neural network model set. For example, the task identifier of the target task can be determined, and the neural network model corresponding to the task identifier can be screened out from the preset neural network model set to obtain the target neural network model, etc.
[0051] In some embodiments, according to the task scenario, the method of screening out the target neural network model from the preset neural network model may also include: determining the corresponding target task according to the task scenario, screening out the neural network model corresponding to the target task from the preset neural network model set, obtaining the initial neural network model, obtaining the model configuration parameters corresponding to the target task, and configuring the initial neural network model based on the model configuration parameters to obtain the target neural network model.
[0052] The initial neural network model can be understood as an untrained or pre-trained general neural network model for performing at least one task.
[0053] The model configuration parameters can be understood as the network parameters of the neural network model corresponding to the target task obtained after training the neural network model. There are many ways to configure the initial neural network model based on the model configuration parameters. For example, the network parameters in the initial neural network model can be replaced with the network parameters in the model configuration parameters to obtain the target neural network model.
[0054] In some embodiments, before selecting a target neural network model from a preset neural network model based on the task scenario, the neural network model may be trained to obtain a target neural network. There are multiple ways to train the neural network model. For example, a master manipulator may be used to remotely operate a slave manipulator in a real environment to perform a target task, thereby training a neural network model corresponding to the target task, thereby obtaining a target neural network model. Alternatively, a slave manipulator may be used in a virtual environment to perform a target task, thereby training a neural network model corresponding to the target task, thereby obtaining a target neural network model. Specifically, the following methods may be used: (1) The master robot teleoperates the slave robot in the real environment to perform the target task in order to train the neural network model corresponding to the target task.
[0055] For example, after obtaining the first pose data after the main robotic arm moves, the second pose data after the slave robotic arm follows the movement of the main robotic arm and the first image information of the target object can be obtained, and then the step of obtaining the first pose data after the main robotic arm moves is returned to execute until the target task is completed, and a training data set is obtained. Then, the training data set is used to train the neural network model to obtain the target neural network model of the slave robotic arm for the target task.
[0056] Among them, there are many ways to obtain the first position data of the main robotic arm after movement. For example, the first position data of the main robotic arm after movement can be directly read, or the first position data of the main robotic arm after movement can be received from the terminal or client corresponding to the robotic arm system, or the first position of the main robotic arm handle in the Cartesian coordinate system sent by a third-party device can be received, and the first position can be inversely kinematically solved to obtain the first angular position of the N joint motors of the main machine, and the first angular position can be used as the first position data of the main robotic arm after movement, and so on.
[0057] After obtaining the first pose data after the master manipulator moves, the second pose data of the slave manipulator in the real environment after following the master manipulator and the first image information of the target object can be obtained. There are many ways to obtain the second pose data and the first image information of the target object. For example, based on the first pose data, a control instruction for controlling the slave manipulator in the real environment can be generated. Based on the control instruction, the slave manipulator in the real environment is controlled to follow the master manipulator to move, and the second angular position of the slave manipulator after following the master manipulator and the second state information of the actuator are obtained. Of course, only the second angular position can be obtained to obtain the second pose data. Then, the image acquisition device can be used to capture image information of at least one scale of the target object after the slave manipulator follows the master manipulator to obtain the first image information.
[0058] After acquiring the second posture data of the slave manipulator in the real environment and the first image information of the target object, the step of acquiring the first posture data after the movement of the master manipulator can be returned to execute until the target task is completed, thereby obtaining the second training data set. For example, the step of acquiring the first posture data after the movement of the master manipulator can be returned to execute until the slave manipulator completes one target task. When the stop condition is to execute one target task, the acquisition of the first posture data after the movement of the master manipulator can be stopped, and the acquisition of the second posture data and first image information of the slave manipulator following the movement of the master manipulator in the real environment can be stopped. The acquired at least one first posture data, at least one second posture data, and at least one first image information can be used as the training data set corresponding to the target task. When the stop condition is to execute a preset number of target tasks, the step of acquiring the first posture data after the movement of the master manipulator can be returned to execute until the preset number of target tasks are completed. The acquired at least one first posture data, at least one second posture data, and at least one first image information can be used as the training data set corresponding to the target task.
[0059] After obtaining the training data set corresponding to the target task, the neural network model corresponding to the target task can be trained to obtain the trained target neural network model. For example, in the training data set, the neural network model is used to predict the pose data of the next time step of each time step to obtain a predicted pose data set. The target data label corresponding to the predicted pose data in the predicted pose data set is screened out from the training data set. Based on the target data label, the pose loss corresponding to the predicted pose data is determined. The neural network model is then converged based on the pose loss to obtain the target neural network model.
[0060] The predicted pose data set may include the predicted pose data of the slave manipulator at each time step. In this solution, an initial training dataset for the target task within the training dataset may be input into a neural network model, which may then output a predicted pose data set for the target task. Because the initial training dataset includes pose data for multiple time steps and image information of the target object, the pose data of the slave manipulator for the next time step can be predicted sequentially according to the order of the time steps, thereby obtaining a predicted pose data set.
[0061] Among them, there are many ways to filter out the target data labels corresponding to the predicted pose data in the predicted pose data set in the training data set. For example, the data label of the time step corresponding to each step-by-step action can be filtered out in the training data set as the target data label of the time step, thereby obtaining the target data label corresponding to the predicted pose data of each time step.
[0062] After filtering out the target data labels corresponding to the predicted pose data, the pose loss corresponding to the predicted pose data can be determined based on the target data labels. There are many ways to determine the pose loss corresponding to the predicted pose data. For example, the difference between the predicted pose data and the corresponding target loss can be calculated to obtain the pose loss corresponding to the predicted pose data.
[0063] After determining the pose loss corresponding to the predicted pose data, the neural network model can be converged based on the pose loss to obtain the target neural network model. There are many ways to converge the neural network model based on the pose loss. For example, a gradient descent algorithm can be used to update the network parameters of the neural network model based on the pose loss to obtain an updated neural network model, and the updated neural network model can be used as the neural network model to return to the step of determining the pose loss corresponding to the predicted pose data based on the target data label until the neural network model converges to obtain the target neural network model. Alternatively, a momentum algorithm can be used to update the network parameters of the neural network model to obtain an updated neural network model, and the updated neural network model can be used as the neural network model to return to the step of determining the pose loss corresponding to the predicted pose data based on the target data label until the neural network model converges to obtain the target neural network model, and so on.
[0064] (2) The target task is performed by the slave robotic arm in the virtual environment to train the neural network model corresponding to the target task.
[0065] For example, a data acquisition page can be displayed, which includes a slave robotic arm and a target object in a virtual environment; in response to a control instruction for the slave robotic arm, the slave robotic arm is controlled to operate on the target object in the virtual environment to perform the target task; then the posture data of the slave robotic arm in the virtual environment and the image information of the target object are collected until the slave robotic arm completes the target task to obtain a training data set, which is used to train a neural network model corresponding to the target task.
[0066] There are various ways to display the data collection page. For example, virtual reality technology can be used to map the slave robotic arm of the robotic arm system in a real environment into a virtual environment to generate a virtual slave robotic arm. Simultaneously, the operation object (i.e., target object) targeted by the robotic arm system 100 for executing a target task can be mapped into the virtual environment to generate a virtual target object. Based on the virtual slave robotic arm and the virtual target object, a data collection page can be generated and displayed.
[0067] Control instructions can be understood as interactive commands set by a user in a real environment for a slave manipulator in a virtual environment. The control instructions act on the slave manipulator in the virtual environment. The process of the slave manipulator operating on a target object in the virtual environment can be understood as the process of the slave manipulator performing a series of actions on the target object under the control of the control instructions. The end of the slave manipulator can be provided with an actuator, and control instructions can include motion instructions and operation instructions. Motion instructions control the movement of the slave manipulator in the virtual environment, while operation instructions control the actuator to operate on the target object in the virtual environment.
[0068] After the slave manipulator is controlled to perform an operation on a target object in a virtual environment in response to a control instruction directed to the slave manipulator, pose data of the slave manipulator in the virtual environment and image information of the target object can be collected until the slave manipulator completes the target task, thereby obtaining a training dataset. For example, pose data of the slave manipulator in the virtual environment and image information of the target object are collected at each time step to obtain an initial training dataset, data labels corresponding to the initial training dataset are obtained, and the data labels are added to the initial training dataset to obtain a training dataset.
[0069] After obtaining the training data set, the neural network model corresponding to the target task can be trained based on the training data set, thereby obtaining the target neural network model corresponding to the target task. The process of training the neural network model based on the training data set can be seen above and will not be repeated here.
[0070] 240. Based on the target neural network model, control the robotic arm to operate on at least one target object.
[0071] For example, the posture data of the robotic arm and the image information of at least one target object can be obtained. Based on the posture data and image information, the target neural network model is used to predict the posture data of the robotic arm at the next time step at the current moment to obtain the predicted posture data. Based on the predicted posture data, a control instruction for the robotic arm is generated. According to the control instruction, the robotic arm is controlled to perform an operation on at least one target object, and the step of obtaining the posture data of the robotic arm and the image information of at least one target object is returned to execute until the preset stop condition is reached to perform the target task in the task scenario.
[0072] The robotic arm may include M joint motors and an end for mounting an actuator. The M joints provide the end with m degrees of freedom, and the actuator is used to manipulate a target object. There are various ways to obtain the robotic arm's pose data and image information of at least one target object. For example, the angular positions of the M joint motors and the status information of the actuators may be read to obtain the robotic arm's pose data, and image information of at least one target object may be captured using an image acquisition device.
[0073] After obtaining the posture data of the robotic arm and the image information of at least one target object, the target neural network model can be used to predict the posture data of the robotic arm at the next time step at the current moment based on the posture data and image information, thereby obtaining predicted posture data. There are many ways to use the target neural network model to predict the posture data of the robotic arm at the next time step at the current moment. For example, the next time step at the current moment can be used as the target time step. Based on the posture data and image information, the target neural network model can be used to predict the posture data of at least one time step after the current time step corresponding to the current moment to obtain predicted motion sequence data. The predicted motion sequence data corresponding to a preset number of time steps before the current time step can be filtered out from the historical predicted motion sequence data to obtain historical predicted motion sequence data. The predicted motion sequence data and the historical motion sequence data can be fused to obtain predicted posture data for the target time step.
[0074] Among them, there can be multiple ways to use the target neural network model to predict the pose data of at least one time step after the current time step corresponding to the current moment based on the pose data and image information. For example, the target neural network model can be used to extract features of the pose data to obtain second style variable features, and multi-dimensional feature extraction can be performed on the image information to obtain second image features of at least one dimension. The second image features are fused to obtain second target image features. Based on the pose data, the second style variable features and the second target image features, the pose data of at least one time step after the current time step is predicted, thereby obtaining predicted motion sequence data.
[0075] Among them, there are many ways to fuse the predicted motion sequence data and the historical motion sequence data. For example, the pose data of the target time step can be extracted from the predicted motion sequence data to obtain the first candidate pose data, and the pose data of the target time step can be extracted from the historical predicted motion sequence data to obtain the second candidate pose data. The first candidate pose data and the second candidate pose data can be fused to obtain the predicted pose data of the target time step.
[0076] Among them, there are many ways to fuse the first candidate pose data and the second candidate pose data. For example, the pose data fusion weight can be obtained, and based on the pose data fusion weight, the first candidate pose data and the second candidate pose data are weighted respectively to obtain the weighted first pose data and the weighted second pose data, and the weighted first pose data and the weighted second pose data are added together to obtain the predicted pose data of the target time step.
[0077] After the target neural network model is used to predict the posture data of the robot arm at the next time step at the current moment based on the posture data and image information, control instructions for the robot arm can be generated based on the predicted posture data. The control instructions may include motion instructions and operation instructions. The motion instructions control the robot arm to move, and the operation instructions control the actuator to operate or work on the target object. There are many ways to generate control instructions for the robot arm based on the predicted posture data. For example, the predicted angular position of M joint motors and the predicted state information of the actuator can be extracted from the predicted posture data. Based on the predicted angular position and the predicted state information, motion instructions for the M joint motors and operation instructions for the actuator are generated respectively. The motion instructions and operation instructions are used as control instructions for the robot arm, thereby generating control instructions for the robot arm.
[0078] After generating control instructions for the robotic arm based on the predicted pose data, the robotic arm can be controlled to perform operations on at least one target object according to the control instructions. There are various ways to control the robotic arm to perform operations on the at least one target object according to the control instructions, such as controlling the movement of the M joint motors of the robotic arm through motion instructions and controlling the actuator to operate the at least one target object through operation instructions.
[0079] After controlling the robotic arm to operate on at least one target object according to the control instructions, the process can return to the step of obtaining the posture data of the robotic arm and the image information of at least one target object until the preset stop condition is reached to complete the target task in the task scenario.
[0080] The preset stop condition can be understood as a pre-set condition for stopping the robot arm from operating on at least one target object. There are various preset stop conditions, including, for example, the target object reaching a preset state or a specified position, or the robot arm having operated on at least one target object a preset number of times, or other pre-set stop conditions.
[0081] From the above, it can be seen that after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task recognition model to identify at least one modal information to determine the task scenario, and then screens out a target neural network model from a preset neural network model set according to the task scenario. Based on the target neural network model, the robotic arm is controlled to operate on at least one target object in the working area, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm.
[0082] An embodiment of the present application also provides a control device for a robotic arm, which can be integrated into an electronic device, such as a server or a terminal, and the terminal can include a tablet computer, a laptop computer and / or a personal computer, etc.
[0083] For example, Figure 3 As shown, the control device of the robotic arm has a display interface, which can display the acquisition module 310, the identification module 320, the screening module 330 and the control module 340, which can be specifically as follows: (1) An acquisition module 310 , which has at least one line of code and is used to acquire at least one modal information of a working area where the robot arm is located, where the working area allows at least one target object to be placed.
[0084] (2) Identification module 320, having at least one line of code, and configured to identify at least one modal information using a task identification model to determine a task scenario.
[0085] (3) A screening module 330 , which has at least one line of code and is used to screen out a target neural network model from a preset set of neural network models according to a task scenario.
[0086] (4) A control module 340 having at least one line of code and configured to control the robotic arm to operate on at least one target object based on the target neural network model.
[0087] In some embodiments, the robot arm control device may further include a training unit 350 having at least one line of code and configured to train a neural network model corresponding to a target task to obtain a target neural network. Figure 4 shown.
[0088] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.
[0089] From the above, it can be seen that in this application, after the acquisition module 310 acquires at least one modal information of the working area where the robotic arm is located, the identification module 320 uses the task identification model to identify the at least one modal information to determine the task scenario, and then the screening module 330 filters out the target neural network model from the preset neural network model set according to the task scenario, and the control module 340 controls the robotic arm to operate on at least one target object in the working area based on the target neural network model; so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm.
[0090] The present application also provides an electronic device, such as Figure 5 , which shows a schematic structural diagram of an electronic device involved in an embodiment of the present application, specifically: The electronic device may include components such as a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, and an input unit 440. It will be understood by those skilled in the art that Figure 5 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. The processor 410 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 420 and calling data stored in the memory 420.
[0091] Memory 420 can be used to store software programs and modules. Processor 410 executes various functional applications and data processing by running the software programs and modules stored in memory 420. Memory 420 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and applications required for at least one function (such as sound playback and image playback). The data storage area may store data generated based on the use of the electronic device.
[0092] The power supply 430 can be logically connected to the processor 410 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system.
[0093] The input unit 440 may be configured to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0094] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 410 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420 to implement various functions as follows: After obtaining at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed. The at least one modal information is identified using a task recognition model to determine the task scenario. Then, according to the task scenario, a target neural network model is screened out from a preset neural network model set. Based on the target neural network model, the robotic arm is controlled to operate on the at least one target object.
[0095] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.
[0096] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0097] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the robot arm control methods provided in the embodiments of the present application. For example, the instructions can execute the following steps: After obtaining at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed. The at least one modal information is identified using a task recognition model to determine the task scenario. Then, according to the task scenario, a target neural network model is screened out from a preset neural network model set. Based on the target neural network model, the robotic arm is controlled to operate on the at least one target object.
[0098] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.
[0099] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0100] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the robot arm control methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the robot arm control methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0101] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the aforementioned application and robotic arm control aspects of a robotic arm system.
[0102] The above is a detailed introduction to a control method for a robotic arm and a robotic arm system provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A control method for a robotic arm, It is characterized in that include: Acquire at least one modal information of a working area where the robot arm is located, wherein the working area allows at least one target object to be placed; Using a task identification model to identify the at least one modal information to determine a task scenario; According to the task scenario, a target neural network model is selected from a preset neural network model set; Based on the target neural network model, the robotic arm is controlled to perform operation on the at least one target object.
2. The control method according to claim 1, It is characterized in that The obtaining of at least one modal information of the working area where the robot arm is located includes: Obtaining an operation instruction for the robotic arm; In response to the operation instruction, at least one modal information of the area where the robot arm is located is obtained.
3. The control method according to claim 2, It is characterized in that The obtaining of the control instruction for the robotic arm comprises: Display a task list, wherein the task list includes at least one preset task; In response to a selection operation on the task list, determining task information; Based on the task information, the work instruction is generated.
4. The control method according to claim 2, It is characterized in that The obtaining of the operation instruction for the robot arm comprises: Acquiring voice information for the robotic arm; Performing speech recognition on the speech information to obtain task information; Based on the task information, the work instruction is generated.
5. The control method according to claim 2, It is characterized in that The adopting the task identification model to identify the at least one modal information to determine the task scenario includes: Using the task identification model, identifying object information of the target object in the at least one modal information; When there is only one target object, taking the object information as first object information, and filtering out the task scene from a preset task scene set according to the first object information; When there are multiple target objects, the object information is used as second object information, the target objects are compared based on the second object information, and the task scene is screened out from a preset task scene set based on the comparison result and the operation instruction.
6. The control method according to claim 5, It is characterized in that The step of selecting the task scenario from a preset task scenario set based on the comparison result and the operation instruction includes: When the target objects are of the same type, the task scene is selected from a preset task scene set according to the second object information; When the target objects are objects of different categories, candidate task scenes corresponding to each target object are screened out from a preset task scene set according to the second object information, and the task scene is determined from the candidate task scenes based on the work instruction.
7. The control method according to claim 6, It is characterized in that The step of determining the task scenario from the candidate task scenarios based on the job instruction includes: When the job instruction is generated based on the task information, the task scenario is screened out from the candidate task scenarios based on the task information; When the work instruction is generated based on the object detection result of the work area, the attribute information of the candidate task scenes is obtained, and based on the attribute information, the candidate task scenes are sorted to obtain the task scene.
8. The control method according to claim 1, It is characterized in that The controlling the robot arm to operate the at least one target object based on the target neural network model includes: Acquire the position data of the robotic arm and the image information of the at least one target object; Based on the posture data and the object image, using the target neural network model to predict the posture data of the robotic arm at the next time step at the current moment to obtain predicted posture data; Based on the predicted posture data, generating a control instruction for the robotic arm; According to the control instruction, controlling the robotic arm to operate on the at least one target object; Return to the step of acquiring the posture data of the robotic arm and the object image of the at least one target object until a preset stop condition is reached, so as to complete the target task in the task scenario.
9. A robotic arm system, It is characterized in that include: A robotic arm, comprising M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to operate a target object; An image acquisition device, configured to acquire image information containing the target object; The control device is configured to obtain an operation instruction for the robotic arm, determine a task scenario through the operation instruction, and select a target neural network model corresponding to the task scenario from a preset neural network model set, so as to control the robotic arm to perform an operation on the target object based on the target neural network model.
10. The robotic arm system according to claim 9, It is characterized in that The operation instruction is generated by the image information; or, The robot arm system further includes a voice collection device, which is configured to collect voice information of a user, and the operation instruction is generated through the voice information.
Citation Information
Patent Citations
Position control device and position control method
CN110268358A
Collision early warning method for dragging teaching spraying robot, medium and equipment
CN114055479A
Control unit and method for controlling robot to grip object
CN116652974A
Information processing apparatus, method for controlling information processing apparatus, and storage medium
US20190130216A1
Cited By
Manipulator control method and device and electronic equipment
CN119910642A
Mechanical hand control method and device and electronic equipment
CN119910642B
Hydraulic mechanical arm multi-loop hierarchical control method and system based on hybrid modeling
CN120245005A
