Robot applied to robot learning, training device and robot equipment

Abstract

The utility model discloses a robot applied to robot learning, a training device and robot equipment.The robot comprises a robot main frame, two execution manipulators, a main camera, at least two auxiliary cameras and a vector detection unit, the two execution manipulators are arranged on the left side and the right side of the robot main frame, the execution manipulator comprises a mechanical arm and an execution tail end, the mechanical arm is connected with the robot main frame, and the execution tail end is connected with the tail end of the mechanical arm; the main camera is arranged on the upper portion of the robot main frame and faces the two execution manipulators. The at least two auxiliary cameras are arranged on the two execution manipulators correspondingly and face the corresponding execution tail ends. The vector detection unit is used for detecting real action vectors of all joints of the execution manipulator. The main camera and the auxiliary camera capture image information of the robot, the vector detection unit is used for detecting and executing a real action vector of the manipulator, the real action vector is input into the robot learning model, and a target robot learning model is obtained.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of robots, and in particular, to a robot, a training device, and a robot equipment. Background Art

[0002] With the rapid development of artificial intelligence, robots have also developed rapidly and have been widely used in various fields. Especially in material handling, the application of robots can save a lot of labor.

[0003] However, most commercial robots usually only support PLC control programs and lack mobility and dexterity. A small number of robots also use neural network models in their applications. Due to the lack of better training methods, there are still many problems in the application of robots. In particular, with the increase of additional degrees of freedom, small deviations in the basic posture may cause large drifts in the posture of the end effector of the arm. Utility Model Content

[0004] The utility model aims to solve at least one of the technical problems existing in the prior art. To this end, the utility model provides a robot, a training device and a robot equipment, the robot can be used for training a robot learning model, and the robot can effectively complete a preset task in actual application.

[0005] In a first aspect, an embodiment of the present application provides a robot applied to robot learning, comprising:

[0006] Robot main frame;

[0007] Two execution manipulators are arranged on the left and right sides of the robot main frame, wherein the execution manipulator includes a manipulator arm and an execution end, the manipulator arm is connected to the robot main frame, and the execution end is connected to the end of the manipulator arm, for performing a grasping action;

[0008] A main camera is arranged on the upper part of the main frame of the robot and is arranged toward the two execution manipulators, and the main camera is used to globally capture the movement changes of the two execution manipulators;

[0009] At least two auxiliary cameras, respectively disposed on the two execution manipulators and facing the corresponding execution terminals, the auxiliary cameras being used to focus on and capture the movement changes of the execution terminals in the environment;

[0010] The vector detection unit is used to detect the real motion vectors of each joint of the execution robot.

[0011] According to some embodiments of the utility model, the robotic arm includes an upper arm and a front arm, wherein one end of the upper arm is movably connected to the robot main frame, one end of the front arm is movably connected to the other end of the upper arm, the execution end is movably connected to the other end of the front arm, and the auxiliary camera is arranged at the end of the front arm close to the execution end.

[0012] According to some embodiments of the utility model, the robot also includes a first movable chassis, the robot main frame is arranged on the first movable chassis, and the first movable chassis is used for linear motion and / or rotational motion; wherein the main camera is also used to capture the position change of the executing robot in the environment, and the vector detection unit is also used to detect the real motion vector of the first movable chassis.

[0013] According to some embodiments of the present utility model, the first movable chassis is an AGV navigation vehicle, and the vector detection unit is used to detect the angular velocity vector and the linear velocity vector of the first movable chassis.

[0014] According to some embodiments of the present invention, the robot has a connecting portion for detachably connecting to a training device.

[0015] According to some embodiments of the present utility model, the robot further includes a control unit, which is used to communicate with the training device and receive operation signals from the training device to control the two execution manipulators to perform actions.

[0016] In a second aspect, an embodiment of the present application provides a robot training device for training the above-mentioned robot, the training device comprising:

[0017] Training framework;

[0018] Two training manipulators having the same structure as the executing manipulator, the two training manipulators are arranged on the left and right sides of the training frame, and are used for one-to-one communication connection with the executing manipulator; wherein the parameter settings of the training manipulator and the executing manipulator are consistent, and the training manipulator can be operated to control the synchronous movement of the executing manipulator.

[0019] According to some embodiments of the present invention, the training frame and the robot are detachably connected; or, the training frame and the robot are separately arranged.

[0020] According to some embodiments of the utility model, the training device also includes a second movable chassis, the training frame is arranged on the second movable chassis, and the second movable chassis is used for linear motion and / or rotational motion; wherein the second movable chassis is used to communicate with the first movable chassis, and the second movable chassis can be operated to operate the first movable chassis to move synchronously.

[0021] In a third aspect, an embodiment of the present application provides a robot device, including:

[0022] The robots mentioned above;

[0023] The training device described above.

[0024] In a fourth aspect, an embodiment of the present application provides a robot training method, which is applied to the above-mentioned robot or the above-mentioned robot device, comprising:

[0025] Acquire training sample data of the robot, wherein the training sample data includes image information captured by the auxiliary camera and the main camera and a real motion vector detected by the vector detection unit, and the real motion vector is used to represent the result of the image information;

[0026] Get pre-built robot learning models;

[0027] The training sample data is input into the robot learning model to obtain a trained target robot learning model.

[0028] According to some embodiments of the present utility model, the image information includes the posture movements and position movements of the two execution manipulators in the environment;

[0029] The real motion vector includes the posture motion vectors of the joints of the two executing robots and the position motion vectors of the two executing robots in the environment.

[0030] According to some embodiments of the present invention, the position motion vector includes a linear velocity and / or an angular velocity.

[0031] According to some embodiments of the present utility model, the step of inputting the training sample data into the robot learning model to obtain a trained target robot learning model includes:

[0032] Inputting the image information captured by the auxiliary camera and the main camera into a robot learning model, and obtaining a predicted motion vector output by the robot learning model;

[0033] The proximity between the predicted motion vector and the real motion vector is evaluated, and the parameters of the robot learning model are adjusted until a target robot learning model is obtained.

[0034] According to some embodiments of the present utility model, the evaluating the proximity between the predicted action vector and the real action vector, and adjusting the parameters of the robot learning model until a target robot learning model is obtained, includes:

[0035] Determining whether the relationship between the predicted motion vector and the true motion vector satisfies a convergence condition;

[0036] If so, stop training the robot learning model and obtain the target robot learning model;

[0037] If not, the robot learning model is trained according to a preset loss function, and a predicted action vector output by the robot learning model is obtained based on the trained robot learning model, and the step of determining whether the relationship between the predicted action vector and the true action vector meets a convergence condition is executed again.

[0038] According to some embodiments of the present utility model, before obtaining the training sample data of the robot, the method includes:

[0039] Acquire operation information of two training manipulators of the user operating the training device;

[0040] Based on the operation information, the two execution manipulators are controlled to move synchronously with the two training manipulators of the training device.

[0041] According to some embodiments of the present invention, the robot learning model is a neural network model, a classification model or a regression model.

[0042] In a fifth aspect, an embodiment of the present application provides a robot training system, which is applied to the above-mentioned robot or the above-mentioned robot device, and the training system includes:

[0043] A data acquisition unit, used to obtain training sample data of the robot, wherein the training sample data includes image information captured by the auxiliary camera and the main camera and a real motion vector detected by the vector detection unit, and the real motion vector is used to represent the result of the image information;

[0044] The processor is used to obtain a pre-built robot learning model, and to input the training sample data into the robot learning model to obtain a trained target robot learning model.

[0045] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: the main camera is arranged on the upper part of the robot main frame, the two execution manipulators are arranged on the left and right sides of the robot main frame and are arranged toward the two execution manipulators, and the two auxiliary cameras are respectively arranged on the two mechanical arms and are arranged toward the corresponding execution terminals. In this way, during the training process of the robot, such as the process of grabbing or placing the material, the main camera has a wide field of view and can perceive the surrounding environment, so that the movement changes of the two execution manipulators in the environment can be captured globally; the auxiliary camera is close to the corresponding execution terminal, so that the fine movement of the execution terminal in the environment can be captured in a focused manner, and the vector detection unit is used to detect the real movement vector of each joint of the execution manipulator. Therefore, the robot of the present application can use the main camera and the auxiliary camera to capture the robot's movement changes in the environment in an all-round and accurate manner, that is, image information, and use the vector detection unit to detect the real movement vector of each joint of the execution manipulator, and the real movement vector is used to characterize the result of the image information. These data can be input into the robot learning model, and the robot learning model can be trained to obtain the required target robot learning model. The robot application uses the above-mentioned neural network model. Even when faced with more joint activities or multi-dimensional movements, the robot can effectively perform actions under the control of the target robot learning model, thereby accurately completing the grasping or placement of materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention is further described below with reference to the accompanying drawings and embodiments, wherein:

[0047] Figure 1 This is a schematic diagram of the overall structure of the robot in the embodiment of the utility model;

[0048] Figure 2 This is a schematic diagram of the overall structure of the robot and the training device in the embodiment of the utility model;

[0049] Figure 3 Schematic diagram of the flow of the training method in the embodiment of the utility model.

[0050] Reference numerals:

[0051] 10. Robot; 100. Robot main frame; 200. Execution manipulator; 210. Robot arm; 211. Upper wrist arm; 212. Front wrist arm; 220. Execution terminal; 300. Auxiliary camera; 400. Main camera; 500. First movable chassis; 510. Connection part; 600. Training device; 610. Training frame; 620. Training manipulator. DETAILED DESCRIPTION

[0052] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0053] In the description of the present invention, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0054] In the description of the present utility model, "several" means more than one, "many" means more than two, "greater than", "less than", "exceed" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used to distinguish the technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0055] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0056] In the description of the utility model, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the utility model. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0057] The method provided in the embodiment of the present application can be executed by a terminal device, including but not limited to: a smart phone, a tablet computer, a laptop computer, a desktop computer and other devices.

[0058] Alternatively, it may be executed by a chip or a chip system, which may automatically generate a malicious command detection model and detect malicious commands, and the chip may be embedded in a terminal device.

[0059] Alternatively, it may be executed by a server, including but not limited to: an independent physical server, a server cluster or distributed system consisting of multiple physical servers, a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms and other basic cloud computing services.

[0060] The present application discloses a robot 10 for robot learning, referring to Figure 1 , including a robot main frame 100, two execution manipulators 200, a main camera 400, at least two auxiliary cameras 300 and a vector detection unit, wherein:

[0061] Two execution manipulators 200 are arranged on the left and right sides of the robot main frame 100, wherein the execution manipulator 200 includes a robot arm 210 and an execution end 220, the robot arm 210 is connected to the robot main frame 100, and the execution end 220 is connected to the end of the robot arm 210 for performing grasping actions.

[0062] The main camera 400 is arranged on the upper part of the robot main frame 100 and is set toward the two execution manipulators 200. The main camera 400 is used to globally capture the movement changes of the two execution manipulators 200; one or more main cameras 400 can be set, and this application is not limited.

[0063] At least two auxiliary cameras 300 are respectively disposed on the two robotic arms 210 and are disposed toward the corresponding execution terminals 220 . The auxiliary cameras 300 are used to focus on capturing the movement changes of the execution terminals 220 in the environment.

[0064] The vector detection unit is used to detect the real motion vector of each joint of the robot 200.

[0065] The motion change refers to the posture change of the execution manipulator and the execution unit in the environment to complete the grasping or placement of the material. The vector detection unit can be understood as a signal detection module of each joint position, such as an encoder, which is used to detect the motion change of each joint position of the execution manipulator.

[0066] It is understandable that the main camera 400 is arranged on the upper part of the robot main frame 100, the two execution manipulators 200 are arranged on the left and right sides of the robot main frame 100 and are arranged toward the two execution manipulators 200, and the two auxiliary cameras 300 are respectively arranged on the two mechanical arms 210 and are arranged toward the corresponding execution terminals 220. In this way, during the training process of the robot 10, such as the process of grasping or placing materials, the main camera 400 has a wide field of view and can perceive the surrounding environmental conditions, so that the movement changes of the two execution manipulators 200 in the environment can be captured globally; the auxiliary camera 300 is facing the corresponding execution terminal 220 at a close distance, so that the fine movements of the execution terminal 220 in the environment can be captured in a focused manner, and the vector detection unit is used to detect the real movement vectors of each joint of the execution manipulator 200.

[0067] Therefore, the robot 10 of the present application can use the main camera 400 and the auxiliary camera 300 to fully and accurately capture the changes in the movements of the robot 10 in the environment, that is, image information, and use the vector detection unit to detect the real motion vectors of each joint of the manipulator 200, and the real motion vectors are used to characterize the results of the image information. These data can be input into the robot learning model, and the robot learning model can be trained to obtain the required target robot learning model. The application program of the robot 10 adopts the above-mentioned neural network model. Even in the face of more joint activities or multi-dimensional movements, the robot 10 can effectively perform actions under the control of the target robot learning model, thereby accurately completing the grasping or placement of materials.

[0068] In addition, in the present application, the robot 10 mainly adopts an execution manipulator 200, a main camera 400 and an auxiliary camera 300. The overall structure of the robot 10 is simple, and the training process applied to the robot learning model is relatively simple. The application cost of the robot 10 is low, and it is suitable for popularization in industry and households, for example, as a nanny robot.

[0069] In some embodiments, reference Figure 1 The robot arm 210 includes an upper arm 211 and a front arm 212, wherein one end of the upper arm 211 is movably connected to the robot main frame 100, for example, through a universal joint or a rotating shaft; one end of the front arm 212 is movably connected to the other end of the upper arm 211, for example, through a universal joint or a rotating shaft, and the base of the execution terminal 220 is movably connected to the other end of the front arm 212, for example, through a universal joint or a rotating shaft. The auxiliary camera 300 is arranged at the end of the front arm 212 close to the execution terminal 220, and is arranged just toward the execution terminal 220.

[0070] It is understandable that a movable joint is formed between the upper wrist 211 and the robot main frame 100, a movable joint is formed between the upper wrist 211 and the front wrist 212, and a movable joint is formed between the front wrist 212 and the execution terminal 220. The robot arm 210 adopts the above-mentioned structural form, and the flexibility of the robot arm 210 is good, so that the execution terminal 220 can be controlled at multiple angles and multiple positions, and the execution terminal 220 can flexibly grab or place the material. In addition, the overall structure of the robot arm 210 is relatively simple. When the various joint positions of the robot arm 210 are moving, the main camera 400 can capture the activity changes of each joint position, thereby ensuring the integrity of the input image information, and then ensuring that the robot learning model can be effectively trained.

[0071] Furthermore, the structural form of the robot arm 210 is substantially consistent with the structural form of the arm. Therefore, when training the robot arm 200, the movements of the staff's arms can be used to train the movements of the robot arm 210, thereby ensuring the accuracy and authenticity of the training of the robot arm 200.

[0072] In addition, the auxiliary camera 300 is arranged at the end of the front arm 212 close to the execution terminal 220. In this way, the auxiliary camera 300 can focus on capturing the movement changes of the execution terminal 220, thereby ensuring the integrity and accuracy of the image information, and further ensuring that the robot learning model can be effectively trained. In addition, the auxiliary camera 300 may also capture the posture changes between the front arm 212 and the execution terminal 220, thereby supplementing the main camera 400 that may not be able to clearly capture the movement changes at this position, thereby ensuring the integrity of the image information.

[0073] In some embodiments, the robot 10 further includes a first movable chassis 500, the bottom of the robot main frame 100 is disposed on the first movable chassis 500, and the first movable chassis 500 is used for linear motion and / or rotational motion, that is, the first movable chassis 500 drives the robot 10 to perform rotational motion to adjust the orientation direction of the robot 10, or the first movable chassis 500 drives the robot 10 to perform linear motion to adjust the overall position of the robot 10. Alternatively, the first movable chassis 500 can drive the robot 10 to perform both rotational motion and linear motion.

[0074] Among them, when the first movable chassis 500 drives the robot 10 to move, the main camera 400 is also used to capture the position change of the execution manipulator 200 in the environment. More specifically, the main camera 400 captures the movement change of the execution manipulator 200 during the movement of the robot 10, and the auxiliary camera 300 captures the movement change of the execution terminal 220 during the movement of the robot 10. In addition, the vector detection unit is also used to detect the real movement vector of the first movable chassis 500, such as the angular velocity and / or linear velocity. In this way, the acquired image information and the real movement vector can train the robot learning model more comprehensively. Correspondingly, when the acquired target robot learning model is applied to the robot 10, it can control the robot 10 to complete the grabbing or placing of the material during the movement, and the robot 10 is more applicable in industry.

[0075] In a specific embodiment, referring to Figure 1 The first movable chassis 500 is an AGV navigation vehicle, and the robot main frame 100 is arranged on the AGV navigation vehicle. The AGV navigation vehicle has a moving speed of up to 1.6 m / s, a maximum payload of 100 kg, and a minimum ground clearance of about 30 mm. The AGV navigation vehicle can drive the robot 10 to move linearly and rotationally. The vector detection unit is used to detect the angular velocity vector and the linear velocity vector of the AGV navigation vehicle, and the angular velocity vector and the linear velocity vector are collectively referred to as the action vector.

[0076] Specifically, each execution manipulator 200 has 7 degrees of freedom, and the two execution manipulators 200 can have 14 degrees of freedom, but it is not limited to only 14 degrees of freedom. 14 degrees of freedom can be understood as a 14-dimensional motion vector. The first movable chassis 500 can be understood as a 2-dimensional motion vector (a combination of angular velocity and linear velocity). Therefore, the overall motion posture of the robot 10 constitutes a real 16-dimensional motion vector, that is, the vector detection unit detects and obtains a real 16-dimensional motion vector. The real 16-dimensional motion vector covers the precise position of each joint of the robot 10, and also reflects the moving state of the chassis, thereby completely determining the instantaneous posture of the robot 10. The image information captured by the main camera 400 and the auxiliary camera 300 can be mapped to a 16-dimensional motion vector. Therefore, after the image information is input into the initial neural model, the robot learning model outputs a 16-dimensional motion vector, and the output 16-dimensional motion vector is compared with the real 16-dimensional motion vector. According to the gap between the two, the neural network model is trained until the target robot learning model is obtained.

[0077] In some embodiments, reference Figure 1 and Figure 2The robot 10 has a connection portion 510, which can be disposed on the robot main frame 100 or on the top of the first movable chassis 500. The connection portion 510 is detachably connected to the training device 600, for example, the connection portion 510 is snap-connected to the training device 600, or is fixedly connected by bolts, which is not limited in the present application.

[0078] It is understandable that the robot 10 is connected to the training device 600. In this way, when the robot 10 is trained by the training device 600, since the first movable chassis 500 is provided at the bottom of the robot 10, the staff can push the robot 10 to move when using the training device 600, thereby conveniently realizing the movement training of the robot 10. After the training of the robot 10 is completed, the staff can disassemble the training device 600, and the robot 10 can be independently put into industrial use.

[0079] In some embodiments, the robot 10 further includes a control unit, which is in communication with the training device 600 and is used to receive an operation signal from the training device 600 to control the two execution manipulators 200 to perform actions. Specifically, during the training process of the robot 10, the staff sends an operation signal to the robot 10 through the training device 600, and after receiving the operation signal, the control unit of the robot 10 controls the execution manipulator 200 to perform actions to complete the grasping or placement of the material, thereby performing motion training on the robot 10.

[0080] The present application also discloses a training device 600 for the robot 10, referring to Figure 1 and Figure 2 , applied to the robot 10 described above, the training device 600 includes a training frame 610 and two training manipulators 620, wherein the training frame 610 can adopt a variety of structural forms and be specifically configured according to actual needs. For example, the training frame 610 can be configured to be a structure that facilitates the staff to push the robot 10 to move. The two training manipulators 620 have the same structure and the same parameters as the two execution manipulators 200. The two training manipulators 620 are arranged on the left and right sides of the training frame 610 and are connected to the execution manipulators 200 one by one in communication, such as wireless communication connection or wired communication connection. The training manipulators 620 can be operated to control the synchronous movement of the execution manipulators 200.

[0081] During the specific training process, the staff's arm is bound to the mechanical arm 210 of the training manipulator 620, and the staff's palm is bound to the execution end 220 of the training manipulator 620. Therefore, during the training process of the robot 10, the staff controls the two training manipulators 620 to perform actions, and synchronously, the two execution manipulators 200 and the two training manipulators 620 move synchronously, thereby completing the grasping or placement of the material, and then training the execution manipulators 200. It can be understood that this design method ensures that the staff can truly control the actions of the robot 10 when performing two-handed actions, thereby ensuring the accuracy of the image information and the action vector, which is helpful to better train the robot learning model.

[0082] In a possible embodiment, the training frame 610 is detachably connected to the robot 10. Specifically, the robot 10 has a connecting portion 510, which can be disposed on the robot main frame 100 or on the top of the first movable chassis 500. The connecting portion 510 is detachably connected to the training device 600, for example, the connecting portion 510 is snap-connected to the training device 600, or is fixedly connected by bolts, which is not limited in the present application.

[0083] It is understandable that the robot 10 is connected to the training device 600. In this way, when the robot 10 is trained by the training device 600, since the first movable chassis 500 is provided at the bottom of the robot 10, the staff can push the robot 10 to move when operating the training device 600, so that the robot 10 can be conveniently trained in movement. After the training of the robot 10 is completed, the staff disassembles the training device 600, and the robot 10 can be independently put into industrial use.

[0084] In another possible embodiment, the training frame 610 is separately provided from the robot 10 (not shown in the figure), more specifically, the training device 600 is separately provided from the robot 10. When the robot 10 is trained, the staff can use the training device 600 to remotely control the robot 10 to perform actions, thereby training the robot 10.

[0085] In some embodiments, the training device 600 further includes a second movable chassis (not shown in the figure), the training frame 610 is arranged on the second movable chassis, and the second movable chassis is used for linear motion and / or rotational motion, that is, the first movable chassis 500 and the second movable chassis have the same structure and the same parameter settings. The second movable chassis is connected to the first movable chassis 500 in communication, and the second movable chassis can be operated to operate the first movable chassis 500 to move synchronously. Specifically, if the training device 600 is separately arranged from the robot 10, when the training device 600 is applied, the staff pushes the second movable chassis to move through the training frame 610, and the first movable chassis 500 moves synchronously with the second movable chassis, thereby performing motion training on the robot 10. More specifically, the training device 600 can train the actions of the manipulator 200 during the motion process.

[0086] The present application discloses a robot device, including the robot 10 and the training device 600 mentioned above.

[0087] During the specific training process, the staff's arm is bound to the mechanical arm 210 of the training manipulator 620, and the staff's palm is bound to the execution end 220 of the training manipulator 620. During the training process of the robot 10, the staff controls the two training manipulators 620 to perform actions, and synchronously, the two execution manipulators 200 and the two training manipulators 620 move synchronously, thereby completing the grasping or placement of the material.

[0088] Therefore, during the operation, the two execution manipulators 200 use the main camera 400 and the auxiliary camera 300 to fully and accurately capture the movement changes of the robot 10 in the environment, that is, the image information, and use the vector detection unit to detect the real movement vectors of each joint of the execution manipulator 200. The real movement vectors are used to characterize the results of the image information. These data can be input into the robot learning model, and the robot learning model is trained to obtain the required target robot learning model. The application program of the robot 10 adopts the above-mentioned neural network model. Even in the face of more joints or multi-dimensional movements, the robot 10 can effectively perform actions under the control of the target robot learning model, thereby accurately completing the material grabbing or placement.

[0089] See also Figure 3 , is a flow chart of a robot 10 training method provided in an embodiment of the present application, which is applied to the above-mentioned robot 10 or robot equipment, and the training method includes S101 to S103.

[0090] S101 obtains training sample data of the robot 10, wherein the training sample data includes image information captured by the auxiliary camera 300 and the main camera 400 and real motion vectors detected by the vector detection unit, and the real motion vectors are used to characterize the results of the image information.

[0091] If the first movable chassis 500 is provided at the bottom of the robot 10, the image information includes the posture movements and position movements of the two execution manipulators 200 in the environment. Accordingly, the real action vector includes the posture movement vectors of the joints of the two execution manipulators 200 and the position movement vectors of the two execution manipulators 200 in the environment. For example, the first movable chassis 500 is an AGV navigation vehicle, which can rotate and move linearly, and the position movement vector includes the linear velocity and the angular velocity. Of course, if other types of first movable chassis 500 are selected, the position movement vector may also only include the linear velocity or the angular velocity. The corresponding first movable chassis is selected according to actual needs.

[0092] Of course, if the first movable chassis 500 is not provided at the bottom of the robot 10 , the image information may only include the posture movements of the two execution manipulators 200 in the environment, and accordingly, the real motion vectors also only include the posture motion vectors of the joints of the two execution manipulators 200 .

[0093] Specifically, during the training process of the robot 10, such as the grasping or placing of materials, the main camera 400 has a wide field of view and can perceive the surrounding environmental conditions, thereby being able to globally capture the motion changes of the two execution manipulators 200 in the environment; the auxiliary camera 300 is facing the corresponding execution terminal 220 at a close distance, so that it can focus on capturing the fine movements of the execution terminal 220 in the environment. Thus, the main camera 400 and the auxiliary camera 300 capture the motion changes of the robot 10 in the environment, that is, image information, in an all-round and accurate manner, and the vector detection unit is used to detect the real motion vectors of each joint of the execution manipulator 200.

[0094] For example, each execution manipulator 200 has 7 degrees of freedom, and two execution manipulators 200 can have 14 degrees of freedom, but it is not limited to only 14 degrees of freedom. 14 degrees of freedom can be understood as a 14-dimensional motion vector. The first movable chassis 500 can be understood as a 2-dimensional motion vector (a combination of angular velocity and linear velocity). Therefore, the robot 10 as a whole constitutes a real 16-dimensional motion vector, that is, the vector detection unit detects and obtains a real 16-dimensional motion vector. The real 16-dimensional motion vector covers the precise position of each joint of the robot 10, and also reflects the moving state of the chassis, thereby completely determining the instantaneous posture of the robot 10. Among them, the image information captured by the main camera 400 and the auxiliary camera 300 is mapped to a 16-dimensional motion vector, that is, the real motion vector is used to characterize the result of the image information.

[0095] S1O2Get pre-built robot learning models.

[0096] Among them, a machine learning model is an algorithm or model that makes predictions and decisions by learning rules and patterns from data. A robot learning model refers to a machine learning model applied in the field of robotics.

[0097] In robot 10 learning, there are several common models and algorithms, including but not limited to the following:

[0098] Supervised learning model: Supervised learning is a method of training a model by using labeled training data. In robot 10 learning, supervised learning models can be used for tasks such as target detection, object recognition, pose estimation, etc.

[0099] Reinforcement learning model: Reinforcement learning is a method of learning optimal behavior by interacting with the environment. In robot learning, reinforcement learning models can be used for tasks such as path planning and motion control.

[0100] Unsupervised learning models: Unsupervised learning is a method for discovering hidden structures and patterns from unlabeled data. In robot 10 learning, unsupervised learning models can be used for tasks such as clustering and dimensionality reduction.

[0101] Deep learning model: Deep learning is a machine learning method based on neural networks, which learns complex feature representations through a multi-level neural network structure. In robot learning, deep learning models can be used for tasks such as image recognition, speech recognition, and autonomous navigation.

[0102] Transfer learning model: Transfer learning is a method of transferring learned knowledge and models to new tasks. In robot 10 learning, transfer learning models can be used to improve the learning effect and adaptability of robot 10 in new environments.

[0103] S1O3 inputs the training sample data into the robot learning model to obtain a trained target robot learning model.

[0104] The training sample data includes real motion vectors and image information, and the real motion vectors are used to represent the results of the image information. For example, the real motion vector is a 16-dimensional motion vector, but is not limited to a 16-dimensional motion vector, that is, the angular velocity and linear velocity of the first movable chassis and the motion variables of the two execution manipulators 200.

[0105] Specifically, after obtaining the image information and the real action vector, the image information and the real action vector are input into the robot learning model, so that the robot learning model is trained. The angles of the joints of the two execution manipulators 200 are adjusted and the movement of the first movable chassis 500 is controlled, and the obtained image information and the real action vector are re-input into the robot learning model, and the robot learning model is trained in this way in a cycle until the 16-dimensional action vector output by the robot learning model completes the preset action, thereby obtaining the required target robot learning model.

[0106] In summary, according to steps S101 to S103, the robot 10 of the present application can use the main camera 400 and the auxiliary camera 300 to fully and accurately capture the changes in the movements of the robot 10 in the environment, that is, image information, and use the vector detection unit to detect the real motion vectors of each joint of the manipulator 200, and the real motion vectors are used to characterize the results of the image information. These data can be input into the robot learning model, and the robot learning model can be trained to obtain the required target robot learning model. The application program of the robot 10 adopts the above-mentioned neural network model. Even in the face of more joint activities or multi-dimensional movements, the robot 10 can effectively perform actions under the control of the target robot learning model, thereby accurately completing the grasping or placement of materials.

[0107] In some embodiments, step S103 inputs the training sample data into the robot learning model to obtain a trained target robot learning model, including A1-A2.

[0108] A1 inputs the image information captured by the auxiliary camera 300 and the main camera 400 into the robot learning model, and obtains the predicted motion vector output by the robot learning model.

[0109] A2 evaluates the proximity between the predicted action vector and the real action vector, and adjusts the parameters of the robot learning model until a target robot learning model is obtained.

[0110] Specifically, after obtaining the image information and the real action vector, the image information and the real action vector are input into the robot learning model, and the robot learning model outputs a 16-dimensional action vector, i.e., a predicted action vector. The output 16-dimensional action vector is used to characterize the angular velocity variable, linear velocity variable and action variable of the first movable chassis 200 of the two execution manipulators 200. Compare the gap between the output 16-dimensional action vector and the real action vector, and adjust the parameters of the robot learning model to train the robot training model. Then, the staff adjusts the angles of the joints of the two execution manipulators 200, and controls the movement of the first movable chassis 500. The auxiliary camera 300 and the main camera 400 capture the corresponding image information and input it into the robot learning model. The robot learning model obtains the output 16-dimensional action vector. Compare the gap between the output 16-dimensional action vector and the real action vector again, and adjust the parameters of the robot learning model to train the robot training model. The above steps are trained in this way, and the 16-dimensional action vector output by the robot learning model completes the real action vector. At this time, the robot learning model is a trained target robot learning model.

[0111] In a further embodiment, step A2 evaluates the proximity between the predicted motion vector and the real motion vector, and adjusts the parameters of the robot learning model until the target robot learning model is obtained, including steps B1-B3.

[0112] B1 determines whether the relationship between the predicted motion vector and the real motion vector meets the convergence condition.

[0113] B2: If yes, stop training the robot learning model and obtain the target robot learning model.

[0114] If B3 is not, the robot learning model is trained according to a preset loss function, and the predicted action vector output by the robot learning model is obtained based on the trained robot learning model, and the step of determining whether the relationship between the predicted action vector and the true action vector meets the convergence condition is executed again.

[0115] In specific implementation, the training sample data can be input into the pre-built robot learning model in batches, and the model extracts the string of sample data and outputs the corresponding detection results. Then, the gap between the output action vector and the corresponding real action vector is compared, and the model parameters of the pre-built robot learning model are adjusted according to the gap and the preset loss function. Then, the next batch of sample data is input for training, and the above training process is repeated until the pre-built robot learning model meets the convergence conditions, thereby obtaining a trained target robot learning model. For example, in the training process of the BP (back propagation) network, the learning algorithm uses the Steepest Descent Back Propagation (SDBP) algorithm to calculate the output of each neuron from the first layer of the network backwards, and calculate the influence of the model parameter value on the total error from the last layer forward. The model parameters of the network are continuously adjusted through the back propagation algorithm to minimize the total square sum of the network error and achieve network convergence. It should be noted that the learning algorithm can also be other, such as LM (Leverberg-Marquardt) algorithm, Momentum BP (Momentum Backpropagation, MOBP) algorithm, Variable Learning rate Backpropagation (Variable Learning rate Backpropagation, VLBP) algorithm, Resilient Backpropagation (RPROP) algorithm, variable gradient algorithm, quasi-Newton algorithm, etc.; the error function used for adjustment can select the mean square error function, etc.; the number of hidden layer nodes is set to 50; the maximum number of training times is set to 10000, the target error is set to 0.001; the minimum gradient is 1e-6.

[0116] In some embodiments, before step S103 acquires the training sample data of the robot 10 , steps C1 - C2 are included.

[0117] C1 obtains the operation information of the user when operating the two training manipulators 620 of the training device 600.

[0118] C2 controls the two execution robots 200 and the two training robots 620 of the training device 600 to move synchronously based on the operation information.

[0119] Specifically, in combination with the above-mentioned training device 600, the staff's arm is bound to the mechanical arm 210 of the training manipulator 620, and the staff's palm is bound to the execution end 220 of the training manipulator 620. During the training process of the robot 10, the staff controls the two training manipulators 620 to move, and controls the movement of the robot 10. The two execution manipulators 200 move synchronously with the two training manipulators 620, and complete the grabbing or placement of the material during the movement. During the operation, the two execution manipulators 200 use the main camera 400 and the auxiliary camera 300 to capture the movement changes of the robot 10 in the environment in an all-round and accurate manner, that is, image information, and use the vector detection unit to detect the real action vectors of each joint of the execution manipulator 200. The real action vector is used to characterize the result of the image information. These data can be input into the robot learning model, and the robot learning model is trained to obtain the required target robot learning model. The application of robot 10 adopts the above-mentioned neural network model. Even when faced with a large number of joint movements or multi-dimensional movements, robot 10 can effectively perform actions under the control of the target robot learning model, thereby accurately completing the grasping or placement of materials.

[0120] In some embodiments, the robot learning model is a neural network model, a classification model or a regression model. It is understandable that the robot learning model adopts the above model, and the robot learning model can effectively predict the action of the robot 10 when applied to complete the grasping or placement of the material.

[0121] In some embodiments, the present application discloses a robot 10 training system, including:

[0122] The data acquisition unit is used to obtain training sample data of the robot 10, wherein the training sample data includes image information captured by the auxiliary camera 300 and the main camera 400 and real motion vectors detected by the vector detection unit, and the real motion vectors are used to characterize the results of the image information.

[0123] The processor is used to obtain a pre-built robot learning model and to input training sample data into the robot learning model to obtain a trained target robot learning model.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0125] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A robot used for robot learning, comprising: Robot main frame; Two execution manipulators are arranged on the left and right sides of the robot main frame, wherein the execution manipulator includes a manipulator arm and an execution end, the manipulator arm is connected to the robot main frame, and the execution end is connected to the end of the manipulator arm, for performing a grasping action; A main camera is arranged on the upper part of the main frame of the robot and is arranged toward the two execution manipulators, and the main camera is used to globally capture the movement changes of the two execution manipulators; At least two auxiliary cameras, respectively disposed on the two execution manipulators and facing the corresponding execution terminals, the auxiliary cameras being used to focus on and capture the movement changes of the execution terminals in the environment; The vector detection unit is used to detect the real motion vectors of each joint of the execution robot.

2. A robot for robot learning according to claim 1, characterized in that: The robotic arm includes an upper arm and a front arm, wherein one end of the upper arm is movably connected to the robot main frame, one end of the front arm is movably connected to the other end of the upper arm, the execution end is movably connected to the other end of the front arm, and the auxiliary camera is arranged at the end of the front arm close to the execution end.

3. A robot for robot learning according to claim 1, characterized in that: The robot also includes a first movable chassis, the robot main frame is arranged on the first movable chassis, and the first movable chassis is used for linear motion and / or rotational motion; wherein the main camera is also used to capture the position change of the execution robot in the environment, and the vector detection unit is also used to detect the real motion vector of the first movable chassis.

4. A robot for robot learning according to claim 3, characterized in that: The first movable chassis is an AGV navigation vehicle, and the vector detection unit is used to detect the angular velocity vector and the linear velocity vector of the first movable chassis.

5. A robot for robot learning according to claim 3, characterized in that: The robot has a connecting part for detachably connecting with the training device.

6. A robot for robot learning according to claim 1, characterized in that: The robot also includes a control unit, which is used to communicate with the training device and receive operation signals from the training device to control the two execution manipulators to perform actions.

7. A robot training device, characterized in that: Used for training the robot according to any one of claims 1 to 6, the training device comprising: Training framework; Two training manipulators having the same structure as the executing manipulator, the two training manipulators are arranged on the left and right sides of the training frame, and are in one-to-one communication connection with the executing manipulator; wherein the parameter settings of the training manipulator and the executing manipulator are consistent, and the training manipulator can be operated to control the synchronous movement of the executing manipulator.

8. A robot training device according to claim 7, characterized in that: The training frame and the robot are detachably connected; or, the training frame and the robot are separately arranged.

9. A robot training device according to claim 8, characterized in that: The training device also includes a second movable chassis, the training frame is arranged on the second movable chassis, and the second movable chassis is used for linear motion and / or rotational motion; wherein the second movable chassis is communicatively connected with the first movable chassis, and the second movable chassis can be operated to operate the first movable chassis to move synchronously.

10. A robot device, characterized in that: include: The robot according to any one of claims 1 to 6; A training device as claimed in any one of claims 7 to 9.

Images