Multi-degree-of-freedom robotic arm interaction device and control method thereof

Abstract

The application provides a multi-degree-of-freedom mechanical arm interactive device and a control method thereof, the device comprising: a collection module configured to collect behavior data of an experimental object; a mechanical arm mechanism comprising a mechanical arm and an end effector; the end effector is detachably installed at the end of the mechanical arm, and the end effector is configured to allow the experimental object to perform different operation tasks; the mechanical arm comprises at least six degrees of freedom, and the mechanical arm is configured to move in a three-dimensional space; a control module is electrically connected to the mechanical arm, the control module generates a motion instruction based on at least one of the behavior data and pose feedback data of the mechanical arm, and controls the mechanical arm to drive the end effector to move according to the motion instruction, so as to drive the experimental object to perform a target operation task; the target operation task belongs to different operation tasks. The flexibility of the mechanical arm of the device moving in the three-dimensional space and the grasping ability of the system for different tasks can improve the interactive performance of the mechanical interactive system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm motion control, in particular to a multi-degree-of-freedom mechanical arm interactive device and a control method thereof. BACKGROUND

[0002] In the cognitive and neuroscience research of non-human primates (such as monkeys), flexible and accurate experimental equipment is essential for the motion control, learning ability and decision-making mechanism of the research animals.

[0003] In related technologies, the current mechanical interaction system for primate experiments mainly focuses on joystick tasks, two-dimensional screen interaction and fixed trajectory mechanical arm systems. Since the mechanical interaction system is difficult to simulate natural behavior, the interactive motion pattern of the experimental object is limited, and it is difficult to achieve true autonomous control and natural three-dimensional operation. Moreover, the traditional mechanical interaction system usually uses fixed grabbing devices or tools to train the learning ability of the experimental object, which is difficult to adapt to different types of task requirements. SUMMARY

[0004] The present application provides a multi-degree-of-freedom mechanical arm interactive device and a control method thereof, to solve the defects that the mechanical interaction system in the prior art is difficult to simulate natural behavior, resulting in limited interactive motion pattern of the experimental object, and the prior art uses fixed grabbing devices or tools to train the learning ability of the experimental object, which is difficult to adapt to different types of task requirements. The device improves the flexibility of the mechanical arm in three-dimensional space movement and the grabbing ability of the system for different tasks, thereby improving the interactive performance of the mechanical interaction system.

[0005] The present application provides a multi-degree-of-freedom mechanical arm interactive device, comprising:

[0006] A collection module for collecting behavior data of an experimental object;

[0007] A mechanical arm mechanism comprising a mechanical arm and an end effector;

[0008] The end effector is detachably installed at the end of the mechanical arm, and the end effector is used for the experimental object to perform different operation tasks;

[0009] The mechanical arm comprises at least 6 degrees of freedom, and the mechanical arm is used for moving in three-dimensional space;

[0010] A control module electrically connected to the mechanical arm, the control module generates a motion instruction through at least one of the behavior data and the pose feedback data of the mechanical arm, and controls the mechanical arm to drive the end effector to move according to the motion instruction, so as to drive the experimental object to perform a target operation task; the target operation task belongs to the different operation tasks.

[0011] According to the multi-degree-of-freedom mechanical arm interactive device provided by the application, the device further comprises:

[0012] A general control computer is in communication connection with the controller, and the general control computer is used to send a first time synchronization instruction to the controller;

[0013] The general control computer is further used to send a first instruction to the mechanical arm through the controller, and the first instruction is used to edit different operation tasks;

[0014] A collection card is in electrical connection with the general control computer and the end effector, and the collection card is used to realize data interaction and time synchronization among different modules.

[0015] According to the multi-degree-of-freedom mechanical arm interactive device provided by the application, the device further comprises:

[0016] A binocular motion tracking module is in communication connection with the general control computer, and the binocular motion tracking module collects eye movement data of the experimental object;

[0017] An electroencephalogram signal collection module is in communication connection with the general control computer, and the electroencephalogram signal collection module is used to collect electroencephalogram signals of the experimental object;

[0018] The general control computer is used to send a second time synchronization instruction to the controller and a third time synchronization instruction to the electroencephalogram signal collection module;

[0019] The general control computer is further used to generate a second instruction according to at least one of the eye movement data and the electroencephalogram signals, so as to control the controller to control the pose state of the mechanical arm according to the second instruction.

[0020] According to the multi-degree-of-freedom mechanical arm interactive device provided by the application, the device further comprises:

[0021] A scene indicating lamp group is arranged in a visual field range of the experimental object and is in electrical connection with the collection card; the scene indicating lamp group comprises a plurality of scene indicating lamps, and different scene indicating lamps correspond to different operation tasks;

[0022] The general control computer is used to send a fourth time synchronization instruction to the scene indicating lamp group through the collection card;

[0023] The general control computer is further used to send a third instruction to the scene indicating lamp group through the collection card, so as to control the plurality of scene indicating lamps to change the working state.

[0024] According to the multi-degree-of-freedom mechanical arm interactive device provided by the application, the device further comprises:

[0025] a reward module, electrically connected with the general control computer, configured to provide water source or food for the experimental subject;

[0026] the general control computer is configured to send a fifth time synchronization instruction to the reward module;

[0027] the general control computer is further configured to send a fourth instruction to the reward module to control the working state of the reward module.

[0028] According to the multi-degree-of-freedom mechanical arm interactive device provided by the application, the end effector comprises:

[0029] a photoelectric sensor installed at the end of the end effector, the photoelectric sensor being configured to detect contact information of the experimental subject with the end effector, the contact information comprising at least one of contact area, contact time and pressure information.

[0030] The application further provides a control method of the multi-degree-of-freedom mechanical arm interactive device, comprising:

[0031] acquiring behavior data of the experimental subject based on the acquisition module;

[0032] generating a motion instruction based on the control module through at least one of the behavior data and pose feedback data of the mechanical arm, and controlling the mechanical arm to drive the end effector to move according to the motion instruction, so as to drive the experimental subject to perform a target operation task;

[0033] The end effector is detachably installed at the end of the mechanical arm, and the end effector is configured to be used by the experimental subject to perform different operation tasks; the mechanical arm comprises at least six degrees of freedom, and the mechanical arm is configured to move in a three-dimensional space; and the target operation task belongs to the different operation tasks. The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the control method of the multi-degree-of-freedom mechanical arm interactive device according to any one of the above when executing the computer program.

[0034] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the control method of the multi-degree-of-freedom mechanical arm interactive device according to any one of the above.

[0035] The application further provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the control method of the multi-degree-of-freedom mechanical arm interactive device according to any one of the above.

[0036] The application provides a multi-degree-of-freedom mechanical arm interactive device and a control method thereof. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0038] Figure 1 Fig. 1 is a structural schematic diagram of a multi-degree-of-freedom mechanical arm interactive device provided by the application.

[0039] Figure 2 Fig. 2 is another structural schematic diagram of the multi-degree-of-freedom mechanical arm interactive device provided by the application.

[0040] Figure 3 Fig. 3 is a third structural schematic diagram of the multi-degree-of-freedom mechanical arm interactive device provided by the application.

[0041] Figure 4 Fig. 4 is a flow schematic diagram of a control method of the multi-degree-of-freedom mechanical arm interactive device provided by the application.

[0042] Figure 5 Fig. 5 is a structural schematic diagram of an electronic device provided by the application.

[0043] Reference signs:

[0044] 100: acquisition module; 200: mechanical arm mechanism; 210: mechanical arm;

[0045] 220: end effector; 221: photoelectric sensor; 300: control module;

[0046] 400: general control computer; 500: acquisition card; 600: binocular motion tracking module;

[0047] 700: electroencephalogram acquisition module; 800: scene indicator light group; 900: reward module. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] The present application provides a multi-degree-of-freedom mechanical arm interactive device and a control method thereof. Figures 1-4 The present application provides a multi-degree-of-freedom mechanical arm interactive device and a control method thereof.

[0050] Figure 1 is one of the structural schematic diagrams of the multi-degree-of-freedom mechanical arm interactive device provided by the present application, as shown in the figure, the multi-degree-of-freedom mechanical arm interactive device comprises a collection module 100, a mechanical arm mechanism 200, a mechanical arm 210, an end effector 220 and a control module 300. Figure 1

[0051] The collection module 100 is used for collecting behavior data of an experimental object.

[0052] In this embodiment, the collection module 100 can be an image or video acquisition device, for example, the collection module 100 is a camera or an image recognition device with a camera function.

[0053] In this embodiment, the behavior data includes posture and movement information of the experimental object.

[0054] Taking a monkey as an example, the upper limb movement and interactive behavior of the monkey are recorded by the collection module 100, and the depth information is obtained through the collected behavior data, so as to identify the position of the hand and arm of the monkey in the three-dimensional space in real time, thereby facilitating subsequent data analysis.

[0055] The mechanical arm 210 mechanism 200 comprises the mechanical arm 210 and the end effector 220; the end effector 220 is detachably installed at the end of the mechanical arm 210, and the end effector 220 is used for the experimental object to perform different operation tasks; the mechanical arm 210 comprises at least 6 degrees of freedom, and the mechanical arm 210 is used for moving in a three-dimensional space.

[0056] In this embodiment, the mechanical arm 210 comprises a multi-joint linkage structure, for example, the mechanical arm 210 comprises a large arm, a small arm, a wrist and a base, and supports at least 6 degrees of freedom, so as to realize accurate positioning and flexible movement in a three-dimensional space through joint cooperative movement.

[0057] ​In this embodiment, the end effector 220 is replaceable to adapt to different task requirements; for example, the end effector 220 can adopt a standardized adapter interface (mechanical buckle or electromagnetic locking) to support quick installation and disassembly, and be compatible with different experimental task requirements (such as grasping, lifting, and touch feedback).

[0058] In this embodiment, the end effector 220 can also be installed on the end of the mechanical arm 210 through a screw.

[0059] In this embodiment, the end effector 220 can also integrate different functional modules, for example, the end effector 220 integrates a tactile sensor or a flexible clamping structure for real-time monitoring of the monkey's contact force to avoid damage caused by excessive force, and feedback to the control system through a data interface.

[0060] In this embodiment, the end effector 220 can be configured with customized actuators (such as a spherical grasping module or a lever triggering device) to adapt to different training scenarios (such as grasping target objects or pressing switches).

[0061] The control module 300 is electrically connected to the mechanical arm 210, and the control module 300 generates a motion instruction based on at least one of behavior data and pose feedback data of the mechanical arm 210, and controls the mechanical arm 210 to drive the end effector 220 to move according to the motion instruction to drive the experimental object to perform a target operation task; the target operation task belongs to different operation tasks.

[0062] In this embodiment, the control module 300 can be connected to the mechanical arm 210 through an electrical signal interface (such as UART or Bluetooth), receive task parameters (such as target pose and operation type) from the upper computer, and generate a motion instruction based on behavior data (experimental object motion trajectory) or real-time pose feedback (joint angle and end coordinate) of the mechanical arm 210.

[0063] In this embodiment, the control module 300 can receive real-time pose feedback recorded by the mechanical arm 210 through a network cable and send a motion instruction to the mechanical arm 210.

[0064] Specifically, in the case of detecting that the monkey contacts the end effector 220 to trigger an action, the control module 300 dynamically adjusts the pulling force to strengthen the motion training according to the behavior data of the monkey and the pose feedback of the mechanical arm 210; this embodiment adopts a six-degree-of-freedom mechanical arm 210 combined with a replaceable physical operation module to support the monkey to perform grasping, pushing and pulling actions in a three-dimensional space, which is highly consistent with the natural motion behavior pattern of primates.

[0065] The multi-degree-of-freedom mechanical arm interaction device provided by the embodiment of the application comprises a collecting module, a mechanical arm, a control module and an end effector.

[0066] In some embodiments, the multi-degree-of-freedom mechanical arm 210 interaction device further comprises a general control computer 400 and a collecting card 500.

[0067] The general control computer 400 is in communication connection with the controller, and the general control computer 400 is used for sending a first time synchronization instruction to the controller.

[0068] In this embodiment, the general control computer 400 can be connected with the controller through a UART / USB or CAN bus interface, and the NTP protocol or a custom timestamp protocol is adopted to generate and send the first time synchronization instruction, so as to ensure that the clock references of the two are aligned.

[0069] The general control computer 400 is further used for sending a first instruction to the mechanical arm 210 through the controller, and the first instruction is used for editing different operation tasks.

[0070] In this embodiment, the general control computer 400 can customize different test task paradigms, real-time control and feedback calculation, so as to ensure that the modules work cooperatively.

[0071] In this embodiment, the first instruction contains a task type, path planning parameters and end effector 220 configuration (such as clamping jaw opening and closing force), and the controller can dynamically adjust the joint angle of the mechanical arm 210 and the end posture according to the first instruction.

[0072] The general control computer 400 and the end effector 220 are respectively in electrical connection with the collecting card 500, and the collecting card 500 is used for realizing data interaction and time synchronization among different modules.

[0073] In this embodiment, the collecting card 500 is the data hub of the multi-degree-of-freedom mechanical arm 210 interaction device, and the collecting card 500 can be responsible for collecting and synchronizing transmission data among multiple modules through an IO port.

[0074] The multi-degree-of-freedom mechanical arm interaction device provided by the embodiment of the application realizes time synchronization through the controller, the general control computer also sends the first instruction to the mechanical arm through the controller to edit different operation tasks, supports the experimental object to perform more complex training tasks, the acquisition card is arranged to realize data interaction and time synchronization between different modules, and the interaction efficiency of the multi-degree-of-freedom mechanical arm interaction device is improved.

[0075] In some embodiments, the multi-degree-of-freedom mechanical arm interaction device further comprises an eye movement tracking module 600 and an electroencephalogram signal acquisition module 700100.

[0076] The eye movement tracking module 600 is in communication connection with the general control computer 400, and the eye movement tracking module 600 acquires eye movement data of the experimental object.

[0077] In this embodiment, the eye movement tracking module 600 comprises two monocular eye movement instruments, which are used to monitor the eye movement information of the monkey in real time and provide the gaze point position of the monkey during the movement.

[0078] The electroencephalogram signal acquisition module 700100 is in communication connection with the general control computer 400, and the electroencephalogram signal acquisition module 700100 is used to acquire the electroencephalogram signal of the experimental object.

[0079] In this embodiment, the electroencephalogram signal acquisition module 700100 comprises two electroencephalogram acquisition devices to realize multi-brain region and 128-channel signal acquisition.

[0080] The general control computer 400 is used to send a second time synchronization instruction to the controller and a third time synchronization instruction to the electroencephalogram signal acquisition module 700100.

[0081] In this embodiment, the general control computer 400 realizes multi-module clock alignment through the second time synchronization instruction (for the controller) and the third time synchronization instruction (for the electroencephalogram module).

[0082] In this embodiment, the second time synchronization instruction and the third time synchronization instruction can be NTP protocol or hardware trigger signal (such as EXTI interruption of STM32).

[0083] For example, the general control computer 400 sends the second time synchronization instruction to the eye movement tracking module 600 through a network cable, realizes time synchronization between the general control computer 400 and the eye movement tracking module 600, and ensures that the joint movement timing of the mechanical arm 210 matches the eye movement data acquisition.

[0084] For example, the general control computer 400 sends a third time synchronization instruction to the electroencephalogram signal acquisition module 700100 through the IO port, realizes time synchronization between the general control computer 400 and the electroencephalogram signal acquisition module 700100, and ensures that the motion state of the experimental object matches the corresponding electroencephalogram signal fluctuation trend.

[0085] The general control computer 400 is further configured to generate a second instruction according to at least one of the eye movement data and the electroencephalogram signal, so as to control the controller to control the pose state of the mechanical arm 210 according to the second instruction.

[0086] In this embodiment, the general control computer 400 obtains the change of the fixation point position of the monkey during the motion process according to the eye movement data, and determines the motion intention of the monkey during the motion process according to the electroencephalogram signal; when the eye movement data and the electroencephalogram signal are valid at the same time, the general control computer 400 can comprehensively generate the second instruction by using a dynamic weighted fusion algorithm.

[0087] For example, the weight of the eye movement data focuses on the spatial positioning accuracy, and the weight of the electroencephalogram signal focuses on the motion intention confidence; if the eye movement data indicates target deviation but the electroencephalogram signal maintains the original instruction, the path re-planning (such as a circular arc obstacle avoidance trajectory) is triggered by the second instruction to avoid collision between the mechanical arm 210 and the obstacle; when the emergency stop intention in the electroencephalogram signal is detected, the current task can be interrupted immediately and the safety shutdown protocol is activated by the second instruction.

[0088] The multi-degree-of-freedom mechanical arm interaction device provided in the embodiment of the application collects the eye movement data of the experimental object through the dual eye movement tracking module, collects the electroencephalogram signal of the experimental object through the electroencephalogram signal acquisition module, and then generates a second instruction according to at least one of the eye movement data and the electroencephalogram signal by using the general control computer, so as to control the controller to control the pose state of the mechanical arm according to the second instruction. The behavior data and the eye movement data are integrated to realize synchronous recording of hand-eye coordination data, and the multi-brain region invasive neural signal (i.e., the electroencephalogram signal) is recorded synchronously, so that high-quality multi-channel multi-dimensional biological data recording is realized.

[0089] In some embodiments, the multi-degree-of-freedom mechanical arm 210 interaction further includes a scene indicator light group 800.

[0090] The scene indicator light group 800 is disposed in the visual field range of the experimental object and is electrically connected with the acquisition card 500; the scene indicator light group 800 includes a plurality of scene indicator lights, and different scene indicator lights correspond to different operation tasks.

[0091] In this embodiment, at least one of the shape, color or brightness of the scene indicator light and the like can be used as a condition for triggering the stress behavior of the experimental object.

[0092] For example, three different color (red, green, blue) scene indicator lights can be set in the monkey's field of view, wherein the monkey sees the red scene indicator light lit up, executes the A action; the monkey sees the green scene indicator light lit up, executes the B action; the monkey sees the blue scene indicator light lit up, executes the C action.

[0093] It should be noted that the monkey automatically triggers the corresponding emergency behavior when seeing different color lights lit up, which can be achieved by pre-training the monkey; for example, through previous multiple artificial training, help the monkey to establish the conditioned reflex of executing different actions when different color lights are lit.

[0094] The general control computer 400 is configured to send a fourth time synchronization instruction to the scene indicator light group 800 through the acquisition card 500.

[0095] In this embodiment, the general control computer 400 sends a hardware trigger pulse signal (such as TTL level) as the fourth time synchronization instruction through the acquisition card 500, to ensure that the flashing frequency of the scene indicator light group 800 is strictly synchronized with the action of the mechanical arm 210, the brain electrical signal acquisition module and the like.

[0096] The general control computer 400 is also configured to send a third instruction to the scene indicator light group 800 through the acquisition card 500 to control the multiple scene indicator lights to change the working state.

[0097] In this embodiment, the general control computer 400 can send the third instruction to the scene indicator light group 800 through the PWM dimming protocol, dynamically adjust the brightness (0-100% adjustable), flashing mode (constant / 1Hz / 5Hz) and color combination (RGB mixed color) of each indicator light, to match the operation requirements of different experimental stages.

[0098] The multi-degree-of-freedom mechanical arm interactive device provided by the embodiment of the present application can use the scene indicator light to assist in training the experimental object, improve the attention of the monkey to the target, and further improve the interaction efficiency between the experimental object and the multi-degree-of-freedom mechanical arm interactive device.

[0099] In some embodiments, the multi-degree-of-freedom mechanical arm 210 interactive device further comprises a reward module 900.

[0100] The reward module 900 is electrically connected with the general control computer 400, and the reward module 900 is configured to provide water source or food for the experimental object.

[0101] In this embodiment, the reward module 900 is deployed in the reach of the experimental subject, and is connected with the general control computer 400 through a CAN bus or an RS-485 interface to receive instructions for adjusting the reward content.

[0102] In this embodiment, the reward module 900 can adopt a quantitative output mechanism driven by a micro-servo motor, support water infusion or food particle release, and realize dose feedback control through a high-precision flow sensor and a photoelectric switch to ensure that the reward amount error is lower than a set error threshold.

[0103] In this embodiment, the general control computer 400 is configured to send a fifth time synchronization instruction to the reward module 900.

[0104] In this embodiment, the general control computer 400 can send a hardware-level synchronization pulse signal (such as a TTL level) to the reward module 900 through ROS node communication, that is, the fifth time synchronization instruction, to ensure that the timing deviation of the reward trigger and the motion of the robot arm 210, the brain electrical signal acquisition module, and other modules is within a controllable range.

[0105] In this embodiment, the general control computer 400 strictly aligns the joint motion completion event of the robot arm 210 with the reward release instruction through the fifth time synchronization instruction, avoiding the training effect attenuation caused by behavior feedback delay.

[0106] The general control computer 400 is further configured to send a fourth instruction to the reward module 900 to control the working state of the reward module 900.

[0107] In this embodiment, the working state of the reward module 900 includes: (1) opening the water source slot and closing the food slot; (2) closing the water source slot and opening the food slot; (3) opening the water source slot and the food slot; and (4) closing the water source slot and the food slot.

[0108] In this embodiment, the general control computer 400 generates the fourth instruction according to the experimental task state (such as target grasping success or path planning timeout), and sets the following reward modes for the reward module 900:

[0109] (1) Trigger mode: releasing a preset dose of reward immediately after the robot arm 210 completes a task;

[0110] (2) Cumulative mode: increasing the reward amount according to the number of consecutive task completions to strengthen the long-term learning effect.

[0111] In this embodiment, the reward module 900 dynamically adjusts the reward type (water source / food) and the dose according to the generated fourth instruction to strengthen the positive behavior feedback of the experimental subject.

[0112] The multi-degree-of-freedom mechanical arm interaction device provided by the embodiment of the present application can provide water or food for the experimental object through the reward module, establish time synchronization between the general control computer and the reward module, and send the fourth instruction to the reward module to control the working state of the reward module, so that the learning efficiency and task participation of the experimental object are improved.

[0113] In some embodiments, the end effector 220 includes a photoelectric sensor 221.

[0114] The photoelectric sensor 221 is installed at the end of the end effector 220, and is used to detect contact information of the experimental object and the end effector 220, the contact information including at least one of contact area, contact time and pressure information.

[0115] In this embodiment, the general control computer 400 can determine the contact condition of the experimental object and the end effector 220 according to the contact information, to decide whether to start the current interaction training or dynamically adjust the grasping strategy of the experimental object acting on the end effector 220.

[0116] For example, if the contact area of the experimental object and the end effector 220 exceeds the area threshold, it is determined that the experimental object is ready to perform the target operation task, and the current interaction detection process is started.

[0117] For example, the general control computer 400 dynamically adjusts the contact information of the experimental object to the end effector 220 based on the photoelectric sensor 221 signal:

[0118] (1) Contact area control: when the contact area is lower than the preset threshold (such as 50% of the target object surface area), the posture of the end effector 220 is adjusted to increase the contact area;

[0119] (2) Contact time control: if the contact time is lower than the time threshold (such as 10 seconds), the posture of the end effector 220 is adjusted to increase the contact area.

[0120] (3) Contact pressure control: if the contact pressure is lower than the pressure threshold (set according to user requirements, not lower than the force of the experimental object normally grasping an object), the posture of the end effector 220 is adjusted to increase the contact pressure.

[0121] The multi-degree-of-freedom mechanical arm interaction device provided by the embodiment of the present application can detect the contact information of the experimental object and the end effector by setting the photoelectric sensor at the end of the end effector, so as to ensure the safety of the experimental object operating the end effector and the stability of the interaction experiment.

[0122] Figure 2Figure 2 is a structural schematic diagram of the multi-degree-of-freedom mechanical arm interactive device provided by the present application, and Figure 2 In the embodiment shown, the multi-brain region invasive brain electrical signal acquisition system (corresponding to the brain electrical signal acquisition module) is used to acquire the brain electrical signal of the experimental animal, and the binocular movement tracking module is used to acquire the eye movement data of the experimental animal; the controller (see Figure 2 the lower right side of the computer icon) acquires the behavior data of the experimental animal through the motion capture system (corresponding to the acquisition module), generates a motion instruction according to the behavior data, and controls the six-degree-of-freedom mechanical arm (corresponding to the mechanical arm) to drive the modular end effector (corresponding to the end effector) to move, so as to drive the experimental animal to perform the motion training; the general control computer is electrically connected with the controller, and controls the working state of the scene indicator light system and the feeding / water replacement device (corresponding to the reward system) through the acquisition card, so as to assist the experimental animal in the reinforcement learning training.

[0123] Figure 3 Figure 3 is a structural schematic diagram of the multi-degree-of-freedom mechanical arm interactive device provided by the present application, and Figure 3 In the embodiment shown, the computer 1 (corresponding to the controller) is connected with the camera through a USB, connected with the mechanical arm through a network cable, and connected with the general control computer through an IO port; the general control computer is connected with the feeding / water system (corresponding to the reward system) through a USB, connected with the brain electrical signal acquisition module through an IO port, and connected with the binocular movement tracking module through a network cable; the general control computer can send a time synchronization instruction to the above directly connected modules to realize time alignment, and the general control computer is also electrically connected with the acquisition card; the end effector and the scene indicator light group are respectively connected with the acquisition card through an IO port, that is, the general control computer can send a time synchronization instruction to the end effector and the scene indicator light group through the acquisition card to realize time alignment; the mechanical arm is physically connected with the end effector; the camera can acquire images in real time and send them to the computer for storage; the computer 1 can generate a motion instruction according to the received images and send it to the mechanical arm to execute the motion instruction and record the real-time pose of the mechanical arm.

[0124] In this embodiment, when the photoelectric sensor on the end effector detects that the experimental object contacts the end effector, a photoelectric signal is generated and sent to the general control computer through the acquisition card, the general control computer confirms that the experimental object is ready to perform the target operation task according to the photoelectric signal, and sends the behavior data to the computer 1, the computer 1 generates a motion instruction according to the behavior data, and controls the mechanical arm to execute the motion instruction, so as to drive the experimental object at the end effector to perform the motion learning.

[0125] The control method of the multi-degree-of-freedom mechanical arm interactive device provided by the present application is described below, and the control method of the multi-degree-of-freedom mechanical arm interactive device described below can be mutually corresponding to the multi-degree-of-freedom mechanical arm interactive device described above.

[0126] Figure 4 is a flowchart of a control method of a multi-degree-of-freedom robotic arm interactive device provided by the present application, as shown in Figure 4 The control method of the multi-degree-of-freedom robotic arm interactive device comprises the following steps:

[0127] Step 410, collecting behavior data of the experimental object based on the collection module.

[0128] In this step, the collection module can be an image or video acquisition device, for example, the collection module is a camera or an image recognition device with camera function.

[0129] In this step, the behavior data includes the posture and movement information of the experimental object.

[0130] Taking a monkey as an example, the upper limb movement and interactive behavior of the monkey are recorded by the collection module, and the depth information is obtained through the collected behavior data to identify the position of the monkey's hand and arm in the three-dimensional space in real time, which facilitates subsequent data analysis.

[0131] Step 420, generating a motion instruction based on at least one of the behavior data and the pose feedback data of the robotic arm, and controlling the robotic arm to drive the end effector to move according to the motion instruction to drive the experimental object to perform a target operation task; wherein the end effector is detachably installed at the end of the robotic arm, the end effector is used for the experimental object to perform different operation tasks; the robotic arm includes at least 6 degrees of freedom, and the robotic arm is used for moving in a three-dimensional space; the target operation task belongs to different operation tasks.

[0132] In this step, the robotic arm includes a multi-joint linkage structure, for example, the robotic arm includes a large arm, a small arm, a wrist and a base structure, supports at least 6 degrees of freedom, and realizes accurate positioning and flexible movement in a three-dimensional space through joint cooperative movement.

[0133] In this step, the end effector can be replaced to adapt to different task requirements; for example, the end effector can adopt a standardized adapter interface (mechanical buckle or electromagnetic locking), support quick installation and disassembly, and be compatible with different experimental task requirements (such as grasping, lifting, touch feedback).

[0134] In this embodiment, the end effector can also be installed at the end of the robotic arm through a screw.

[0135] In this embodiment, the end effector can also integrate different functional modules, for example, the end effector integrates a tactile sensor or a flexible clamping structure, which is used to monitor the contact force of the monkey in real time, avoid injury caused by excessive force, and feedback to the control system through a data interface.

[0136] In this embodiment, the end effector can be configured as a customized end effector (such as a sphere gripping module, a lever triggering device), which is suitable for different training scenarios (such as grabbing target objects, pressing switches).

[0137] The control module is electrically connected with the robot arm, and the control module generates a motion instruction through at least one of behavior data and pose feedback data of the robot arm, and controls the robot arm to drive the end effector to move according to the motion instruction, so as to drive the experimental object to perform a target operation task; the target operation task belongs to different operation tasks.

[0138] In this embodiment, the control module can be connected with the robot arm through an electrical signal interface (such as UART, Bluetooth), receive task parameters (such as target pose, operation type) from the upper computer, and generate a motion instruction based on behavior data (motion trajectory of the experimental object) or real-time pose feedback (joint angle, end coordinate) of the robot arm.

[0139] In this embodiment, the control module can receive real-time pose feedback recorded by the robot arm through a network cable, and send a motion instruction to the robot arm.

[0140] Specifically, in the case of detecting that the monkey contacts the end effector to trigger the action, the control module dynamically adjusts the traction degree according to the behavior data of the monkey and the pose feedback of the robot arm to strengthen the motion training; this embodiment adopts a six-degree-of-freedom robot arm matched with a replaceable physical operation module, supports the monkey to perform grabbing, pushing and pulling actions in a three-dimensional space, and is highly consistent with the natural motion behavior pattern of primates.

[0141] The control method of the multi-degree-of-freedom robot arm interaction device provided by the embodiment of the application improves the flexibility of the robot arm in the three-dimensional space and the grasping ability of the system for different tasks, and further improves the interaction performance of the mechanical interaction system.

[0142] Figure 5 An example of an electronic device is shown in the physical structure diagram, such as Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke the logic instructions in the memory 530 to execute the control method of the multi-degree-of-freedom mechanical arm interactive device, the method comprising: collecting behavior data of an experimental object based on a collection module; generating a motion instruction based on a control module through at least one of the behavior data and pose feedback data of the mechanical arm, and controlling the mechanical arm to drive an end effector to move according to the motion instruction to drive the experimental object to perform a target operation task; wherein the end effector is detachably installed at the end of the mechanical arm, and the end effector is used for the experimental object to perform different operation tasks; the mechanical arm includes at least 6 degrees of freedom, and the mechanical arm is used to move in a three-dimensional space; and the target operation task belongs to different operation tasks.

[0143] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0144] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer readable storage medium, and the computer program being capable of executing the control method of the multi-degree-of-freedom robotic arm interactive device provided by the above-mentioned methods when executed by a processor, the method comprising: collecting behavior data of an experimental object based on a collection module; generating a motion instruction based on a control module through at least one of the behavior data and pose feedback data of the robotic arm, and controlling the robotic arm to drive an end effector to move according to the motion instruction, so as to drive the experimental object to perform a target operation task; wherein the end effector is detachably installed at the end of the robotic arm, and the end effector is used for the experimental object to perform different operation tasks; the robotic arm comprises at least 6 degrees of freedom, and the robotic arm is used for moving in a three-dimensional space; and the target operation task belongs to different operation tasks.

[0145] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is capable of executing the control method of the multi-degree-of-freedom robotic arm interactive device provided by the above-mentioned methods when executed by a processor, the method comprising: collecting behavior data of an experimental object based on a collection module; generating a motion instruction based on a control module through at least one of the behavior data and pose feedback data of the robotic arm, and controlling the robotic arm to drive an end effector to move according to the motion instruction, so as to drive the experimental object to perform a target operation task; wherein the end effector is detachably installed at the end of the robotic arm, and the end effector is used for the experimental object to perform different operation tasks; the robotic arm comprises at least 6 degrees of freedom, and the robotic arm is used for moving in a three-dimensional space; and the target operation task belongs to different operation tasks.

[0146] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0147] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-degree-of-freedom robotic arm interaction device, characterized in that, include: The data acquisition module is used to collect behavioral data of the experimental subjects, including the posture and movement information of the experimental subjects. The experimental subjects were non-human primates; Robotic arm mechanism, including robotic arm and end effector; The end effector is detachably mounted on the end of the robotic arm, and the end effector is used to enable the experimental object to perform different operational tasks; The robotic arm includes at least 6 degrees of freedom and is used to move in three-dimensional space to support the experimental object in performing grasping, pulling and pushing operations in three-dimensional space; A control module is electrically connected to the robotic arm. The control module generates motion commands based on the behavior data and the pose feedback data of the robotic arm, and controls the robotic arm to drive the end effector to move according to the motion commands, so as to drive the experimental object to perform the target operation task. The target operation task belongs to the different operation tasks; The end effector includes: A photoelectric sensor is installed at the end of the end effector. The photoelectric sensor is used to detect the contact information between the experimental object and the end effector. The contact information includes at least one of the following: contact area, contact time, and pressure information.

2. The multi-degree-of-freedom robotic arm interaction device according to claim 1, characterized in that, The device further includes: A central control computer is communicatively connected to the control module, and the central control computer is used to send a first-time synchronization command to the control module; The central control computer is also used to send the first instruction to the robotic arm through the control module, and the first instruction is used to edit different operation tasks; The acquisition card is electrically connected to both the central control computer and the end effector. The acquisition card is used to realize data interaction and time synchronization between different modules.

3. The multi-degree-of-freedom robotic arm interaction device according to claim 2, characterized in that, The device further includes: A binoculars tracking module is communicatively connected to the central control computer, and the binoculars tracking module collects eye movement data of the experimental subjects; The EEG signal acquisition module is communicatively connected to the central control computer, and the EEG signal acquisition module is used to acquire the EEG signals of the experimental subject. The central control computer is used to send a second time synchronization command to the control module and a third time synchronization command to the EEG signal acquisition module; The central control computer is also configured to generate a second instruction based on at least one of the eye-tracking data and the electroencephalogram (EEG) signal, so as to control the control module to control the pose state of the robotic arm according to the second instruction.

4. The multi-degree-of-freedom robotic arm interaction device according to claim 2, characterized in that, The device further includes: A scene indicator light group is deployed within the field of view of the experimental object and is electrically connected to the data acquisition card; the scene indicator light group includes multiple scene indicator lights, and different scene indicator lights correspond to different operation tasks; The central control computer is used to send a fourth time synchronization command to the scene indicator light group through the acquisition card; The central control computer is also used to send a third instruction to the scene indicator group through the acquisition card to control the multiple scene indicator lights to change their working status.

5. The multi-degree-of-freedom robotic arm interaction device according to claim 2, characterized in that, The device further includes: A reward module, which is electrically connected to the central control computer, is used to provide water or food to the experimental subjects. The central control computer is used to send a fifth time synchronization command to the reward module; The central control computer is also used to send a fourth instruction to the reward module to control the working status of the reward module.

6. A control method for a multi-degree-of-freedom robotic arm interaction device, characterized in that, include: The behavioral data of the experimental object is collected by the acquisition module, and the behavioral data includes the posture and movement information of the experimental object. The experimental subjects were non-human primates; The control module generates motion commands based on the behavioral data and the pose feedback data of the robotic arm, and controls the robotic arm to drive the end effector to move according to the motion commands, so as to drive the experimental object to perform the target operation task. The end effector is detachably mounted on the end of the robotic arm and is used to enable the experimental object to perform different operational tasks. The robotic arm has at least six degrees of freedom and is used to move in three-dimensional space to support the experimental object in performing grasping, pulling, and pushing operations in three-dimensional space. The target operational task belongs to the different operational tasks. The end effector includes: A photoelectric sensor is installed at the end of the end effector. The photoelectric sensor is used to detect the contact information between the experimental object and the end effector. The contact information includes at least one of the following: contact area, contact time, and pressure information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the control method of the multi-degree-of-freedom robotic arm interaction device as described in claim 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the control method of the multi-degree-of-freedom robotic arm interaction device as described in claim 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method of the multi-degree-of-freedom robotic arm interaction device as described in claim 6.

Images