Multi-source heterogeneous data synchronous mechanical arm bionic control method and device and electronic equipment

Abstract

The invention provides a multi-source heterogeneous data synchronous mechanical arm bionic control method and device and electronic equipment, and relates to the technical field of robot control, and the method comprises the following steps: decoding hand joint angle data and a synchronization timestamp thereof, wrist pose data and a synchronization timestamp thereof, and arm pose data and a synchronization timestamp thereof, and obtaining a decoding result; the decoded hand joint angle data, the decoded wrist pose data and the decoded arm pose data are transmitted to a mechanical arm controller through a low-delay network; and the mechanical arm controller executes dynamic remapping based on the decoded hand joint angle data, the decoded wrist pose data and the decoded arm pose data, and simulation control of the mechanical arm is achieved. According to the method, the arm pose data participates in dynamic correction of the base coordinate system, space positioning deviation is eliminated, and the mechanical arm reproduces actions; timing sequence block optimization considers fine grabbing and long track actions, and action restoration authenticity is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a bionic control method, device and electronic equipment for a robotic arm with multi-source heterogeneous data synchronization. Background Art

[0002] In the field of bionic manipulation of robotic arms, existing technologies have three core bottlenecks: 1. The motion capture equipment, robotic arm controller, and vision system use independent clocks, resulting in significant time skew between hand motion data, robotic arm control commands, and the video stream. When the operator performs delicate movements like grasping, the robotic arm's response lags, resulting in "missed grasps" or "object collisions." Multi-source data, due to protocol differences, creates information silos, making it difficult for the recording system to establish accurate spatiotemporal correlations.

[0003] 2. Traditional inverse kinematics solvers produce sudden changes in joint angles during state transitions, causing the robot arm's trajectory to be distorted or even freeze. This is especially true during high-dynamic movements like handshakes and push-pull movements, where the robot arm cannot replicate the subtle manipulations of a human hand.

[0004] 3. Data collection, forward reasoning, and playback verification require independent deployment of separate systems. Operators must repeatedly restart the hardware and manually switch software modules, resulting in lengthy preparation times for crawling tasks and incompatible data formats between modes.

[0005] Therefore, a bionic control method, device and electronic device for a robotic arm with multi-source heterogeneous data synchronization are proposed. Summary of the Invention

[0006] This specification provides a bionic control method, device and electronic equipment for a robotic arm with synchronized multi-source heterogeneous data. The arm posture data participates in the dynamic correction of the base coordinate system, eliminating spatial positioning deviations and enabling the robotic arm to reproduce movements. The timing block optimization takes into account both fine grasping and long-trajectory movements, improving the authenticity of movement restoration.

[0007] This specification provides a bionic control method for a robotic arm with synchronized multi-source heterogeneous data, including: Obtaining the hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by the motion capture device; Decoding the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmitting the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to the robotic arm controller via a low-latency network; The robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data and the decoded arm posture data to achieve simulated control of the robotic arm.

[0008] Optionally, the generation of the synchronization timestamp includes: using a computer host as the master clock, a robotic arm controller and a motion capture device as slave clocks, the master clock timing the slave clock via Ethernet, and the time error between the master clock and the slave time satisfies a preset error value.

[0009] Optionally, before the robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to realize simulated manipulation of the robotic arm, the process includes: Determining whether the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data meet a preset deviation value; When the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data do not meet the preset deviation value, normalization processing is performed on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data.

[0010] Optionally, performing normalization processing on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data includes: in, is the normalized hand joint angle data or the normalized wrist posture data or the normalized arm posture data, is the decoded hand joint angle data or the decoded wrist posture data or the decoded arm posture data, and All are historical training set statistics.

[0011] Optionally, the robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data, including: Mapping the decoded human hand joint angle data into target joint angle data of the dexterous hand; Converting the decoded arm posture data into a correction matrix of the robot arm base coordinate system; The robot arm kinematic model is updated based on the correction matrix, and the decoded wrist posture data is converted into the target posture data of the robot arm end in the correction base coordinate system; the target posture data is solved by inverse kinematics to obtain the target angle of the robot arm joint.

[0012] Optionally, the realization of simulated manipulation of the robotic arm includes: The simulation control mode of the robotic arm includes forward reasoning mode, data acquisition mode, and playback verification mode; When the simulation control mode of the robotic arm is the forward reasoning mode, the master control node is started to coordinate the control flow, and the behavior clone model is loaded to generate a prediction sequence of the robotic arm joint angles; When the simulation control mode of the robotic arm is the data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events; When the simulation control mode of the robotic arm is the playback verification mode, the playback engine is started to drive the robotic arm to reproduce the historical action sequence.

[0013] Optionally, when the simulation control mode of the robotic arm is a data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events, further includes: Identify the current action type; When the current action type is a fine grasping action, a short block length of the first preset frame is used to retain the finger micro-manipulation details; When the current motion type is a continuous trajectory motion, a long block length of the second preset frame is used to capture the complete motion trend; The block length is dynamically calibrated based on the average rate of change of joint angles.

[0014] This specification provides a bionic manipulator control device for synchronizing multi-source heterogeneous data, including: An acquisition module is used to acquire the hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by the motion capture device; a decoding module, configured to decode the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmit the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to a robotic arm controller via a low-latency network; The simulation module is used to perform dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data and the decoded arm posture data through the robotic arm controller to realize simulated control of the robotic arm.

[0015] Optionally, the generation of the synchronization timestamp includes: using a computer host as the master clock, a robotic arm controller and a motion capture device as slave clocks, the master clock timing the slave clock via Ethernet, and the time error between the master clock and the slave time satisfies a preset error value.

[0016] Optionally, before the simulation module, the following is included: Determining whether the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data meet a preset deviation value; When the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data do not meet the preset deviation value, normalization processing is performed on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data.

[0017] Optionally, performing normalization processing on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data includes: in, is the normalized hand joint angle data or the normalized wrist posture data or the normalized arm posture data, is the decoded hand joint angle data or the decoded wrist posture data or the decoded arm posture data, and All are historical training set statistics.

[0018] Optionally, the robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data, including: Mapping the decoded human hand joint angle data into target joint angle data of the dexterous hand; Converting the decoded arm posture data into a correction matrix of the robot arm base coordinate system; The robot arm kinematic model is updated based on the correction matrix, and the decoded wrist posture data is converted into the target posture data of the robot arm end in the correction base coordinate system; the target posture data is solved by inverse kinematics to obtain the target angle of the robot arm joint.

[0019] Optionally, the realization of simulated manipulation of the robotic arm includes: The simulation control mode of the robotic arm includes forward reasoning mode, data acquisition mode, and playback verification mode; When the simulation control mode of the robotic arm is the forward reasoning mode, the master control node is started to coordinate the control flow, and the behavior clone model is loaded to generate a prediction sequence of the robotic arm joint angles; When the simulation control mode of the robotic arm is the data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events; When the simulation control mode of the robotic arm is the playback verification mode, the playback engine is started to drive the robotic arm to reproduce the historical action sequence.

[0020] Optionally, when the simulation control mode of the robotic arm is a data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events, further includes: Identify the current action type; When the current action type is a fine grasping action, a short block length of the first preset frame is used to retain the finger micro-manipulation details; When the current motion type is a continuous trajectory motion, a long block length of the second preset frame is used to capture the complete motion trend; The block length is dynamically calibrated based on the average rate of change of joint angles.

[0021] This specification also provides an electronic device, wherein the electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform any of the above methods.

[0022] This specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, any of the above methods is implemented.

[0023] In the present invention, a master-slave clock timing mechanism aligns motion capture, robotic arm control, and video stream timestamps, achieving synchronization between human hand grasping movements and robotic arm responses. Arm posture data participates in the dynamic correction of the base coordinate system, eliminating spatial positioning deviations and enabling the robotic arm to reproduce movements. Normalization balances scale differences in multi-source data and suppresses the risk of joint mutations. Timing block optimization takes into account both fine grasping and long-trajectory movements, improving the authenticity of movement restoration. Forward reasoning, data acquisition, and playback verification are conveniently switched between the three modes, sharing a spatiotemporal alignment data stream. The behavioral cloning model outputs adaptive hierarchical control of joint angle sequences, autonomously switching between high-speed motion and precision operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of the principle of a bionic control method for a robotic arm with multi-source heterogeneous data synchronization provided in an embodiment of this specification; Figure 2 A schematic diagram of the principle of time synchronization provided in the embodiments of this specification; Figure 3 A schematic diagram of the principle of switching the simulation control mode of the robotic arm provided in the embodiments of this specification; Figure 4 A schematic diagram of the structure of a bionic manipulator control device for synchronizing multi-source heterogeneous data provided in an embodiment of this specification; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification; Figure 6 A schematic diagram of a computer-readable medium provided in accordance with an embodiment of this specification. DETAILED DESCRIPTION

[0026] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0027] The following is combined with Figure 1-6The exemplary embodiments of the present invention are described more fully. However, the exemplary embodiments can be implemented in various forms, and the present invention should not be construed as being limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, and more conveniently conveys the inventive concept to those skilled in the art. In the figures, the same reference numerals represent the same or similar elements, components, or parts, and thus their repeated description will be omitted.

[0028] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0029] In the description of specific embodiments, the features, structures, characteristics, or other details of the present invention are described to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from practicing the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0030] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0033] Figure 1 A schematic diagram of the principle of a bionic manipulation method for a robotic arm with multi-source heterogeneous data synchronization provided in an embodiment of this specification may include: S110: Acquire hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by a motion capture device; In this specific implementation, a human operator wears motion capture gloves to perform grasping movements. The gloves' built-in sensors collect real-time data on the hand's 21 joint angles, the wrist's 6-degree-of-freedom position, and the arm's spatial orientation. A host computer assigns a PTP-synchronized timestamp to each frame and transmits the data to the robotic arm controller via low-latency Wi-Fi. The controller also receives a YUV-formatted hand video stream captured by a binocular camera, ensuring spatiotemporal alignment of the motion data with the visual information.

[0034] Optionally, the generation of the synchronization timestamp includes: using a computer host as the master clock, a robotic arm controller and a motion capture device as slave clocks, the master clock timing the slave clock via Ethernet, and the time error between the master clock and the slave time satisfies a preset error value.

[0035] In the specific implementation of this specification, Figure 2 As shown, the host computer serves as the reference clock source, broadcasting synchronization frames to the robotic arm controller via Ethernet. The motion capture glove is aligned with the host clock using the NTP protocol, and the visual camera receives hardware synchronization signals triggered by the controller. When the operator flexes or extends their fingers, the system records the start of the movement. The controller receives timestamped finger joint data within a preset time, achieving closed-loop synchronization between human movement and mechanical response.

[0036] S120: Decoding the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmitting the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to a robotic arm controller via a low-latency network; Optionally, before S130, the steps include: Determining whether the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data meet a preset deviation value; When the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data do not meet the preset deviation value, normalization processing is performed on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data.

[0037] In a specific embodiment of this specification, when a human hand suddenly accelerates to grasp an object, and the joint angle variation exceeds a normal threshold, the system calls a database of historical grasping actions to calculate the average range of joint angle variation. This normalizes the wrist pose data to eliminate outliers caused by rapid flips, allowing the robot hand to smoothly replicate the gradual change in grasping force.

[0038] Optionally, performing normalization processing on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data includes: in, is the normalized hand joint angle data or the normalized wrist posture data or the normalized arm posture data, is the decoded hand joint angle data or the decoded wrist posture data or the decoded arm posture data, and All are historical training set statistics.

[0039] In this specific implementation, an expert action library is used as a historical training set to calculate the mean and standard deviation of finger joint flexion angles during grasping. Real-time joint data is collected, with the mean subtracted and divided by the standard deviation, to map the original angle values ​​to a preset range. This processed data is then fed into a dynamic remapping module to prevent a single joint from dominating overall control.

[0040] S130: The robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to achieve simulated control of the robotic arm.

[0041] Optionally, the robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data, including: Mapping the decoded human hand joint angle data into target joint angle data of the dexterous hand; Converting the decoded arm posture data into a correction matrix of the robot arm base coordinate system; The robot arm kinematic model is updated based on the correction matrix, and the decoded wrist posture data is converted into the target posture data of the robot arm end in the correction base coordinate system; the target posture data is solved by inverse kinematics to obtain the target angle of the robot arm joint.

[0042] In the specific implementation of this specification, the bending angle of the human index finger is mapped to the corresponding joint of the dexterous robotic hand; the offset matrix of the robotic arm base is calculated based on the pitch angle of the operator's arm; in the corrected base coordinate system, the spatial position of the wrist is converted into the target position of the end of the robotic arm; the robotic arm motion chain is analyzed through the URDF model, and the Jacobi pseudo-inverse method is used to solve the joint target angle to complete the closed-loop control of the human hand push-pull-robotic arm reproduction.

[0043] Optionally, the realization of simulated manipulation of the robotic arm includes: The simulation control mode of the robotic arm includes forward reasoning mode, data acquisition mode, and playback verification mode; When the simulation control mode of the robotic arm is the forward reasoning mode, the master control node is started to coordinate the control flow, and the behavior clone model is loaded to generate a prediction sequence of the robotic arm joint angles; When the simulation control mode of the robotic arm is the data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events; When the simulation control mode of the robotic arm is the playback verification mode, the playback engine is started to drive the robotic arm to reproduce the historical action sequence.

[0044] In the specific implementation of this specification, Figure 3 As shown in the figure, in forward inference mode, a pre-trained behavioral cloning model is loaded to generate a sequence of robotic arm joint angles based on a real-time hand video stream, enabling autonomous grasping. In data acquisition mode, a remapping process converts human hand movements into robotic arm pose commands, while a keyboard listener records start / end marker events to construct an expert demonstration dataset. In playback verification mode, the robotic arm is driven to reproduce historical grasping movements to verify the validity of newly acquired data and model outputs.

[0045] Optionally, when the simulation control mode of the robotic arm is a data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events, further includes: Identify the current action type; When the current action type is a fine grasping action, a short block length of the first preset frame is used to retain the finger micro-manipulation details; When the current motion type is a continuous trajectory motion, a long block length of the second preset frame is used to capture the complete motion trend; The block length is dynamically calibrated based on the average rate of change of joint angles.

[0046] In this specific implementation, short blocks are used to preserve fine-tuned pressure during the fine-grained grasping phase, while long blocks are used to capture the overall motion trajectory during continuous picking. Block length is dynamically adjusted based on the average rate of change of joint angles, ensuring that short blocks prevent distortion during rapid swings and long blocks reduce noise during slow positioning.

[0047] In the present invention, a master-slave clock timing mechanism aligns motion capture, robotic arm control, and video stream timestamps, achieving synchronization between human hand grasping movements and robotic arm responses. Arm posture data participates in the dynamic correction of the base coordinate system, eliminating spatial positioning deviations and enabling the robotic arm to reproduce movements. Normalization balances scale differences in multi-source data and suppresses the risk of joint mutations. Timing block optimization takes into account both fine grasping and long-trajectory movements, improving the authenticity of movement restoration. Forward reasoning, data acquisition, and playback verification are conveniently switched between the three modes, sharing a spatiotemporal alignment data stream. The behavioral cloning model outputs adaptive hierarchical control of joint angle sequences, autonomously switching between high-speed motion and precision operation.

[0048] Figure 4 The schematic diagram of a bionic manipulator control device for synchronizing multi-source heterogeneous data provided in the embodiments of this specification may include: An acquisition module 10 is used to acquire the hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by a motion capture device; a decoding module 20 for decoding the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmitting the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to the robotic arm controller via a low-latency network; The simulation module 30 is used to perform dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data and the decoded arm posture data through the robotic arm controller to realize simulated control of the robotic arm.

[0049] Optionally, the generation of the synchronization timestamp includes: using a computer host as the master clock, a robotic arm controller and a motion capture device as slave clocks, the master clock timing the slave clock via Ethernet, and the time error between the master clock and the slave time satisfies a preset error value.

[0050] Optionally, before the simulation module 30, the following steps are included: Determining whether the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data meet a preset deviation value; When the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data do not meet the preset deviation value, normalization processing is performed on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data.

[0051] Optionally, performing normalization processing on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data includes: in, is the normalized hand joint angle data or the normalized wrist posture data or the normalized arm posture data, is the decoded hand joint angle data or the decoded wrist posture data or the decoded arm posture data, and All are historical training set statistics.

[0052] Optionally, the robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data, including: Mapping the decoded human hand joint angle data into target joint angle data of the dexterous hand; Converting the decoded arm posture data into a correction matrix of the robot arm base coordinate system; The robot arm kinematic model is updated based on the correction matrix, and the decoded wrist posture data is converted into the target posture data of the robot arm end in the correction base coordinate system; the target posture data is solved by inverse kinematics to obtain the target angle of the robot arm joint.

[0053] Optionally, the realization of simulated manipulation of the robotic arm includes: The simulation control mode of the robotic arm includes forward reasoning mode, data acquisition mode, and playback verification mode; When the simulation control mode of the robotic arm is the forward reasoning mode, the master control node is started to coordinate the control flow, and the behavior clone model is loaded to generate a prediction sequence of the robotic arm joint angles; When the simulation control mode of the robotic arm is the data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events; When the simulation control mode of the robotic arm is the playback verification mode, the playback engine is started to drive the robotic arm to reproduce the historical action sequence.

[0054] Optionally, when the simulation control mode of the robotic arm is a data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events, further includes: Identify the current action type; When the current action type is a fine grasping action, a short block length of the first preset frame is used to retain the finger micro-manipulation details; When the current motion type is a continuous trajectory motion, a long block length of the second preset frame is used to capture the complete motion trend; The block length is dynamically calibrated based on the average rate of change of joint angles.

[0055] The functions of the device in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not fully described in this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0056] The functions of the device in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not fully described in this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0057] Based on the same inventive concept, an embodiment of this specification also provides an electronic device.

[0058] The following describes an electronic device embodiment of the present invention, which can be considered a specific physical implementation of the method and apparatus embodiments of the present invention described above. Details described in the electronic device embodiment of the present invention should be considered supplementary to the above-mentioned method or apparatus embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above-mentioned method or apparatus embodiments.

[0059] Figure 5 This is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Figure 5 The electronic device 300 according to this embodiment of the present invention will be described. Figure 5 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0060] like Figure 5 As shown, electronic device 300 is implemented as a general-purpose computing device. Components of electronic device 300 may include, but are not limited to, at least one processing unit 310, at least one storage unit 320, a bus 330 connecting various system components (including storage unit 320 and processing unit 310), and a display unit 340.

[0061] The storage unit stores program codes that can be executed by the processing unit 310, so that the processing unit 310 performs the steps according to various exemplary embodiments of the present invention described in the above processing method section of this specification. For example, the processing unit 310 can perform the following steps: Figure 1Steps shown.

[0062] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit 3202 , and may further include a read-only memory unit (ROM) 3203 .

[0063] The storage unit 320 may also include a program / utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0064] Bus 330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0065] The electronic device 300 may also communicate with one or more external devices 400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable viewers to interact with the electronic device 300, and / or any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 350. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 360. The network adapter 360 may communicate with other modules of the electronic device 300 through the bus 330. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0066] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium is enabled to implement the above method of the present invention, that is: Figure 1 The method shown.

[0067] Figure 6 A schematic diagram of a computer-readable medium provided in accordance with an embodiment of this specification.

[0068] accomplish Figure 1 The computer program of the illustrated method can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0069] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0070] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the viewer computing device, partially on the viewer device, as a stand-alone software package, partially on the viewer computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the viewer computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0071] In summary, the present invention can be implemented in hardware, or as a software module running on one or more processors, or a combination thereof. Those skilled in the art will appreciate that, in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program or computer program product) for performing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium or in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0072] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0073] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0074] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A bionic control method for a robotic arm with synchronous multi-source heterogeneous data, characterized in that: include: Obtaining the hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by the motion capture device; Decoding the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmitting the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to the robotic arm controller via a low-latency network; The robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data and the decoded arm posture data to achieve simulated control of the robotic arm.

2. The bionic control method for a robotic arm with multi-source heterogeneous data synchronization according to claim 1, characterized in that: The generation of the synchronous timestamp includes: using a computer host as a master clock, a robotic arm controller and a motion capture device as slave clocks, the master clock timing the slave clocks via Ethernet, and the time error between the master clock and the slave time meets a preset error value.

3. The bionic control method for a robotic arm with multi-source heterogeneous data synchronization according to claim 1, characterized in that: The robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to realize the simulated control of the robotic arm, including: Determining whether the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data meet a preset deviation value; When the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data do not meet the preset deviation value, normalization processing is performed on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data.

4. The method for bionic manipulation of a robotic arm with multi-source heterogeneous data synchronization according to claim 3, characterized in that: The performing normalization processing on the decoded hand joint angle data and / or the decoded wrist posture data and / or the decoded arm posture data comprises: in, is the normalized hand joint angle data or the normalized wrist posture data or the normalized arm posture data, is the decoded hand joint angle data or the decoded wrist posture data or the decoded arm posture data, and All are historical training set statistics.

5. The method for bionic manipulation of a robotic arm with multi-source heterogeneous data synchronization according to claim 4, characterized in that: The robotic arm controller performs dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data, including: Mapping the decoded human hand joint angle data into target joint angle data of the dexterous hand; Converting the decoded arm posture data into a correction matrix of the robot arm base coordinate system; The robot arm kinematic model is updated based on the correction matrix, and the decoded wrist posture data is converted into the target posture data of the robot arm end in the correction base coordinate system; the target posture data is solved by inverse kinematics to obtain the target angle of the robot arm joint.

6. The method for bionic manipulation of a robotic arm with multi-source heterogeneous data synchronization according to claim 5, characterized in that: The method of realizing the simulation control of the robotic arm includes: The simulation control mode of the robotic arm includes forward reasoning mode, data acquisition mode, and playback verification mode; When the simulation control mode of the robotic arm is the forward reasoning mode, the master control node is started to coordinate the control flow, and the behavior clone model is loaded to generate a prediction sequence of the robotic arm joint angles; When the simulation control mode of the robotic arm is the data acquisition mode, calling a remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling a keyboard listener to capture operation mark events; When the simulation control mode of the robotic arm is the playback verification mode, the playback engine is started to drive the robotic arm to reproduce the historical action sequence.

7. The method for bionic manipulation of a robotic arm with multi-source heterogeneous data synchronization according to claim 6, characterized in that: When the simulation control mode of the robotic arm is the data acquisition mode, calling the remapping process to convert the decoded human hand joint angle data, the decoded wrist posture data, and the decoded arm posture data into robotic arm posture instructions, and enabling the keyboard listener to capture operation mark events, further comprising: Identify the current action type; When the current action type is a fine grasping action, a short block length of the first preset frame is used to retain the finger micro-manipulation details; When the current motion type is a continuous trajectory motion, a long block length of the second preset frame is used to capture the complete motion trend; The block length is dynamically calibrated based on the average rate of change of joint angles.

8. A bionic manipulator control device with multi-source heterogeneous data synchronization, characterized in that: include: An acquisition module is used to acquire the hand joint angle data and its synchronization timestamp, wrist posture data and its synchronization timestamp, and arm posture data and its synchronization timestamp collected in real time by the motion capture device; a decoding module, configured to decode the hand joint angle data and its synchronization timestamp, the wrist posture data and its synchronization timestamp, and the arm posture data and its synchronization timestamp, and transmit the decoded hand joint angle data, the decoded wrist posture data, and the decoded arm posture data to a robotic arm controller via a low-latency network; The simulation module is used to perform dynamic remapping based on the decoded human hand joint angle data, the decoded wrist posture data and the decoded arm posture data through the robotic arm controller to realize simulated control of the robotic arm.

9. An electronic device, wherein: The electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images