Robot control method, computer device, and computer-readable storage medium

Abstract

This application provides a robot control method, a computer device, and a computer-readable storage medium. The method includes: acquiring end-effector posture data of a user's arms, estimating the posture data of each joint of the user, wherein the end-effector posture data includes a first timestamp and the joint posture data includes a second timestamp; mapping the joint posture data to each joint of the robot based on the joint posture data and robot configuration data, and determining reference angle information of each joint of the robot including the second timestamp and a reference angle; performing data synchronization processing on the end-effector posture data and the reference angles of each joint of the robot based on the first timestamp and the second timestamp to obtain synchronized end-effector posture data and reference angles; obtaining fused posture data through data fusion processing based on the synchronized end-effector posture data and reference angles; and controlling the robot's movement based on the fused posture data to flexibly and accurately control the robot's movement.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a robot control method, a computer device, and a computer-readable storage medium. Background Technology

[0002] In the field of robot control, robot teleoperation is one of the key technologies of human-computer interaction. Operators can remotely control robots to perform complex tasks in scenarios such as industrial manufacturing, medical surgery, deep-sea exploration, and disaster relief through human-computer interaction devices.

[0003] Currently, robot teleoperation is mainly achieved through motion capture-based teleoperation technology or physical control devices. Motion capture-based teleoperation requires high-speed camera arrays, complex marker systems, and precise calibration settings, resulting in high costs and complex setups. Physical control devices cannot provide sufficient degrees of freedom to capture subtle operator movements, limiting operational flexibility and control precision. This is especially true in tasks requiring complex movements and high coordination, where they cannot accurately capture operator intentions, leading to less natural, intuitive, and precise human-robot interaction.

[0004] Therefore, existing robot teleoperation methods have certain limitations. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a robot control method, computer device, and computer-readable storage medium to solve the practical problem that the existing robot teleoperation methods are not natural, intuitive, and precise enough, and have certain limitations.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a robot control method, the method comprising:

[0008] Acquire the user's end-effector posture data and estimate the user's joint posture data; wherein the end-effector posture data includes a first timestamp and the joint posture data includes a second timestamp;

[0009] Based on the joint posture data and the preset robot configuration data, the joint posture data is mapped to each joint of the robot to determine the reference angle information of each joint of the robot. The reference angle information of each joint of the robot includes the second timestamp and the reference angle.

[0010] Based on the first timestamp and the second timestamp, the end-effector posture data of the two arms and the reference angles of each joint of the robot are synchronized to obtain synchronized end-effector posture data of the two arms and synchronized reference angles of each joint of the robot.

[0011] Based on the synchronized end-effector posture data and the reference angles of each joint of the robot after synchronization, fused posture data is obtained through data fusion processing, and the robot's movement is controlled according to the fused posture data; wherein, the fused posture data is the target angle of each joint of the robot.

[0012] As an optional implementation, acquiring the user's end-effector posture data includes:

[0013] The first image sequence, which is a robot's perspective, is acquired in real time by the first visual sensor of the virtual reality headset worn by the user, and the first image sequence is displayed on the interface of the virtual reality headset so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the robot's perspective.

[0014] The virtual reality device held by the user acquires the user's arm end-effector posture data in real time.

[0015] As an optional implementation, estimating the user's joint pose data includes:

[0016] The system acquires a second image sequence of the user collected in real time by a second visual sensor, wherein the viewpoint of the second visual sensor is the viewpoint from which the user operates the virtual reality device.

[0017] Based on the second image sequence, the user's joint pose data are estimated using a world-class human algorithm.

[0018] As an optional implementation, the step of acquiring the user's arm end-effector posture data in real time through a virtual reality device held by the user, based on the first image sequence, includes:

[0019] Image preprocessing, image segmentation, contour extraction and template matching are performed on each first image in the first image sequence to determine the feature points of the target object in each first image;

[0020] Based on the feature points, the first image sequence is combined with the virtual hand image by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and constructing and fusing the virtual hand image.

[0021] The infrared sensor in the virtual reality device acquires real-time data on the end-effector postures of the user's arms when operating the virtual reality device.

[0022] As an optional implementation, the step of mapping the joint pose data to each joint of the robot based on the joint pose data and preset robot configuration data, and determining the reference angle information of each joint of the robot, includes:

[0023] Based on the posture data of each joint, determine the rotation matrix of each joint of the user;

[0024] Based on the rotation matrix of each joint and the preset robot configuration data, the Euler angles of each joint of the robot are determined;

[0025] Based on the Euler angles, the reference angles of each joint of the robot are determined, and the reference angles and the second timestamp in the joint posture data are combined to form the reference angle information.

[0026] As an optional implementation, the step of synchronizing the end-effector posture data of the two arms and the reference angles of each joint of the robot based on the first timestamp and the second timestamp includes:

[0027] Generate a synchronization signal according to the first timestamp and the second timestamp;

[0028] The timing of aligning the end-effector posture data of the two arms with the reference angles of each joint of the robot is determined based on the synchronization signal, the first timestamp, and the second timestamp.

[0029] As an optional implementation, the step of obtaining fused posture data through data fusion processing based on the synchronized end-effector posture data and the reference angles of each joint of the synchronized robot includes:

[0030] The reference angles of each joint of the synchronized robot are used as constraint information;

[0031] Based on the constraint information, the inverse kinematics algorithm is used to perform inverse kinematics calculations on the synchronized end-effector posture data of the two arms to determine the target angles of each joint of the robot.

[0032] As an optional implementation, controlling the robot's motion based on the fused posture data includes:

[0033] Based on the fused posture data, robot control commands are generated and sent to the robot's controller, so that the controller determines the torque of each joint of the robot according to the robot control commands, and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint.

[0034] Secondly, embodiments of this application provide a robot control device, the device comprising:

[0035] The acquisition module is used to acquire the user's end-effector posture data and estimate the user's joint posture data; wherein the end-effector posture data includes a first timestamp and the joint posture data includes a second timestamp.

[0036] The determination module is used to map the posture data of each joint to each joint of the robot according to the posture data of each joint and the preset robot configuration data, and determine the reference angle information of each joint of the robot, wherein the reference angle information of each joint of the robot includes the second timestamp and the reference angle.

[0037] The synchronization module is used to perform data synchronization processing on the end-effector posture data of the two arms and the reference angles of each joint of the robot according to the first timestamp and the second timestamp, so as to obtain synchronized end-effector posture data of the two arms and synchronized reference angles of each joint of the robot.

[0038] The fusion module is used to obtain fused posture data through data fusion processing based on the synchronized end-effector posture data of the two arms and the reference angles of each joint of the robot after synchronization, and to control the robot's movement based on the fused posture data; wherein, the fused posture data is the target angle of each joint of the robot.

[0039] As an optional implementation, the acquisition module is specifically used for:

[0040] The first image sequence, which is a robot's perspective, is acquired in real time by the first visual sensor of the virtual reality headset worn by the user, and the first image sequence is displayed on the interface of the virtual reality headset so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the robot's perspective.

[0041] The virtual reality device held by the user acquires the user's arm end-effector posture data in real time.

[0042] As an optional implementation, the acquisition module is specifically used for:

[0043] The system acquires a second image sequence of the user collected in real time by a second visual sensor, wherein the viewpoint of the second visual sensor is the viewpoint from which the user operates the virtual reality device.

[0044] Based on the second image sequence, the user's joint pose data are estimated using a world-class human algorithm.

[0045] As an optional implementation, the acquisition module is specifically used for:

[0046] Image preprocessing, image segmentation, contour extraction and template matching are performed on each first image in the first image sequence to determine the feature points of the target object in each first image;

[0047] Based on the feature points, the first image sequence is combined with the virtual hand image by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and constructing and fusing the virtual hand image.

[0048] The infrared sensor in the virtual reality device acquires real-time data on the end-effector postures of the user's arms when operating the virtual reality device.

[0049] As an optional implementation, the determining module is specifically used for:

[0050] Based on the posture data of each joint, determine the rotation matrix of each joint of the user;

[0051] Based on the rotation matrix of each joint and the preset robot configuration data, the Euler angles of each joint of the robot are determined;

[0052] Based on the Euler angles, the reference angles of each joint of the robot are determined, and the reference angles and the second timestamp in the joint posture data are combined to form the reference angle information.

[0053] As an optional implementation, the synchronization module is specifically used for:

[0054] Generate a synchronization signal according to the first timestamp and the second timestamp;

[0055] The timing of aligning the end-effector posture data of the two arms with the reference angles of each joint of the robot is determined based on the synchronization signal, the first timestamp, and the second timestamp.

[0056] As an optional implementation, the fusion module is specifically used for:

[0057] The reference angles of each joint of the synchronized robot are used as constraint information;

[0058] Based on the constraint information, the inverse kinematics algorithm is used to perform inverse kinematics calculations on the synchronized end-effector posture data of the two arms to determine the target angles of each joint of the robot.

[0059] As an optional implementation, the fusion module is specifically used for:

[0060] Based on the fused posture data, robot control commands are generated and sent to the robot's controller, so that the controller determines the torque of each joint of the robot according to the robot control commands, and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint.

[0061] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot control method described in the first aspect above.

[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot control method described in the first aspect above.

[0063] The beneficial effects of this application are:

[0064] This application provides a robot control method, computer device, and computer-readable storage medium. It acquires real-time end-effector posture data of the user's arms while controlling a virtual reality controller, records a first timestamp, and accurately constructs 3D human motion using a 3D human motion capture algorithm to estimate the user's joint posture data, recording a second timestamp. Based on the joint posture data and robot configuration data, the joint posture data is mapped to the robot joints to obtain reference angle information for each joint, including the second timestamp and the reference angles of each joint. Based on the first and second timestamps, the timing of the end-effector posture data and the reference angles of each joint is aligned through data synchronization processing to obtain synchronized end-effector posture data and synchronized reference angles of each joint. Data fusion processing is then performed on the synchronized end-effector posture data and synchronized reference angles of each joint to obtain target angles for each joint. The target angles of each joint conform to the user's operating intention. The robot's movement is controlled according to these target angles, ensuring that the robot's motion state at each moment matches the user's operating intention when controlling the virtual reality controller. Immersive human-computer interaction, data synchronization, and data fusion ensure the temporal consistency and complementarity of synchronized data, thereby improving the flexibility, naturalness, and accuracy of robot control. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A schematic diagram of the architecture of the robot control system provided in an embodiment of this application;

[0067] Figure 2 A flowchart illustrating the robot control method provided in an embodiment of this application;

[0068] Figure 3 A schematic diagram of the process for acquiring user's dual-arm end-effector posture data in the robot control method provided in this application embodiment;

[0069] Figure 4 A schematic flowchart illustrating the estimation of user joint posture data in the robot control method provided in this embodiment of the application;

[0070] Figure 5 Another flowchart illustrating the process of acquiring user's dual-arm end-effector posture data for the robot control method provided in this application embodiment;

[0071] Figure 6 A flowchart illustrating the process of determining reference angle information for each joint of a robot using the robot control method provided in this embodiment of the application.

[0072] Figure 7 A schematic diagram of the data synchronization processing of the robot control method provided in the embodiments of this application;

[0073] Figure 8 A schematic diagram of the data fusion processing flow of the robot control method provided in the embodiments of this application;

[0074] Figure 9 This is a modular structure diagram of the robot control device provided in the embodiments of this application;

[0075] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0077] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0078] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0079] In the field of robot control, operators can remotely control robots to perform complex tasks in various scenarios. Currently, robot teleoperation technology suffers from high costs, complex setup, and low flexibility and control precision, resulting in human-robot interaction that is not natural, intuitive, or precise enough.

[0080] Based on the aforementioned problems, this application proposes a robot control method that integrates augmented reality technology and a high-precision human posture estimation algorithm. By synchronizing and fusing the posture data of the two arm ends and the reference angles of each joint of the robot, fused data is obtained, ensuring the consistency of the posture data and avoiding inverse kinematics singularities. The robot's movement is controlled based on the fused data, improving the flexibility, naturalness, and accuracy of robot control.

[0081] Figure 1 This is a schematic diagram of the architecture of the robot control system provided in the embodiments of this application, as shown below. Figure 1 As shown, the robot control system includes a virtual reality (VR) headset worn by the user, a VR controller held by the user, the robot, a first visual sensor mounted on the robot's head, a second visual sensor mounted in front of the user, a robot controller, and a computer. The VR headset is communicatively connected to the first visual sensor, which transmits the real-time captured robot-view image to the user's VR headset, achieving visual synchronization between the user and the robot. The computer is communicatively connected to the VR controller, the second visual sensor, and the robot controller.

[0082] VR controllers include infrared sensors ( Figure 1 (Not shown in the image), when the user operates the VR controller from the robot's perspective, the infrared sensor can capture the user's end-effector posture data in real time and transmit it to the computer. The second vision sensor can acquire images of the user operating the VR controller in real time and transmit them to the computer. The computer estimates the user's joint posture data based on the images sent by the second vision sensor, and performs data synchronization and data fusion on the end-effector posture data and the joint posture data. Based on the fused data, it generates robot control commands and sends them to the robot controller. The robot controller is used to control the movement of each joint of the robot under the instruction of the robot control commands.

[0083] Figure 2 This is a flowchart illustrating the robot control method provided in an embodiment of this application. The execution entity of this method can be... Figure 1 Computer equipment in the environment. For example... Figure 2 As shown, the method includes:

[0084] S201. Obtain the user's end-effector posture data and estimate the user's joint posture data; wherein, the end-effector posture data includes a first timestamp, and the joint posture data includes a second timestamp.

[0085] Optionally, when remotely controlling the robot, the user holds a VR controller and wears a VR headset. Wearing the VR headset allows the user to see the robot's perspective, achieving visual synchronization between the user and the robot. After human-machine visual synchronization, the user can manipulate the handheld VR controller. Augmented Reality (AR) technology combines real-world reality with information from the virtual world. AR technology merges the robot's perspective with a pre-created virtual hand image based on the VR controller, achieving a hybrid human-machine interaction. The high-precision tracking capabilities of the VR controller accurately collect the user's arm end-effector posture data when manipulating the controller. This data includes the spatial posture of the user's arms during the operation. Furthermore, strict timestamp recording is performed during the collection of the user's arm end-effector posture data, ensuring that the data also includes the first timestamp of the user's VR controller operation.

[0086] When a user operates the VR controller, external cameras or sensors monitor the user's actions in real time. A 3D human motion capture algorithm detects the user's skeletal points and estimates the posture data of each joint. In other words, the 3D human motion capture algorithm accurately constructs 3D human motion, thereby obtaining the spatial posture of each joint. Furthermore, when estimating the user's joint posture data, strict timestamp recording is performed, ensuring that the joint posture data includes a second timestamp of the user's VR controller operation.

[0087] S202. Based on the posture data of each joint and the preset robot configuration data, map the posture data of each joint to each joint of the robot, and determine the reference angle information of each joint of the robot. The reference angle information of each joint of the robot includes a second timestamp and a reference angle.

[0088] Optionally, the robot's configuration differs from the human body structure. Robot configuration data is pre-acquired, including the installation positions of each joint, which characterizes the robot's physical structure and movement. Based on the pose estimation data of each joint and the pre-acquired robot configuration data, the pose data of each joint is mapped to each joint of the robot, and the reference angle information of each joint is calculated. This reference angle information includes a second timestamp from the user's VR controller operation and the reference angles of each joint.

[0089] Specifically, the range of motion of each joint in the human body is different from that of a robot. It is necessary to solve the posture data of each joint based on the robot's configuration data so that the reference angles of each joint of the robot conform to the robot's configuration data, thereby ensuring that the robot simulates the user's actions to the greatest extent.

[0090] S203. Based on the first timestamp and the second timestamp, perform data synchronization processing on the end-effector posture data of the two arms and the reference angles of each joint of the robot to obtain the synchronized end-effector posture data of the two arms and the synchronized reference angles of each joint of the robot.

[0091] Optionally, based on the first timestamp in the end-effector posture data and the second timestamp in the reference angle information of each joint of the robot, the end-effector posture data obtained based on the VR controller and the reference angles of each joint of the robot obtained by mapping are synchronized to obtain synchronized end-effector posture data and synchronized reference angles of each joint of the robot, so that the synchronized data are consistent in time. In other words, through data synchronization processing, the user's end-effector posture data and the reference angles of each joint of the robot are kept consistent in time at all times.

[0092] S204. Based on the synchronized end-effector posture data and the reference angles of each joint of the robot after synchronization, the fused posture data is obtained through data fusion processing, and the robot motion is controlled according to the fused posture data; wherein, the fused posture data is the target angle of each joint of the robot.

[0093] Optionally, the synchronized end-effector posture data of the two arms and the reference angles of each joint of the robot after synchronization are fused to ensure the temporal consistency and complementarity of the synchronized data. The target angles of each joint of the robot are obtained after fusion, thus avoiding the inverse kinematics singularity problem.

[0094] Robot control commands are generated based on the target angles of each joint of the robot. These commands indicate the target angles of each joint, corresponding to the user's operational intent. The robot's movement is controlled by these commands, ensuring that the robot's movement at any given moment matches the user's operational intent when using the VR controller.

[0095] In this embodiment, the end-effector posture data of the user's arms as they manipulate the virtual reality controller is acquired in real time, and a first timestamp is recorded. A 3D human motion capture algorithm is then used to accurately construct 3D human motion, estimate the user's joint posture data, and record a second timestamp. Based on the joint posture data and robot configuration data, the joint posture data is mapped to the robot's joints to obtain reference angle information for each joint, including the second timestamp and the reference angles. Based on the first and second timestamps, the timing of the end-effector posture data and the reference angles of the robot's joints is synchronized through data synchronization processing to obtain synchronized end-effector posture data and synchronized reference angles of the robot's joints. Data fusion processing is then performed on the synchronized end-effector posture data and synchronized reference angles of the robot's joints with consistent timing to obtain the target angles of each robot joint. The target angles of each robot joint conform to the user's operational intent. The robot's movement is controlled according to these target angles, ensuring that the robot's motion state at each moment matches the user's operational intent when manipulating the virtual reality controller. Through immersive human-computer interaction, data synchronization, and data fusion, the temporal consistency and complementarity of the synchronized data are guaranteed, thereby improving the flexibility, naturalness, and accuracy of robot control.

[0096] The following is a detailed explanation of the process of acquiring the user's end-effector posture data.

[0097] Figure 3 A schematic diagram illustrating the process of acquiring the user's dual-arm end-effector posture data in the robot control method provided in this application embodiment is shown below. Figure 3 As shown, the step of obtaining the user's end-effector posture data in step S201 above includes:

[0098] S301. The user-worn virtual reality headset acquires a first image sequence from the robot's perspective, which is collected in real time by the first visual sensor, and displays the first image sequence on the interface of the virtual reality headset so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the robot's perspective.

[0099] Optionally, the first visual sensor mounted on the robot's head has the robot's perspective. This sensor acquires a real-time data stream from the robot's perspective, which forms a first image sequence representing the robot's field of view. This first image sequence is transmitted in real-time to the user's VR headset via a two-way communication (WebSocket) network transmission protocol, allowing the user to see the robot's first image sequence in real-time on the VR headset interface, thus achieving visual synchronization between the user and the robot.

[0100] WebSocket is a protocol for full-duplex communication over a single Transmission Control Protocol (TCP) connection. It enables real-time transmission of the first image sequence.

[0101] S302. Real-time acquisition of the user's arm end-effector posture data through a virtual reality device held by the user.

[0102] Optionally, an application is developed within the development engine. This application performs multiple image processing operations on the first image sequence, associating the virtual and real environments. A pre-created virtual hand image is visually fused with the first image sequence. Human-computer interaction functionality is also developed within the development engine, allowing users to engage in immersive interaction with virtual elements in the virtual environment based on the robot's perspective, using a handheld virtual reality device (VR controller) and a VR headset. During this immersive interaction, the VR controller controlled by the user captures the user's arm end-effector posture data in real time and records the corresponding first timestamp.

[0103] In this embodiment, a first image sequence from the robot's perspective is acquired in real time by a first visual sensor mounted on the robot's head, and transmitted in real time to the user's virtual reality headset via a two-way communication network transmission protocol, achieving human-machine visual synchronization. The first image sequence from the robot's perspective is visually fused with a pre-created virtual human hand image, allowing the user to engage in immersive interaction with virtual elements in the virtual environment based on the first image sequence. During the immersive interaction, the user's arm end-effector posture data is captured in real time by the virtual reality controller, and the corresponding first timestamp is recorded. Through visual synchronization and virtual-real interaction, the user's arm end-effector posture data is accurately acquired in real time based on the high-precision tracking characteristics of the virtual reality controller.

[0104] The following section details the process of estimating the user's joint pose data.

[0105] Figure 4 A flowchart illustrating the estimation of user joint pose data in the robot control method provided in this application embodiment is shown below. Figure 4 As shown, the step of estimating the user's joint pose data in step S201 above includes:

[0106] S401. Acquire the second image sequence of the user collected in real time by the second visual sensor, wherein the viewpoint of the second visual sensor is the viewpoint from which the user operates the virtual reality device.

[0107] Optionally, when the user operates the VR controller, a second visual sensor monitors the user's actions in real time to capture a second image sequence of the user's VR controller operation and record the corresponding second timestamp. (See reference...) Figure 1 The second visual sensor can be mounted in front of the user to clearly and accurately capture the image when the user is operating the VR controller. The second visual sensor can also be mounted on the robot's waist. There are no specific restrictions on the installation position of the second visual sensor.

[0108] S402. Based on the second image sequence, estimate the user's joint pose data using a world-class human algorithm.

[0109] Optionally, based on the second image sequence, in a dynamic scene where the user is controlling the VR controller, a World-Grounded Humans with Accurate Motion (WHAM) algorithm is used to estimate the user's body posture in real time, outputting the user's joint posture data. Specifically, the WHAM algorithm uses 2D keypoint detection, image feature extraction from the second image sequence, and global trajectory prediction of human motion to accurately and efficiently reconstruct the 3D human motion in the global coordinate system from the second image sequence, estimating the spatial posture of each joint of the user, and obtaining the user's joint posture data. The estimated joint posture data based on the second image sequence also includes the second timestamp of the user's VR controller operation, according to the second timestamp recorded in the second image sequence.

[0110] In this embodiment, a second visual sensor is used to acquire a second image sequence of the user's operation of the virtual reality controller in real time, and a second timestamp is recorded at the time of acquisition. In the dynamic scene of the user operating the virtual reality controller, based on the second image sequence, a world-class human-like algorithm with precise motion is used to accurately reconstruct the three-dimensional human motion, thereby estimating the user's joint posture data in real time. This improves the accuracy of the user's joint posture data.

[0111] The following is a detailed explanation of the process of acquiring the user's arm end-effector posture data in real time through a virtual reality device held by the user, based on the first image sequence.

[0112] Figure 5 Another flowchart illustrating the process of acquiring the user's dual-arm end-effector posture data for the robot control method provided in this application embodiment is shown below. Figure 5 As shown, step S302 above, which involves acquiring the user's arm end-effector posture data in real time using a virtual reality device held by the user based on the first image sequence, includes:

[0113] S501. Perform image preprocessing, image segmentation, contour extraction and template matching on each first image in the first image sequence to determine the feature points of the target object in each first image.

[0114] Optionally, in the application developed by the development engine, image preprocessing is performed on each first image in the first image sequence from the robot's perspective, including image denoising and image enhancement. Specifically, image denoising is performed on each first image using mean filtering, median filtering, Gaussian filtering, or bilateral filtering to reduce noise in the first image, and histogram equalization is used to enhance the contrast of the first image, thereby improving the visibility and accuracy of the first image.

[0115] After image preprocessing, a region-based segmentation algorithm is used to segment the preprocessed first image. By identifying the region of interest (ROI) in the first image, it is divided into multiple regions, and the ROI in each first image is separated from the background. The ROI can be a target object in the real-world scene from the robot's perspective. Each segmented first image is then binarized, and OpenCV functions are used for contour extraction to extract the edge information of the target object. Based on the edge information of the target object, template matching is used to identify and locate the feature points of the target object in each first image, providing accurate positioning information for virtual-real integration.

[0116] S502. Based on the feature points, the first image sequence is combined with the virtual hand image by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and constructing and fusing the virtual hand image.

[0117] Optionally, based on the feature points of the target objects in each first image, the first visual sensor is calibrated to establish a relationship between the world coordinate system, the first visual sensor coordinate system, and the planar coordinate system of the first image. Through 3D registration, the virtual scene is bound to the coordinate system of the real scene, allowing the virtual and real scenes to share the same space. Through bidirectional virtual-real mapping, target objects in the virtual scene are associated with target objects in the real scene, establishing a connection between virtual and real objects. The infrared sensor in the VR controller dynamically captures the dynamic information of the VR controller in real time, providing data support for the fusion of the virtual and real scenes. A virtual hand image is created using 3D modeling software, and the position information of the VR controller is determined based on its dynamic information. The virtual hand image is accurately placed in the corresponding position in the real scene, ensuring that the virtual hand image precisely corresponds to its position in the real world within the user's field of vision through the VR headset. The position and orientation of the virtual hand image are adjusted in real time as the user moves the VR controller.

[0118] After constructing and integrating the virtual human hand image, the virtual scene is visually presented and integrated. The transmission of light in the virtual scene is simulated in high fidelity through a global illumination model, enhancing the realism of the virtual scene in the real scene. Human-computer interaction functions are developed in the development engine, allowing users to interact immersively with virtual elements in the virtual environment through VR controllers, enhancing the interactive experience. The virtual-real fusion parameters are adjusted to ensure the accuracy, stability, and durability of the virtual scene.

[0119] S503: Real-time acquisition of the end-effector posture data of the user's arms when operating the virtual reality device using infrared sensors in the virtual reality device.

[0120] Optionally, through immersive interaction between the user and the robot, and leveraging the high-precision tracking capabilities of the infrared sensors in the VR controllers, the infrared sensors capture real-time end-effector posture data of the user's arms while operating the VR controllers, recording the corresponding first timestamp. This end-effector posture data, including the first timestamp, is then transmitted to a computer device via the TCP protocol. The TCP protocol is a connection-oriented, end-to-end communication protocol that provides reliable data transmission, ensuring the reliability and timeliness of the end-effector posture data transmission.

[0121] In this embodiment, by performing image preprocessing, image segmentation, contour extraction, and template matching on each first image in the first image sequence from the robot's perspective, the feature points of the target object in each first image are located, accurately determining the virtual-real combined positioning information. Based on the feature points of the target object in each first image, by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamically tracking the virtual reality device, and constructing and fusing a virtual human hand image, the first image sequence from the robot's perspective is combined with the virtual human hand image, allowing the user to engage in immersive interaction with virtual elements in the virtual environment through a virtual reality controller. During the immersive interaction, the infrared sensor of the virtual reality controller captures the end-effector posture data of the user's arms in real time while operating the virtual reality controller, and records the corresponding first timestamp. The end-effector posture data containing the first timestamp is transmitted through a transmission control protocol. By utilizing augmented reality technology, the first image sequence from the robot's perspective is perfectly integrated with the virtual human hand image, providing a more intuitive and immersive interactive experience, thereby improving the accuracy of the end-effector posture data.

[0122] The following section details the process of mapping the joint posture data to the robot's joints based on the joint posture data and the preset robot configuration data, thereby determining the reference angle information for each joint.

[0123] Figure 6 This is a flowchart illustrating the process of determining the reference angle information of each joint of a robot using the robot control method provided in this application embodiment. Figure 6As shown, step S202 above, which maps the joint posture data to each joint of the robot based on the joint posture data and the preset robot configuration data, and determines the reference angle information of each joint of the robot, includes:

[0124] S601. Determine the rotation matrix of each joint based on the posture data of each joint.

[0125] Optionally, based on the spatial pose of each joint in the joint pose data and the second timestamp, the rotation matrix of each user joint relative to the subjoints is calculated. The rotation matrix of each user joint may include the rotation matrix of the user's shoulder joint, the rotation matrix of the elbow, and the rotation matrix of the wrist. For example, the rotation matrix can be a 3x3 orthogonal matrix.

[0126] S602. Determine the Euler angles of each joint of the robot based on the rotation matrix of each joint and the preset robot configuration data.

[0127] Optionally, based on the rotation matrices of each joint and the robot configuration data, a target solution for Euler angles is determined. Based on the target solution for Euler angles, the second timestamp, and the rotation matrices of each joint, the Euler angles of each joint of the robot are solved. Here, Euler angles represent a rotation sequence, namely the yaw, pitch, and roll angles of each joint of the robot rotating along the X, Y, and Z axes, respectively. For example, the rotation matrices of the user's shoulder joint, elbow, and wrist can be mapped to obtain the Euler angles of the robot's seven joints.

[0128] S603. Based on Euler angles, determine the reference angles of each joint of the robot, and combine the reference angles and the second timestamp in the joint posture data into reference angle information.

[0129] Optionally, reference angles for each robot joint are calculated based on the Euler angles of each joint and the second timestamp. These reference angles are the reference rotation angles of the robot joints. The reference angles of each robot joint are combined with the second timestamp of the VR controller operated by the user to obtain the reference angle information of each robot joint, and then transmitted to the computer device via the TCP protocol.

[0130] In this embodiment, the rotation matrix of each user joint is determined based on the spatial orientation of each joint in the joint posture data and the second timestamp. Based on the rotation matrix of each user joint and the robot configuration data, a target solution for Euler angles is determined. Then, based on the target solution, the second timestamp, and the rotation matrix of each user joint, the Euler angles of each robot joint are mapped. Based on the Euler angles of each robot joint and the second timestamp, a reference angle for each robot joint is determined, and the reference angle and the second timestamp are combined into reference angle information for transmission. This achieves accurate mapping from the user's joint orientation data to the robot's joints.

[0131] The following is a detailed explanation of the process of synchronizing the end-effector posture data of the two arms and the reference angles of each joint of the robot based on the first and second timestamps.

[0132] Figure 7 This is a schematic diagram of the data synchronization processing flow of the robot control method provided in the embodiments of this application, as shown below. Figure 7 As shown, step S203 above, which involves synchronizing the end-effector posture data and the reference angles of each joint of the robot based on the first and second timestamps, includes:

[0133] S701. Generate a synchronization signal according to the first timestamp and the second timestamp.

[0134] Optionally, the computer device generates a synchronization signal based on the first timestamp in the received end-effector posture data and the second timestamp in the reference angle information of each joint of the robot. The synchronization signal is used to indicate that the data is synchronized according to the timestamp.

[0135] S702. Based on the synchronization signal, the first timestamp, and the second timestamp, align the end-effector posture data of the two arms with the reference angles of each joint of the robot.

[0136] Optionally, the computer device performs data synchronization processing based on the synchronization signal, and aligns the end-effector posture data of the two arms with the reference angles of each joint of the robot according to the time sequence of the first and second timestamps, so that the end-effector posture data of the two arms at the same moment corresponds one-to-one with the reference angles of each joint of the robot, ensuring the temporal consistency of the synchronized data.

[0137] In this embodiment, a synchronization signal is generated based on the first timestamp in the end-effector posture data and the second timestamp in the reference angle information of each robot joint. Based on the synchronization signal, the end-effector posture data and the reference angles of each robot joint are aligned according to the time sequence of the first and second timestamps, ensuring a one-to-one correspondence between the end-effector posture data and the reference angles of each robot joint at the same moment. This improves the temporal consistency of the synchronized data.

[0138] The following section details the process of obtaining fused posture data through data fusion processing based on the synchronized end-effector posture data of both arms and the reference angles of each joint of the synchronized robot.

[0139] Figure 8 This is a schematic diagram of the data fusion processing flow of the robot control method provided in the embodiments of this application, as shown below. Figure 8 As shown, step S204 above, which involves obtaining fused posture data through data fusion processing based on the synchronized end-effector posture data and the reference angles of each joint of the synchronized robot, includes:

[0140] S801, Use the reference angles of each joint of the synchronized robot as constraint information.

[0141] Optionally, the reference angles of each joint of the synchronized robot can be used as constraint information for the inverse kinematics (IK) algorithm, providing initial guesses or constraints to the IK algorithm. This can avoid the failure of inverse kinematics calculation when performing IK calculation on the end-effector posture data of the synchronized two arms.

[0142] S802. Based on the constraint information, perform inverse kinematics calculations on the end-effector posture data of the synchronized dual arms using an inverse kinematics algorithm to determine the target angles of each joint of the robot.

[0143] Optionally, when performing IK calculations on the synchronized end-effector posture data of the two arms using the IK algorithm, the reference angles of each joint of the synchronized robot are input into the IK algorithm as constraint information. Then, the IK calculations are performed on the synchronized end-effector posture data of the two arms based on the constraint information to obtain the target angles of each joint of the robot. Specifically, the IK operation determines the target angles of each joint of the robot by inversely deriving the position and direction of each skeletal joint.

[0144] By fusing the reference angles of each joint of the robot after synchronization with consistent timing, as well as the pose data of the end effectors of the synchronized arms, fused pose data, i.e., the target angles of each joint of the robot, is obtained. The temporal consistency and complementarity of the synchronized data avoid the IK singularity problem and prevent inverse kinematics failure.

[0145] In this embodiment, the reference angles of each joint of the synchronized robot are used as constraint information for the inverse kinematics algorithm, providing initial guesses or constraints. When performing inverse kinematics calculations on the synchronized end-effector posture data, the reference angles of each joint are input as constraint information into the inverse kinematics algorithm for data fusion. The target angles of each joint are then obtained through inverse kinematics calculations. This data fusion process improves the temporal consistency and complementarity of the synchronized data, avoids inverse kinematics failures, and improves the accuracy of the inverse kinematics solution.

[0146] As an optional implementation, the step of controlling the robot's motion based on the fused attitude data in step S204 above includes:

[0147] Based on the fused posture data, robot control commands are generated and sent to the robot's controller. The controller then determines the torque of each joint of the robot according to the robot control commands and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint.

[0148] Optionally, based on the fused posture data, i.e., the target angles of each joint of the robot, robot control commands are generated and sent to the robot's controller. The controller can be a proportional-derivative (PD) controller. The PD controller determines the torque of each joint by adjusting the proportional and derivative terms according to the target angles indicated by the robot control commands. Based on the torque of each joint, the PD controller quickly and stably controls each joint to move to the target angle according to the corresponding torque over time, ensuring that the robot's motion state at each moment matches the user's operating intent when using the VR controller.

[0149] In this embodiment, robot control commands are generated based on the fused posture data and sent to the robot's controller. The controller determines the torque of each joint based on the target angle indicated by the robot control commands, and then quickly and stably controls each joint to move to the target angle according to the corresponding torque over time. This improves the flexibility and naturalness of robot control.

[0150] Based on the same inventive concept, this application also provides a robot control device corresponding to the robot control method. Since the principle of the device in this application is similar to that of the robot control method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0151] Figure 9This is a modular structure diagram of the robot control device provided in the embodiments of this application, such as... Figure 9 As shown, the device includes:

[0152] The acquisition module 901 is used to acquire the user's end-effector posture data and estimate the user's joint posture data; wherein, the end-effector posture data includes a first timestamp and the joint posture data includes a second timestamp.

[0153] The determination module 902 is used to map the posture data of each joint to each joint of the robot according to the posture data of each joint and the preset robot configuration data, and determine the reference angle information of each joint of the robot. The reference angle information of each joint of the robot includes a second timestamp and a reference angle.

[0154] The synchronization module 903 is used to perform data synchronization processing on the end-effector posture data of the two arms and the reference angles of each joint of the robot according to the first timestamp and the second timestamp, so as to obtain the synchronized end-effector posture data of the two arms and the synchronized reference angles of each joint of the robot.

[0155] The fusion module 904 is used to obtain fused posture data through data fusion processing based on the synchronized end-effector posture data of the two arms and the reference angles of each joint of the robot after synchronization, and to control the robot's movement based on the fused posture data; wherein, the fused posture data is the target angle of each joint of the robot.

[0156] As an optional implementation, the acquisition module 901 is specifically used for:

[0157] The first image sequence, which is captured in real time by the first visual sensor of the robot, is obtained by the virtual reality headset worn by the user and displayed on the interface of the virtual reality headset so that the user can control the virtual reality device in his hand according to the first image sequence. The first visual sensor is the robot's perspective.

[0158] The virtual reality device held by the user acquires real-time posture data of the user's arms.

[0159] As an optional implementation, the acquisition module 901 is specifically used for:

[0160] The system acquires a second image sequence of the user in real time from a second visual sensor. The second visual sensor's perspective is the viewpoint from which the user operates the virtual reality device.

[0161] Based on the second image sequence, the user's joint pose data is estimated using world-class human algorithms.

[0162] As an optional implementation, the acquisition module 901 is specifically used for:

[0163] Image preprocessing, image segmentation, contour extraction, and template matching are performed on each first image in the first image sequence to determine the feature points of the target object in each first image.

[0164] Based on feature points, the first image sequence is combined with the virtual hand image by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamic tracking of virtual reality devices, and constructing and fusing a virtual hand image.

[0165] The infrared sensors in the virtual reality device can acquire real-time data on the posture of the user's arms as they operate the device.

[0166] As an optional implementation, the determining module 902 is specifically used for:

[0167] Based on the posture data of each joint, determine the rotation matrix of each joint for the user.

[0168] Based on the rotation matrix of each joint and the preset robot configuration data, determine the Euler angles of each joint of the robot.

[0169] Based on Euler angles, the reference angles of each joint of the robot are determined, and the reference angles and the second timestamp in the joint posture data are combined to form the reference angle information.

[0170] As an optional implementation, the synchronization module 903 is specifically used for:

[0171] A synchronization signal is generated based on the first and second timestamps.

[0172] Based on the synchronization signal, the first timestamp, and the second timestamp, align the end-effector posture data of the two arms with the reference angles of each joint of the robot.

[0173] As an optional implementation, the fusion module 904 is specifically used for:

[0174] The reference angles of each joint of the synchronized robot are used as constraint information.

[0175] Based on the constraint information, the inverse kinematics algorithm is used to perform inverse kinematics calculations on the end-effector posture data of the synchronized two arms to determine the target angles of each joint of the robot.

[0176] As an optional implementation, the fusion module 904 is specifically used for:

[0177] Based on the fused posture data, robot control commands are generated and sent to the robot's controller. The controller then determines the torque of each joint of the robot according to the robot control commands and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint.

[0178] This application also provides a computer device, such as... Figure 10 The diagram shown is a schematic representation of the structure of a computer device provided in an embodiment of this application, including: a processor 101, a memory 102, and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101 (e.g., ...). Figure 8 The device includes the acquisition module 901, determination module 902, synchronization module 903, and fusion module 904 (and their corresponding execution instructions). When the computer device is running, the processor 101 and the memory 102 communicate via the bus 103. When the machine-readable instructions are executed by the processor 101, the steps of the robot control method in the above embodiment are executed.

[0179] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot control method described above.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0182] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A robot control method, characterized in that, include: Acquire the user's end-effector posture data and estimate the user's joint posture data; wherein the end-effector posture data includes a first timestamp and the joint posture data includes a second timestamp; Based on the joint posture data and the preset robot configuration data, the joint posture data is mapped to each joint of the robot to determine the reference angle information of each joint of the robot. The reference angle information of each joint of the robot includes the second timestamp and the reference angle. Based on the first timestamp and the second timestamp, the end-effector posture data of the two arms and the reference angles of each joint of the robot are synchronized to obtain synchronized end-effector posture data of the two arms and synchronized reference angles of each joint of the robot. Based on the synchronized end-effector posture data and the reference angles of each joint of the robot after synchronization, fused posture data is obtained through data fusion processing, and the robot's movement is controlled according to the fused posture data; wherein, the fused posture data is the target angle of each joint of the robot.

2. The method according to claim 1, characterized in that, The acquisition of the user's end-effector posture data includes: The first image sequence, which is a robot's perspective, is acquired in real time by the first visual sensor of the virtual reality headset worn by the user, and the first image sequence is displayed on the interface of the virtual reality headset so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the robot's perspective. The virtual reality device held by the user acquires the user's arm end-effector posture data in real time.

3. The method according to claim 1, characterized in that, The estimation of the user's joint pose data includes: The system acquires a second image sequence of the user collected in real time by a second visual sensor, wherein the viewpoint of the second visual sensor is the viewpoint from which the user operates the virtual reality device. Based on the second image sequence, the user's joint pose data are estimated using a world-class human algorithm.

4. The method according to claim 2, characterized in that, The step of acquiring the user's arm end-effector posture data in real time using a virtual reality device held by the user, based on the first image sequence, includes: Image preprocessing, image segmentation, contour extraction and template matching are performed on each first image in the first image sequence to determine the feature points of the target object in each first image; Based on the feature points, the first image sequence is combined with the virtual hand image by calibrating the first visual sensor, 3D registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and constructing and fusing the virtual hand image. The infrared sensor in the virtual reality device acquires real-time data on the end-effector postures of the user's arms when operating the virtual reality device.

5. The method according to claim 1, characterized in that, The step of mapping the joint posture data to the robot joints based on the joint posture data and preset robot configuration data, and determining the reference angle information of the robot joints, includes: Based on the posture data of each joint, determine the rotation matrix of each joint of the user; Based on the rotation matrix of each joint and the preset robot configuration data, the Euler angles of each joint of the robot are determined; Based on the Euler angles, the reference angles of each joint of the robot are determined, and the reference angles and the second timestamp in the joint posture data are combined to form the reference angle information.

6. The method according to claim 1, characterized in that, The step of synchronizing the end-effector posture data of the two arms and the reference angles of each joint of the robot based on the first timestamp and the second timestamp includes: Generate a synchronization signal according to the first timestamp and the second timestamp; The timing of aligning the end-effector posture data of the two arms with the reference angles of each joint of the robot is determined based on the synchronization signal, the first timestamp, and the second timestamp.

7. The method according to claim 1, characterized in that, The process of obtaining fused posture data through data fusion processing based on the synchronized end-effector posture data and the reference angles of each joint of the synchronized robot includes: The reference angles of each joint of the synchronized robot are used as constraint information; Based on the constraint information, the inverse kinematics algorithm is used to perform inverse kinematics calculations on the synchronized end-effector posture data of the two arms to determine the target angles of each joint of the robot.

8. The method according to claim 1, characterized in that, The step of controlling the robot's movement based on the fused attitude data includes: Based on the fused posture data, robot control commands are generated and sent to the robot's controller, so that the controller determines the torque of each joint of the robot according to the robot control commands, and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot control method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the robot control method as described in any one of claims 1 to 8.

Images