Robot control method, computer equipment and computer readable storage medium

Abstract

The invention provides a robot control method, computer equipment and a computer readable storage medium, and the method comprises the steps: obtaining two-arm tail end posture data of a user, and estimating each joint posture data of the user, the two-arm tail end posture data including a first timestamp, and each joint posture data including a second timestamp; according to the posture data of each joint and the robot configuration data, mapping the posture data of each joint to each joint of the robot, and determining reference angle information, including a second timestamp and a reference angle, of each joint of the robot; according to the first timestamp and the second timestamp, performing data synchronization processing on the two-arm tail end attitude data and the reference angle of each joint of the robot to obtain synchronized two-arm tail end attitude data and the reference angle; and according to the synchronized posture data of the tail ends of the two arms and the reference angle, fused posture data are obtained through data fusion processing, and the robot is controlled to move according to the fused posture data so as to flexibly and accurately control the robot to move.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular, to a robot control method, a computer device, and a computer-readable storage medium. Background Art

[0002] In the field of robot control, robot teleoperation is one of the key technologies of human-computer interaction. Operators can remotely control robots through human-computer interaction devices to perform complex tasks in scenarios such as industrial manufacturing, medical surgery, deep-sea exploration, and disaster rescue.

[0003] At present, robot teleoperation is mainly achieved through robot teleoperation technology based on motion capture equipment or physical control equipment. Robot teleoperation technology based on motion capture equipment requires high-speed camera arrays, complex marker systems and precise calibration settings, and has the disadvantages of high cost and complex settings. Physical control equipment cannot provide enough degrees of freedom to capture the operator's subtle movements, limiting the flexibility of operation and the accuracy of control capabilities. Especially in tasks that require complex movements and high coordination, it is impossible to accurately capture the operator's intentions, making human-computer interaction less natural, intuitive and precise.

[0004] Therefore, the robot teleoperation method in the prior art has certain limitations. Summary of the invention

[0005] The purpose of this application is to provide a robot control method, computer device and computer-readable storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the practical problems that the robot remote operation method in the prior art is not natural, intuitive and accurate enough and has certain limitations.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a robot control method, the method comprising:

[0008] Acquire the user's arm end posture data, and estimate the user's joint posture data; wherein the arm end posture data includes a first timestamp, and the joint posture data includes a second timestamp;

[0009] According to the joint posture data and the preset robot configuration data, the joint posture data are mapped to the joints of the robot, and reference angle information of the joints of the robot is determined, wherein the reference angle information of the joints of the robot includes the second timestamp and the reference angle;

[0010] According to the first timestamp and the second timestamp, data synchronization processing is performed on the posture data of the two-arm ends and the reference angles of the joints of the robot to obtain the posture data of the two-arm ends after synchronization and the reference angles of the joints of the robot after synchronization;

[0011] According to the synchronized double-arm end posture data and the reference angles of the robot's joints after synchronization, fused posture data is obtained through data fusion processing, and the robot movement is controlled according to the fused posture data; wherein the fused posture data is the target angle of each joint of the robot.

[0012] As an optional implementation, the obtaining of the user's arm end posture data includes:

[0013] The virtual reality helmet worn by the user acquires a first image sequence from the perspective of the robot collected in real time by a first visual sensor, and displays the first image sequence on an interface of the virtual reality helmet, so that the user can control the virtual reality device held by the user according to the first image sequence, and the perspective of the first visual sensor is the perspective of the robot;

[0014] The user's arm end posture data is obtained in real time through the virtual reality device held by the user.

[0015] As an optional implementation, estimating the posture data of each joint of the user includes:

[0016] Acquire a second image sequence of the user captured in real time by a second visual sensor, where the viewing angle of the second visual sensor is a viewing angle for monitoring the user manipulating the virtual reality device;

[0017] According to the second image sequence, each joint posture data of the user is estimated by a world-class human algorithm.

[0018] As an optional implementation, acquiring the posture data of both arms of the user in real time through a virtual reality device held by the user according to the first image sequence includes:

[0019] Performing image preprocessing, image segmentation, contour extraction, and template matching on each first image in the first image sequence to determine feature points of the target object in each first image;

[0020] According to the feature points, the first image sequence is combined with the virtual human hand image by calibrating the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamically tracking the virtual reality device, and constructing and fusing a virtual human hand image;

[0021] The infrared sensor in the virtual reality device is used to obtain the posture data of the ends of both arms of the user when operating the virtual reality device in real time.

[0022] As an optional implementation, mapping the joint posture data to the joints of the robot according to the joint posture data and the preset robot configuration data, and determining the reference angle information of the joints of the robot, includes:

[0023] Determine the rotation matrix of each joint of the user according to the posture data of each joint;

[0024] Determining the Euler angles of each joint of the robot according to the rotation matrix of each joint and the preset robot configuration data;

[0025] According to the Euler angle, a reference angle of each joint of the robot is determined, and the reference angle and a second timestamp in the joint posture data are combined into the reference angle information.

[0026] As an optional implementation, performing data synchronization processing on the posture data of the two-arm ends and the reference angles of the joints of the robot according to the first timestamp and the second timestamp includes:

[0027] Generate a synchronization signal according to the first timestamp and the second timestamp;

[0028] According to the synchronization signal, the first timestamp and the second timestamp, the time of the double-arm end posture data and the reference angle of each joint of the robot are aligned.

[0029] As an optional implementation, the fused posture data is obtained by data fusion processing according to the posture data of the double-arm ends after synchronization and the reference angles of the joints of the robot after synchronization, including:

[0030] Using the reference angles of the joints of the synchronized robot as constraint information;

[0031] According to the constraint information, an inverse kinematics calculation is performed on the synchronized double-arm end posture data through an inverse kinematics algorithm to determine the target angle of each joint of the robot.

[0032] As an optional implementation, controlling the robot movement according to the fused posture data includes:

[0033] Based on the fused posture data, a robot control instruction is generated and sent to the robot controller, so that the controller determines the torque of each joint of the robot according to the robot control instruction, and controls each joint of the robot to move to a corresponding target angle according to the torque of each joint of the robot.

[0034] In a second aspect, an embodiment of the present application provides a robot control device, the device comprising:

[0035] An acquisition module, used to acquire the user's arm end posture data and estimate the user's joint posture data; wherein the arm end posture data includes a first timestamp, and the joint posture data includes a second timestamp;

[0036] a determination module, configured to map the joint posture data to the joints of the robot according to the joint posture data and preset robot configuration data, and determine reference angle information of the joints of the robot, wherein the reference angle information of the joints of the robot includes the second timestamp and the reference angle;

[0037] A synchronization module, used for performing data synchronization processing on the double-arm end posture data and the reference angles of the joints of the robot according to the first timestamp and the second timestamp, to obtain the double-arm end posture data after synchronization and the reference angles of the joints of the robot after synchronization;

[0038] A fusion module is used to obtain fused posture data through data fusion processing based on the synchronized double-arm end posture data and the reference angles of each joint of the robot after synchronization, and control the robot movement according to the fused posture data; wherein the fused posture data is the target angle of each joint of the robot.

[0039] As an optional implementation, the acquisition module is specifically used for:

[0040] The virtual reality helmet worn by the user acquires a first image sequence from the perspective of the robot collected in real time by a first visual sensor, and displays the first image sequence on an interface of the virtual reality helmet, so that the user can control the virtual reality device held by the user according to the first image sequence, and the perspective of the first visual sensor is the perspective of the robot;

[0041] The user's arm end posture data is obtained in real time through the virtual reality device held by the user.

[0042] As an optional implementation, the acquisition module is specifically used for:

[0043] Acquire a second image sequence of the user captured in real time by a second visual sensor, where the viewing angle of the second visual sensor is a viewing angle for monitoring the user manipulating the virtual reality device;

[0044] According to the second image sequence, each joint posture data of the user is estimated by a world-class human algorithm.

[0045] As an optional implementation, the acquisition module is specifically used for:

[0046] Performing image preprocessing, image segmentation, contour extraction, and template matching on each first image in the first image sequence to determine feature points of the target object in each first image;

[0047] According to the feature points, the first image sequence is combined with the virtual human hand image by calibrating the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamically tracking the virtual reality device, and constructing and fusing a virtual human hand image;

[0048] The infrared sensor in the virtual reality device is used to obtain the posture data of the ends of both arms of the user when operating the virtual reality device in real time.

[0049] As an optional implementation manner, the determining module is specifically used for:

[0050] Determine the rotation matrix of each joint of the user according to the posture data of each joint;

[0051] Determining the Euler angles of each joint of the robot according to the rotation matrix of each joint and the preset robot configuration data;

[0052] According to the Euler angle, a reference angle of each joint of the robot is determined, and the reference angle and a second timestamp in the joint posture data are combined into the reference angle information.

[0053] As an optional implementation, the synchronization module is specifically used for:

[0054] Generate a synchronization signal according to the first timestamp and the second timestamp;

[0055] According to the synchronization signal, the first timestamp and the second timestamp, the time of the double-arm end posture data and the reference angle of each joint of the robot are aligned.

[0056] As an optional implementation, the fusion module is specifically used for:

[0057] Using the reference angles of the joints of the synchronized robot as constraint information;

[0058] According to the constraint information, an inverse kinematics calculation is performed on the synchronized double-arm end posture data through an inverse kinematics algorithm to determine the target angle of each joint of the robot.

[0059] As an optional implementation, the fusion module is specifically used for:

[0060] Based on the fused posture data, a robot control instruction is generated and sent to the robot controller, so that the controller determines the torque of each joint of the robot according to the robot control instruction, and controls each joint of the robot to move to a corresponding target angle according to the torque of each joint of the robot.

[0061] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot control method described in the first aspect above.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robot control method described in the first aspect above are executed.

[0063] The beneficial effects of this application are:

[0064] The present application provides a robot control method, a computer device and a computer-readable storage medium, which obtains the posture data of the two arms of the user manipulating the virtual reality handle in real time, records the first timestamp, and accurately constructs the three-dimensional human body motion through the three-dimensional human body motion capture algorithm, estimates the posture data of each joint of the user, and records the second timestamp. According to the posture data of each joint and the robot configuration data, the posture data of each joint is mapped to each joint of the robot, and the reference angle information of each joint of the robot is obtained, including the second timestamp and the reference angle of each joint of the robot. According to the first timestamp and the second timestamp, the posture data of the two arms and the time of the reference angle of each joint of the robot are aligned through data synchronization processing, and the posture data of the two arms and the reference angle of each joint of the robot after synchronization are obtained, and the posture data of the two arms and the reference angle of each joint of the robot after synchronization with the same time sequence are subjected to data fusion processing to obtain the target angle of each joint of the robot. The target angle of each joint of the robot meets the operation intention of the user, and the robot movement is controlled according to the target angle of each joint of the robot, so that the movement state of the robot at each moment meets the operation intention of the user when manipulating the virtual reality handle. Through immersive human-computer interaction, data synchronization and data fusion, the temporal consistency and complementarity of the synchronized data are guaranteed, thereby improving the flexibility, naturalness and precision of robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A schematic diagram of the architecture of a robot control system provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a flow chart of a robot control method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the process of obtaining the posture data of the two ends of the arms of a user in the robot control method provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of a flow chart of estimating the posture data of each joint of a user in a robot control method provided in an embodiment of the present application;

[0070] Figure 5 Another schematic diagram of a process for obtaining the posture data of the two ends of the arms of a user in the robot control method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a flow chart of determining reference angle information of each joint of a robot according to a robot control method provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of a data synchronization process of a robot control method provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of a data fusion process of a robot control method provided in an embodiment of the present application;

[0074] Fig. 9 A module structure diagram of a robot control device provided in an embodiment of the present application;

[0075] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0077] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0078] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0079] In the field of robot control, operators can remotely control robots to perform complex tasks in various scenarios. At present, robot teleoperation technology has the problems of high cost, complex setup, low flexibility and control accuracy, making human-machine interaction less natural, intuitive and precise.

[0080] Based on the above problems, the embodiment of the present application proposes a robot control method, which integrates augmented reality technology and a high-precision human posture estimation algorithm. By synchronizing and fusing the posture data of the two arms and the reference angles of each joint of the robot, fused data is obtained to ensure the consistency of the posture data and avoid the inverse kinematics singularity problem. The robot movement is controlled according to the fused data, which improves the flexibility, naturalness and accuracy of the robot control.

[0081] Figure 1 A schematic diagram of the architecture of a robot control system provided in an embodiment of the present application, such as Figure 1 As shown, the robot control system includes a virtual reality (VR) helmet worn by the user, a VR handle held by the user, a robot, a first visual sensor installed on the head of the robot, a second visual sensor set up in front of the user, a robot controller, and a computer device. Among them, the VR helmet is connected to the first visual sensor for communication, and the first visual sensor transmits the real-time collected images from the robot's perspective to the VR helmet worn by the user in real time, so as to achieve visual synchronization between the user and the robot. The computer device is connected to the VR handle, the second visual sensor, and the robot controller for communication respectively.

[0082] The VR controller includes an infrared sensor ( Figure 1 (not shown in the figure), when the user manipulates the VR handle from the perspective of the robot, the infrared sensor can capture the user's arm end posture data in real time and transmit it to the computer device. The second visual sensor can capture the image of the user manipulating the VR handle in real time and transmit it to the computer device. The computer device estimates the user's joint posture data based on the image sent by the second visual sensor, and synchronizes and fuses the arm end posture data and the joint posture data, generates a robot control instruction based on the fused data and sends it to the robot controller. The robot controller is used to control the movement of each joint of the robot under the instruction of the robot control instruction.

[0083] Figure 2 This is a flow chart of a robot control method provided in an embodiment of the present application. The execution subject of the method may be Figure 1 Computer equipment in. Figure 2 As shown, the method includes:

[0084] S201. Acquire the posture data of both arms of the user, and estimate the posture data of each joint of the user; wherein the posture data of both arms includes a first timestamp, and the posture data of each joint includes a second timestamp.

[0085] Optionally, when the user remotely controls the robot, he holds a VR handle and wears a VR helmet. When the user wears the VR helmet, he can see the image from the robot's perspective, thus achieving visual synchronization between the user and the robot. After human-machine visual synchronization, the user can manipulate the handheld VR handle. Augmented Reality (AR) technology can combine the real world with the information of the virtual world. Through AR technology, the image from the robot's perspective is integrated with the virtual human hand image pre-created based on the VR handle to achieve virtual-real human-machine interaction. Based on the high-precision tracking characteristics of the VR handle, the posture data of the ends of both arms when the user manipulates the VR handle is accurately collected. The posture data of the ends of both arms include the spatial posture of the ends of both arms when the user manipulates the VR handle. Among them, when collecting the posture data of the ends of both arms of the user, strict timestamp recording is performed so that the posture data of the ends of both arms also includes the first timestamp of the user manipulating the VR handle.

[0086] When the user manipulates the VR handle, the user's operation is monitored in real time through an external camera or sensor, and the user's body bone points are detected through a three-dimensional human motion capture algorithm, and the user's joint posture data is estimated. In other words, the three-dimensional human motion is accurately constructed through the three-dimensional human motion capture algorithm to obtain the spatial posture of each joint of the user. Among them, when estimating the user's joint posture data, strict timestamp recording is also performed, so that each joint posture data includes the second timestamp of the user manipulating the VR handle.

[0087] S202. Mapping each joint posture data to each joint of the robot according to each joint posture data and preset robot configuration data, and determining reference angle information of each joint of the robot, wherein the reference angle information of each joint of the robot includes a second timestamp and a reference angle.

[0088] Optionally, the configuration of the robot is different from the human body structure, and the robot configuration data is acquired in advance. The robot configuration data includes the installation position of each joint point of the robot, which can characterize the physical structure and movement mode of the robot. Based on the posture data of each joint obtained by posture estimation and the robot configuration data acquired in advance, the posture data of each joint is mapped to each joint of the robot, and the reference angle information of each joint of the robot is solved. Among them, the reference angle information includes the second timestamp of the user manipulating the VR handle and the reference angle of each joint of the robot.

[0089] Specifically, the range of motion of each joint of the human body is different from that of the robot. It is necessary to solve the posture data of each joint according to the configuration data of the robot so that the reference angle of each joint of the robot conforms to the configuration data of the robot, thereby ensuring that the robot simulates the user's movements to the greatest extent.

[0090] S203. Perform data synchronization processing on the posture data of both arms and the reference angles of the joints of the robot according to the first timestamp and the second timestamp to obtain the posture data of both arms and the reference angles of the joints of the robot after synchronization.

[0091] Optionally, according to the first timestamp in the posture data of the two-arm ends and the second timestamp in the reference angle information of each joint of the robot, the posture data of the two-arm ends acquired based on the VR handle and the reference angles of each joint of the robot obtained by mapping are synchronized to obtain the synchronized posture data of the two-arm ends and the reference angles of each joint of the robot after synchronization, so that the synchronized data remain consistent in time. That is to say, through data synchronization processing, the posture data of the user's two-arm ends and the reference angles of each joint of the robot remain consistent in time at each moment.

[0092] S204. According to the posture data of the two arms ends after synchronization and the reference angles of the joints of the robot after synchronization, the fused posture data are obtained through data fusion processing, and the robot movement is controlled according to the fused posture data; wherein the fused posture data is the target angle of each joint of the robot.

[0093] Optionally, data fusion processing is performed on the time-consistent synchronized double-arm end posture data and the reference angles of each joint of the robot after synchronization to ensure the time consistency and complementarity of the synchronized data. After fusion, the target angles of each joint of the robot are obtained, avoiding the singularity problem of inverse kinematics.

[0094] The robot control instructions are generated according to the target angles of each joint of the robot. The robot control instructions are used to indicate the target angles of each joint of the robot, corresponding to the user's operation intention. The robot movement is controlled by the robot control instructions so that the robot's movement state at each moment is consistent with the user's operation intention when operating the VR handle.

[0095] In this embodiment, the posture data of the two arms of the user manipulating the virtual reality handle is obtained in real time, the first timestamp is recorded, and the three-dimensional human body motion is accurately constructed through the three-dimensional human body motion capture algorithm, the posture data of each joint of the user is estimated, and the second timestamp is recorded. According to the posture data of each joint and the robot configuration data, the posture data of each joint is mapped to each joint of the robot, and the reference angle information of each joint of the robot is obtained, including the second timestamp and the reference angle of each joint of the robot. According to the first timestamp and the second timestamp, the posture data of the two arms and the time of the reference angle of each joint of the robot are aligned through data synchronization processing, and the posture data of the two arms and the reference angle of each joint of the robot after synchronization are obtained, and the posture data of the two arms and the reference angle of each joint of the robot after synchronization with consistent timing are processed by data fusion to obtain the target angle of each joint of the robot. The target angle of each joint of the robot meets the user's operation intention, and the robot movement is controlled according to the target angle of each joint of the robot, so that the movement state of the robot at each moment meets the user's operation intention when manipulating the virtual reality handle. Through immersive human-computer interaction, data synchronization and data fusion, the temporal consistency and complementarity of the synchronized data are guaranteed, thereby improving the flexibility, naturalness and accuracy of robot control.

[0096] The following is a detailed description of the process of obtaining the user's arm end posture data.

[0097] Figure 3 A schematic diagram of the process of obtaining the user's double-arm end posture data of the robot control method provided in the embodiment of the present application, such as Figure 3 As shown, the step of obtaining the user's arm end posture data in the above step S201 includes:

[0098] S301. The virtual reality helmet worn by the user obtains a first image sequence from the perspective of the robot collected in real time by a first visual sensor, and displays the first image sequence on the interface of the virtual reality helmet, so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the perspective of the robot.

[0099] Optionally, the perspective of the first visual sensor installed on the robot head is the perspective of the robot, and the first visual sensor collects a data stream from the perspective of the robot in real time. The data stream is a first image sequence from the perspective of the robot, and the first image sequence is a picture from the perspective of the robot, which is used to represent the robot's field of view. The first image sequence from the perspective of the robot collected in real time is transmitted to the VR helmet worn by the user in real time through a network transmission protocol of two-way communication (WebSocket), so that the user can see the first image sequence from the perspective of the robot in real time in the interface of the VR helmet, thereby achieving visual synchronization between the user and the robot.

[0100] WebSocket is a protocol for full-duplex communication on a single Transmission Control Protocol (TCP) connection, and the first image sequence can be transmitted in real time through WebSocket.

[0101] S302: Acquire the user's arm end posture data in real time through the virtual reality device held by the user.

[0102] Optionally, an application is developed in the development engine, and multiple image processing operations are performed on the first image sequence through the application, the virtual environment and the real environment are associated, the pre-created virtual human hand image is visually integrated with the first image sequence, and a human-computer interaction function is developed in the development engine, so that the user can immersively interact with the virtual elements in the virtual environment based on the first image sequence from the robot's perspective through a handheld virtual reality device, i.e., a VR handle, and a VR helmet worn by the user. During the immersive interaction process, the user's arm end posture data is captured in real time through the VR handle controlled by the user, and the corresponding first timestamp is recorded.

[0103] In this embodiment, a first image sequence from the perspective of the robot is collected in real time by a first visual sensor installed on the head of the robot, and is transmitted in real time to a virtual reality helmet worn by the user through a two-way communication network transmission protocol, thereby achieving human-machine visual synchronization. The first image sequence from the perspective of the robot is visually fused with a pre-created virtual human hand image, so that the user can interact immersively with virtual elements in the virtual environment based on the first image sequence. During the immersive interaction process, the user's arm end posture data is captured in real time through the virtual reality handle controlled by the user, and the corresponding first timestamp is recorded. Through visual synchronization and virtual-reality interaction, the user's arm end posture data is accurately collected in real time based on the high-precision tracking characteristics of the virtual reality handle.

[0104] The following describes in detail the process of estimating the posture data of each joint of the user.

[0105] Figure 4 A schematic diagram of a flow chart of estimating the user's joint posture data of the robot control method provided in an embodiment of the present application, such as Figure 4 As shown, the step of estimating the posture data of each joint of the user in the above step S201 includes:

[0106] S401. Acquire a second image sequence of the user collected in real time by a second visual sensor, where the viewing angle of the second visual sensor is a viewing angle for monitoring the user manipulating the virtual reality device.

[0107] Optionally, when the user manipulates the VR handle, the user's operation is monitored in real time by a second visual sensor, so as to collect a second image sequence of the user manipulating the VR handle in real time and record a corresponding second timestamp. Figure 1 The second visual sensor can be set up in front of the user to clearly and accurately capture the image when the user manipulates the VR handle. The second visual sensor can also be installed on the waist of the robot. There is no specific restriction on the installation position of the second visual sensor.

[0108] S402: Estimate the posture data of each joint of the user through a world-class human algorithm according to the second image sequence.

[0109] Optionally, based on the second image sequence, in a dynamic scene where the user manipulates the VR handle, a world-class human algorithm with accurate motion (World-grounded Humans with Accurate Motion, referred to as WHAM) is used to estimate the user's body posture in real time, and output the user's joint posture data. Specifically, the WHAM algorithm is used to accurately and efficiently reconstruct the three-dimensional human motion in the global coordinate system from the second image sequence through two-dimensional key point detection, image feature extraction of the second image sequence, and global trajectory prediction of human motion, and the spatial posture of each joint of the user is estimated to obtain the user's joint posture data. Among them, according to the second timestamp recorded in the second image sequence, the user's joint posture data estimated based on the second image sequence also includes the second timestamp of the user manipulating the VR handle.

[0110] In this embodiment, the second image sequence of the user manipulating the virtual reality handle is collected in real time through the second visual sensor, and the second timestamp of the collection is recorded. In the dynamic scene where the user manipulates the virtual reality handle, based on the second image sequence, a world-class human algorithm with precise motion is used to accurately reconstruct the three-dimensional human motion, so as to accurately estimate the user's joint posture data in real time. The accuracy of the user's joint posture data is improved.

[0111] The following describes in detail the process of acquiring the user's arm end posture data in real time through the user's handheld virtual reality device according to the first image sequence.

[0112] Figure 5 Another flowchart of the robot control method for obtaining the user's double-arm end posture data provided in the embodiment of the present application is as follows: Figure 5 As shown, the step of acquiring the user's arm end posture data in real time through the virtual reality device held by the user according to the first image sequence in the above step S302 includes:

[0113] S501 , performing image preprocessing, image segmentation, contour extraction, and template matching on each first image in a first image sequence to determine feature points of a target object in each first image.

[0114] Optionally, in the application developed by the development engine, image preprocessing is performed on each first image in the first image sequence from the robot's perspective, including image denoising and image enhancement. Specifically, image denoising is performed on each first image by using mean filtering, median filtering, Gaussian filtering or bilateral filtering to reduce noise in the first image, and contrast of the first image is enhanced by using histogram equalization to improve visibility and accuracy of the first image.

[0115] After image preprocessing, the preprocessed first image is segmented using a region-based segmentation algorithm. The first image is divided into multiple regions by identifying the region of interest in the first image, and the region of interest in each first image is separated from the background. The region of interest may be a target object in a real scene from the robot's perspective. Each segmented first image is binarized, and the OpenCV function is called to extract the contour to extract the edge information of the target object. Based on the edge information of the target object, the feature points of the target object in each first image are identified and located through template matching processing, providing accurate positioning information for the combination of virtual and real.

[0116] S502, combining the first image sequence with the virtual human hand image through calibration of the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and construction and fusion of the virtual human hand image according to the feature points.

[0117] Optionally, the first visual sensor is calibrated according to the feature points of the target object in each first image, and the relationship between the world coordinate system, the first visual sensor coordinate system and the plane coordinate system of the first image is established. The virtual scene is bound to the coordinate system of the real scene through three-dimensional registration, so that the virtual scene and the real scene share the same space. Through virtual-real bidirectional mapping processing, the target object in the virtual scene is associated with the target object in the real scene, and the connection between the virtual object and the real object is established. The infrared sensor in the VR handle is used to dynamically capture the dynamic information of the VR handle controlled by the user in real time according to the VR handle, providing data support for the fusion of the virtual scene and the real scene. A virtual human hand image is created using three-dimensional modeling software, and the position information of the VR handle is determined according to the dynamic information of the VR handle, and the virtual human hand image is accurately placed in the corresponding position in the real scene, ensuring that the virtual human hand image corresponds accurately to the position of the real world in the field of view seen by the user through the VR helmet, and the position and direction of the virtual human hand image are adjusted in real time when the user moves the VR handle.

[0118] After constructing and integrating the virtual human hand image, the virtual scene is visually presented and integrated. The global illumination model is used to simulate the transmission of light in the virtual scene with high fidelity, enhancing the real presence of the virtual scene in the real scene. Human-computer interaction functions are developed in the development engine. Users can use VR controllers to interact immersively with virtual elements in the virtual environment to enhance the interactive experience, and adjust the virtual-reality fusion parameters to ensure the accuracy, stability and durability of the virtual scene.

[0119] S503: Obtain, through the infrared sensor in the virtual reality device, the posture data of the ends of both arms of the user when operating the virtual reality device in real time.

[0120] Optionally, through immersive interaction between the user and the robot, based on the high-precision tracking characteristics of the infrared sensor in the VR handle, the infrared sensor is used to capture the posture data of the two arms when the user manipulates the VR handle in real time, and the corresponding first timestamp is recorded, and the posture data of the two arms including the first timestamp is transmitted to the computer device through the TCP protocol. The TCP protocol is a connection-oriented end-to-end communication protocol that provides reliable data transmission and ensures the reliability and timeliness of the transmission of the posture data of the two arms.

[0121] In this embodiment, by performing image preprocessing, image segmentation, contour extraction and template matching on each first image in the first image sequence from the robot's perspective, the feature points of the target object in each first image are located, and the positioning information of the virtual-real combination is accurately determined. According to the feature points of the target object in each first image, the first image sequence from the robot's perspective is combined with the virtual human hand image by calibrating the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamic tracking virtual reality equipment, and building and fusing the virtual human hand image, so that the user can interact immersively with the virtual elements in the virtual environment through the virtual reality handle. During the immersive interaction process, the infrared sensor of the virtual reality handle captures the posture data of the end of the arms when the user manipulates the virtual reality handle in real time, and records the corresponding first timestamp, and transmits the posture data of the end of the arms containing the first timestamp through the transmission control protocol. Using augmented reality technology, the first image sequence from the robot's perspective is perfectly integrated with the virtual human hand image, providing a more intuitive and immersive interactive experience, thereby improving the accuracy of the posture data of the end of the arms.

[0122] The following describes in detail the process of mapping each joint posture data to each joint of the robot according to each joint posture data and preset robot configuration data, and determining the reference angle information of each joint of the robot.

[0123] Figure 6 A schematic diagram of a flow chart of determining reference angle information of each joint of a robot according to a robot control method provided in an embodiment of the present application, such as Figure 6As shown, in the above step S202, according to the joint posture data and the preset robot configuration data, the steps of mapping the joint posture data to the joints of the robot and determining the reference angle information of the joints of the robot include:

[0124] S601. Determine the rotation matrix of each joint of the user according to the posture data of each joint.

[0125] Optionally, according to the spatial posture of each joint in each joint posture data and the second timestamp, the rotation matrix of each joint of the user relative to the secondary joint is solved, wherein the rotation matrix of each joint of the user may include the rotation matrix of the user's shoulder joint, the rotation matrix of the elbow and the rotation matrix of the wrist. Exemplarily, the rotation matrix may be a 3*3 orthogonal matrix.

[0126] S602: Determine the Euler angle of each joint of the robot according to the rotation matrix of each joint and preset robot configuration data.

[0127] Optionally, according to the rotation matrix of each joint and the robot configuration data, the target solution of the Euler angle is determined, and the Euler angle of each joint of the robot is solved based on the target solution of the Euler angle, the second timestamp and the rotation matrix of each joint. Among them, the Euler angle represents a rotation sequence, that is, the yaw angle, pitch angle and roll angle of each joint of the robot rotated along the X-axis, Y-axis and Z-axis respectively. Exemplarily, the rotation matrix of the user's shoulder joint, the rotation matrix of the elbow and the rotation matrix of the wrist can be mapped to obtain the Euler angles of the seven joints of the robot.

[0128] S603: Determine the reference angle of each joint of the robot according to the Euler angle, and combine the reference angle and the second timestamp in the joint posture data into reference angle information.

[0129] Optionally, the reference angle of each joint of the robot is calculated based on the Euler angle of each joint of the robot and the second timestamp, and the reference angle is the reference rotation angle of the robot joint. The reference angle of each joint of the robot and the second timestamp of the user manipulating the VR handle are combined to obtain the reference angle information of each joint of the robot, and transmitted to the computer device through the TCP protocol.

[0130] In this embodiment, the rotation matrix of each joint of the user is determined according to the spatial posture of each joint in each joint posture data and the second timestamp. According to the rotation matrix of each joint of the user and the robot configuration data, the target solution of the Euler angle is determined, and then based on the target solution, the second timestamp and the rotation matrix of each joint of the user, the Euler angle of each joint of the robot is mapped. According to the Euler angle of each joint of the robot and the second timestamp, the reference angle of each joint of the robot is determined, and the reference angle of each joint of the robot and the second timestamp are combined into reference angle information for transmission. Accurate mapping of each joint posture data of the user to each joint of the robot is achieved.

[0131] The following describes in detail the process of performing data synchronization processing on the posture data of both ends of the arms and the reference angles of the joints of the robot according to the first timestamp and the second timestamp.

[0132] Figure 7 A schematic diagram of the data synchronization process of the robot control method provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, the step of performing data synchronization processing on the posture data of the two-arm ends and the reference angles of the joints of the robot according to the first timestamp and the second timestamp in the above step S203 includes:

[0133] S701. Generate a synchronization signal according to a first timestamp and a second timestamp.

[0134] Optionally, the computer device generates a synchronization signal based on a first timestamp in the received double-arm end posture data and a second timestamp in the reference angle information of each joint of the robot, and the synchronization signal is used to indicate time synchronization of the data according to the timestamp.

[0135] S702: Align the time of the posture data of both arms and the reference angles of the joints of the robot according to the synchronization signal, the first timestamp and the second timestamp.

[0136] Optionally, the computer device performs data synchronization processing based on the synchronization signal, and aligns the time of the double-arm end posture data with the reference angle of each joint of the robot according to the time sequence of the first timestamp and the second timestamp, so that the double-arm end posture data at the same moment corresponds one-to-one with the reference angle of each joint of the robot, thereby ensuring the timing consistency of the synchronized data.

[0137] In this embodiment, a synchronization signal is generated according to the first timestamp in the posture data of the two-arm end and the second timestamp in the reference angle information of each joint of the robot. Based on the synchronization signal, the time of the posture data of the two-arm end and the reference angle of each joint of the robot are aligned according to the time sequence of the first timestamp and the second timestamp, so that the posture data of the two-arm end at the same time corresponds to the reference angle of each joint of the robot one by one. The time sequence consistency of the synchronized data is improved.

[0138] The following describes in detail the process of obtaining the fused posture data through data fusion processing based on the posture data of the double-arm ends after synchronization and the reference angles of each joint of the robot after synchronization.

[0139] Figure 8 A schematic diagram of a data fusion process for a robot control method according to an embodiment of the present application is shown in FIG. Figure 8 As shown, the step of obtaining the fused posture data through data fusion processing according to the posture data of the end points of the two arms after synchronization and the reference angles of each joint of the robot after synchronization in the above step S204 includes:

[0140] S801. Use the reference angles of the joints of the synchronized robot as constraint information.

[0141] Optionally, the reference angles of the robot joints after synchronization are used as constraint information of an inverse kinematics (IK) algorithm to provide an initial guess or constraint condition to the IK algorithm, thereby avoiding inverse solution failure when performing IK calculations on the synchronized double-arm end posture data.

[0142] S802: Based on the constraint information, perform inverse kinematics calculation on the posture data of the two-arm ends after synchronization through an inverse kinematics algorithm to determine the target angle of each joint of the robot.

[0143] Optionally, when performing IK calculation on the double-arm end posture data after synchronization through the IK algorithm, the reference angle of each joint of the robot after synchronization is input into the IK algorithm as constraint information, and then the IK calculation is performed on the double-arm end posture data after synchronization according to the constraint information to obtain the target angle of each joint of the robot. Specifically, the IK operation determines the target angle of each joint of the robot by reversely deducing the position and direction of each skeletal joint.

[0144] By fusing the reference angles of each joint of the synchronized robot and the posture data of the two arms after synchronization, the fused posture data, i.e. the target angles of each joint of the robot, is obtained. The timing consistency and complementarity of the synchronized data can avoid IK singularity problems and prevent inverse solution failure.

[0145] In this embodiment, the reference angle of each joint of the robot after synchronization is used as the constraint information of the inverse kinematics algorithm, which can provide an initial guess or constraint condition for the inverse kinematics algorithm. When performing inverse kinematics calculation on the posture data of the two-arm end after synchronization, the reference angle of each joint of the robot after synchronization is input as constraint information into the inverse kinematics algorithm for data fusion, and the target angle of each joint of the robot is obtained by inverse kinematics calculation. Through data fusion processing, the temporal consistency and complementarity of the synchronized data are improved, the failure of the inverse solution is avoided, and the accuracy of the inverse solution is improved.

[0146] As an optional implementation, the step of controlling the robot movement according to the fused posture data in the above step S204 includes:

[0147] Based on the fused posture data, a robot control instruction is generated and sent to the robot controller, so that the controller determines the torque of each joint of the robot according to the robot control instruction, and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint of the robot.

[0148] Optionally, a robot control instruction is generated based on the fused posture data, i.e., the target angle of each joint of the robot, and the robot control instruction is sent to the controller of the robot. The controller may be a proportional-derivative (PD) controller. The PD controller determines the torque of each joint of the robot by adjusting the proportional term and the differential term according to the target angle of each joint of the robot indicated by the robot control instruction, and controls each joint of the robot to move to the target angle according to the corresponding torque in a time series according to the torque of each joint of the robot, so that the motion state of the robot at each moment meets the user's operation intention when manipulating the VR handle.

[0149] In this embodiment, a robot control instruction is generated based on the fused posture data, and the robot control instruction is sent to the robot controller. The controller determines the torque of each joint of the robot according to the target angle of each joint of the robot indicated by the robot control instruction, and controls each joint of the robot to move to the target angle in a time series according to the torque of each joint of the robot quickly and stably according to the corresponding torque. The flexibility and naturalness of robot control are improved.

[0150] Based on the same inventive concept, a robot control device corresponding to the robot control method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned robot control method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0151] Fig. 9The module structure diagram of the robot control device provided in the embodiment of the present application is as follows: Fig. 9 As shown, the device comprises:

[0152] The acquisition module 901 is used to acquire the user's arm end posture data and estimate the user's joint posture data; wherein the arm end posture data includes a first timestamp, and the joint posture data includes a second timestamp.

[0153] The determination module 902 is used to map each joint posture data to each joint of the robot according to each joint posture data and preset robot configuration data, and determine the reference angle information of each joint of the robot, wherein the reference angle information of each joint of the robot includes a second timestamp and a reference angle.

[0154] The synchronization module 903 is used to perform data synchronization processing on the double-arm end posture data and the reference angles of each joint of the robot according to the first timestamp and the second timestamp to obtain the double-arm end posture data and the reference angles of each joint of the robot after synchronization.

[0155] The fusion module 904 is used to obtain fused posture data through data fusion processing based on the posture data of the two arms ends after synchronization and the reference angles of each joint of the robot after synchronization, and control the robot movement according to the fused posture data; wherein the fused posture data is the target angle of each joint of the robot.

[0156] As an optional implementation, the acquisition module 901 is specifically used for:

[0157] The virtual reality helmet worn by the user obtains a first image sequence from the robot's perspective collected in real time by a first visual sensor, and displays the first image sequence on the interface of the virtual reality helmet, so that the user can control the virtual reality device held by the user according to the first image sequence. The perspective of the first visual sensor is the perspective of the robot.

[0158] The user's arm end posture data is obtained in real time through the user's handheld virtual reality device.

[0159] As an optional implementation, the acquisition module 901 is specifically used for:

[0160] A second image sequence of the user collected in real time by a second visual sensor is obtained, where the viewing angle of the second visual sensor is a viewing angle for monitoring the user manipulating the virtual reality device.

[0161] According to the second image sequence, the posture data of each joint of the user is estimated by a world-class human algorithm.

[0162] As an optional implementation, the acquisition module 901 is specifically used for:

[0163] Perform image preprocessing, image segmentation, contour extraction, and template matching on each first image in the first image sequence to determine feature points of the target object in each first image.

[0164] According to the feature points, the first image sequence is combined with the virtual human hand image through calibration of the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamic tracking of the virtual reality device, and construction and fusion of the virtual human hand image.

[0165] Through the infrared sensor in the virtual reality device, the posture data of the end of both arms of the user when operating the virtual reality device is obtained in real time.

[0166] As an optional implementation, the determination module 902 is specifically configured to:

[0167] According to the posture data of each joint, the rotation matrix of each joint of the user is determined.

[0168] According to the rotation matrix of each joint and the preset robot configuration data, the Euler angle of each joint of the robot is determined.

[0169] According to the Euler angle, the reference angle of each joint of the robot is determined, and the reference angle and the second timestamp in the joint posture data are combined into reference angle information.

[0170] As an optional implementation manner, the synchronization module 903 is specifically used for:

[0171] A synchronization signal is generated according to the first timestamp and the second timestamp.

[0172] According to the synchronization signal, the first timestamp and the second timestamp, the time of the double-arm end posture data and the reference angle of each joint of the robot are aligned.

[0173] As an optional implementation, the fusion module 904 is specifically used for:

[0174] The reference angles of each joint of the synchronized robot are used as constraint information.

[0175] According to the constraint information, the inverse kinematics algorithm is used to perform inverse kinematics calculation on the posture data of the two arms after synchronization to determine the target angle of each joint of the robot.

[0176] As an optional implementation, the fusion module 904 is specifically used for:

[0177] Based on the fused posture data, a robot control instruction is generated and sent to the robot controller, so that the controller determines the torque of each joint of the robot according to the robot control instruction, and controls each joint of the robot to move to the corresponding target angle according to the torque of each joint of the robot.

[0178] The present application also provides a computer device, such as Fig.10 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, including: a processor 101, a memory 102 and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101 (for example, Figure 8 In the device, the execution instructions corresponding to the acquisition module 901, the determination module 902, the synchronization module 903 and the fusion module 904 are obtained, etc.), when the computer device is running, the processor 101 and the memory 102 communicate through the bus 103, and when the machine-readable instructions are executed by the processor 101, the steps of the robot control method in the above embodiment are executed.

[0179] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robot control method in the above embodiment are executed.

[0180] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0181] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0182] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A robot control method, characterized in that: include: Acquire the user's arm end posture data, and estimate the user's joint posture data; wherein the arm end posture data includes a first timestamp, and the joint posture data includes a second timestamp; According to the joint posture data and the preset robot configuration data, the joint posture data are mapped to the joints of the robot, and reference angle information of the joints of the robot is determined, wherein the reference angle information of the joints of the robot includes the second timestamp and the reference angle; According to the first timestamp and the second timestamp, data synchronization processing is performed on the posture data of the two-arm ends and the reference angles of the joints of the robot to obtain the posture data of the two-arm ends after synchronization and the reference angles of the joints of the robot after synchronization; According to the synchronized double-arm end posture data and the reference angles of the robot's joints after synchronization, fused posture data is obtained through data fusion processing, and the robot movement is controlled according to the fused posture data; wherein the fused posture data is the target angle of each joint of the robot.

2. The method according to claim 1, characterized in that The step of obtaining the user's arm end posture data includes: The virtual reality helmet worn by the user acquires a first image sequence from the perspective of the robot collected in real time by a first visual sensor, and displays the first image sequence on an interface of the virtual reality helmet, so that the user can control the virtual reality device held by the user according to the first image sequence, and the perspective of the first visual sensor is the perspective of the robot; The user's arm end posture data is obtained in real time through the virtual reality device held by the user.

3. The method according to claim 1, characterized in that The estimating the posture data of each joint of the user includes: Acquire a second image sequence of the user captured in real time by a second visual sensor, where the viewing angle of the second visual sensor is a viewing angle for monitoring the user manipulating the virtual reality device; According to the second image sequence, each joint posture data of the user is estimated by a world-class human algorithm.

4. The method according to claim 2, characterized in that: The step of acquiring the posture data of both arms of the user in real time through a virtual reality device held by the user according to the first image sequence includes: Performing image preprocessing, image segmentation, contour extraction, and template matching on each first image in the first image sequence to determine feature points of the target object in each first image; According to the feature points, the first image sequence is combined with the virtual human hand image by calibrating the first visual sensor, three-dimensional registration, virtual-real bidirectional mapping, dynamically tracking the virtual reality device, and constructing and fusing a virtual human hand image; The infrared sensor in the virtual reality device is used to obtain the posture data of the ends of both arms of the user when operating the virtual reality device in real time.

5. The method according to claim 1, characterized in that The step of mapping the joint posture data to the joints of the robot according to the joint posture data and the preset robot configuration data, and determining the reference angle information of the joints of the robot comprises: Determine the rotation matrix of each joint of the user according to the posture data of each joint; Determining the Euler angles of each joint of the robot according to the rotation matrix of each joint and the preset robot configuration data; According to the Euler angle, a reference angle of each joint of the robot is determined, and the reference angle and a second timestamp in the joint posture data are combined into the reference angle information.

6. The method according to claim 1, characterized in that The step of performing data synchronization processing on the posture data of the two-arm ends and the reference angles of the joints of the robot according to the first timestamp and the second timestamp includes: Generate a synchronization signal according to the first timestamp and the second timestamp; According to the synchronization signal, the first timestamp and the second timestamp, the time of the double-arm end posture data and the reference angle of each joint of the robot are aligned.

7. The method according to claim 1, characterized in that The method of obtaining fused posture data by data fusion processing according to the posture data of the double-arm ends after synchronization and the reference angles of the joints of the robot after synchronization includes: Using the reference angles of the joints of the synchronized robot as constraint information; According to the constraint information, an inverse kinematics calculation is performed on the synchronized double-arm end posture data through an inverse kinematics algorithm to determine the target angle of each joint of the robot.

8. The method according to claim 1, characterized in that The controlling the robot movement according to the fused posture data comprises: Based on the fused posture data, a robot control instruction is generated and sent to the robot controller, so that the controller determines the torque of each joint of the robot according to the robot control instruction, and controls each joint of the robot to move to a corresponding target angle according to the torque of each joint of the robot.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the robot control method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the robot control method according to any one of claims 1 to 8 are executed.

Images