A gesture recognition method, device and system for remote operation

Abstract

The embodiment of the present application belongs to the field of robotics technology, and relates to a posture recognition method for remote operation, including obtaining N groups of current motion feedback data of N parts of an operator, wherein N is an integer greater than or equal to 2; and performing posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator, wherein the current posture includes the current postures of M parts of the operator, and M is an integer greater than N. The present application also provides a posture recognition device, system, computer equipment and storage medium for remote operation. The present application can obtain a more accurate posture recognition result of the operator based on a smaller number of posture sensors.

Description

Technical Field

[0001] The present application relates to the field of remote operation technology, and in particular to a gesture recognition method, device and system for remote operation. Background Art

[0002] With the development of science and technology, the application of robots is becoming more and more extensive. In some complex and dangerous environments, robots need to be more flexible and have higher human-like working capabilities. For this reason, teleoperated robots have come into being. Teleoperated robots are installed with posture sensors on the operator, such as inertial sensors (Inertial Measurement Unit IMU). The operator performs the target task in another real or virtual scene. The IMU captures the operator's movements during the operation and collects the corresponding motion feedback data and sends it to the controller. The controller generates corresponding motion control instructions based on the feedback data to control the slave robot, thereby achieving the purpose of teleoperation.

[0003] Existing motion capture solutions mainly involve densely arranging inertial sensors on the human body to obtain motion feedback data related to the action. To achieve full-body motion capture, generally 16-20 IMUs are required to be distributed on the operator.

[0004] However, IMUs are usually expensive, require large amounts of data to be processed, and require high computing power, which is not conducive to real-time motion capture. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a gesture recognition method, device and system for remote operation, so as to obtain a more accurate gesture recognition result of the operator based on a smaller number of gesture sensors.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a gesture recognition method for remote operation, which adopts the following technical solution:

[0007] A gesture recognition method for remote operation, used for recognizing the gesture of an operator, comprises the following steps:

[0008] Obtaining N sets of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2; and

[0009] The operator's posture is recognized on the current motion feedback data through a preset posture recognition model to obtain the operator's current posture, wherein the current posture includes the current postures of M parts of the operator, and M is an integer greater than N.

[0010] Further, N is equal to 4; the N groups of current motion feedback data of N parts of the operator include:

[0011] The motion feedback data corresponding to the operator's left wrist;

[0012] The motion feedback data corresponding to the operator's right wrist;

[0013] Feedback data corresponding to the operator's waist;

[0014] Corresponding to the operator's head movement feedback data.

[0015] Furthermore, obtaining N groups of current motion feedback data of N parts of the operator includes:

[0016] Acquire motion feedback data at the current moment and within a preset time window before the current moment that meets a preset time interval; and / or

[0017] The posture recognition model is a Transformer model.

[0018] Furthermore, before the operator's posture recognition is performed on the current motion feedback data by using a preset posture recognition model, the method further includes:

[0019] Constructing N sets of training sample sets of motion feedback data under multiple human postures, wherein the posture representation based on the SMPL upper body model is used as the annotation of the training sample set;

[0020] Set the hyperparameters of the preset gesture recognition model;

[0021] The posture recognition model is trained based on a training sample set of motion feedback data.

[0022] Furthermore, after the step of performing the operator's posture recognition on the current motion feedback data through a preset posture recognition model, the step further includes:

[0023] Based on a human body prior model, tracking optimization is performed with the current posture as a target to obtain an optimized current posture.

[0024] Furthermore, the step of performing tracking optimization based on the human body prior model and taking the current posture as the target specifically includes:

[0025] Taking the current posture as an optimization target;

[0026] The torque of each joint and the force on the foot of the human body prior model are used as control vectors;

[0027] Define the objective function and constraint conditions of the human body prior model tracking the current posture, perform joint optimization on the objective function and the constraint conditions, and obtain the current optimal control vector;

[0028] Executing the current optimal control vector on the human body prior model to obtain a current posture of the human body prior model;

[0029] The current posture of the human body prior model is used as the optimized current posture.

[0030] Furthermore, the objective function is:

[0031]

[0032] Among them, Q and R are both weight matrices; N1 represents the prediction time domain; p(k+it) represents the human body prior model motion posture vector predicted forward to the value at time k+i based on the value at time t; pr(k+it) represents the target motion posture vector output by the neural network predicted forward to the value at time k+i based on the value at time t; t represents the current time; N2 represents the control time domain; Δu(k+it) represents the control vector increment predicted forward to the value at time k+i based on the value at time t;

[0033] The constraint formula is:

[0034]

[0035]

[0036] u_min≤u≤u_max(2.3);

[0037] Among them, formula (2.1) represents the rotation angle of each joint of the human body prior model Need to be constrained within a certain range; Formula (2.2) represents the rotational angular velocity of each joint of the human body prior model It needs to be constrained within a certain range; Formula (2.3) represents the torque of each joint and the force on the foot of the human body prior model, that is, the size of the control vector u is constrained within a certain range.

[0038] In a second aspect, the embodiment of the present application further provides a gesture recognition device for remote operation, which is used to recognize the gesture of the operator, and adopts the following technical solution:

[0039] A gesture recognition device for remote operation, comprising:

[0040] An acquisition module, used to acquire N groups of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2;

[0041] The recognition module is used to perform the operator's posture recognition on the current motion feedback data through a preset posture recognition model to obtain the operator's current posture, wherein the current posture includes the current postures of M parts of the operator, and M is an integer greater than N.

[0042] In a third aspect, the present application also provides a remote operation system, which adopts the following technical solution:

[0043] A teleoperation system, comprising: N attitude sensors, a controller and a robot, wherein N is an integer greater than or equal to 2;

[0044] The N posture sensors and the robot are respectively connected to the controller 130 for communication;

[0045] The N posture sensors are used to respectively collect N groups of current motion feedback data of N parts of the operator;

[0046] The controller is used to obtain N groups of current motion feedback data of N parts of the operator; and perform posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator, wherein the current posture includes the current postures of M parts of the operator, and M is an integer greater than N;

[0047] The robot is used to move according to the indication of the current posture.

[0048] In a fourth aspect, the present application also provides a computer device, which adopts the following technical solution:

[0049] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the gesture recognition method for remote operation as described above when executing the computer program.

[0050] In a fifth aspect, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition method for remote operation are implemented.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] In an embodiment of the present application, the motion feedback data of the joints collected by a smaller number of posture sensors are used to identify the posture of the operator through a posture recognition model. Since the posture recognition model is trained based on a large amount of motion feedback data, a more accurate posture recognition result of the operator can be obtained based on the motion feedback data of the joints collected by a smaller number of posture sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flow chart of an embodiment of a gesture recognition method for remote operation of the present application;

[0057] Figure 3 is a flow chart of another embodiment of a gesture recognition method for remote operation of the present application;

[0058] Figure 4 is a flow chart of another embodiment of a gesture recognition method for remote operation of the present application;

[0059] Figure 5 It is a structural schematic diagram of an embodiment of a gesture recognition device for remote operation of the present application;

[0060] Figure 6 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, Figure 1 is an exemplary system architecture diagram to which the present application can be applied.

[0065] The embodiment of the present application provides a teleoperation system 100, which includes: N posture sensors 110, a robot 120 and a controller 130, wherein N is an integer greater than or equal to 2.

[0066] The N posture sensors 110 and the robot 120 are respectively connected to the controller 130 in a wired or wireless communication manner.

[0067] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0068] The N posture sensors 110 are used to respectively collect N groups of current motion feedback data of N parts of the operator.

[0069] The above-mentioned attitude sensor 110 can be various existing or future developed sensors that can obtain motion feedback data directly or based on certain processing, such as IMU, image sensor, etc. For ease of understanding, the embodiment of the present application is mainly described in detail by taking the attitude sensor as IMU as an example.

[0070] Among them, IMU is an inertial measurement unit, which is used to measure motion data related to the target object, including: three-dimensional acceleration and three-dimensional rotation angle.

[0071] For example, Figure 1 As shown, in the embodiment of the present application, the IMU 110 can be installed on only four parts of the operator 200's upper body: the left wrist, the right wrist, the head, and the waist.

[0072] The robot 120 is configured to move according to the indication of the current posture generated by the controller.

[0073] The robot 120 mentioned above may refer to the entire robot, or may refer to a part of the robot based on remote control, such as the upper body of the robot or the gripper of the robot, etc.

[0074] The controller 130 is used to execute the posture recognition method for remote operation described in the embodiment of the present application to generate the current postures of M parts of the operator.

[0075] In the embodiment of the present application, N IMUs 110 continuously collect motion feedback data of the operator 200 during the operation in a real or virtual environment at a preset frequency, and send the motion feedback data to the controller 130; the controller generates the current posture at the time (t0, t1, ...) based on the N groups of current motion feedback data continuously received, and sends the posture recognition result to the robot 120; various parts of the robot 120 move according to the instructions of the corresponding posture recognition results, thereby achieving the purpose of remote operation.

[0076] The gesture recognition method for remote operation provided in the embodiment of the present invention can be applied to a computer terminal (Personal Computer, PC); an industrial control computer terminal (Industrial Personal Computer, IPC); a mobile terminal; a server; a system including a terminal and a server, and implemented through the interaction between the terminal and the server; a programmable logic controller (Programmable Logic Controller, PLC); a field programmable gate array (Field-Programmable Gate Array, FPGA); a digital signal processor (Digital Signal Processer, DSP) or a microcontroller unit (Microcontroller unit, MCU) and the like to be executed. The controller generates program instructions according to a pre-fixed program, combined with data collected by an external IMU110, etc. For specific definitions of the controller, please refer to the definition of the gesture recognition method for remote operation in the following embodiments.

[0077] Specifically, it can be applied to Figure 6In the computer device shown, the computer device can be a terminal or a server. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a gesture recognition method for remote operation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or it can be a security check, a trackball or a touch pad set on the computer device housing, or it can be an external keyboard, touch pad or mouse, etc.

[0078] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in this application.

[0079] It should be noted that the gesture recognition method for remote operation provided in the embodiment of the present application is generally executed by the controller 130 , and accordingly, the gesture recognition device for remote operation is generally provided in the controller 130 .

[0080] like Figure 2 As shown, Figure 2 The flowchart of an embodiment of the gesture recognition method for remote operation of the present application is as follows. The gesture recognition method for remote operation comprises the following steps:

[0081] Step 201: Obtain N groups of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2.

[0082] In an embodiment of the present application, the controller obtains the current motion feedback data collected and sent by the IMU. For example, the computer terminal can access the storage space corresponding to the storage address, extract the current motion feedback data previously collected and sent by the IMU from the accessed storage space, or obtain the above-mentioned motion feedback data sent by the server, etc.

[0083] It should be noted that for the above current motion feedback data, different attitude sensors can correspond to different data contents. For example, the current motion feedback data returned by each IMU at a certain time interval is obtained. The current motion feedback data includes at least two parts. One part is the acceleration [ax, ay, az] in three directions in the local coordinate system measured by the IMU, and the second part is the attitude of the IMU relative to the inertial coordinate system. This attitude can be expressed as a quaternion, such as [1.0, 0, 0, 0].

[0084] The current motion feedback data may refer to the motion feedback data at a certain moment; or the motion feedback data at the current moment and within a preset time window before the current moment that meets a preset time interval.

[0085] Exemplarily, the IMU collects motion feedback data once per second and sends it to the controller through the data transmission interface. The controller sends the received motion feedback data once per second to the corresponding storage space; taking the preset time window as 10 seconds as an example, the storage space corresponding to the storage address is accessed every 10 seconds to retrieve the motion feedback data within the preset time window, for example: first retrieve all motion feedback data within the 10th second and 1 to 9 seconds; the second retrieve motion feedback data within the 11th to 20th seconds... and so on.

[0086] It should be noted that the above N can be set to any integer greater than or equal to 2 as needed, as long as N is less than M. It belongs to the scope of protection of this application. It can be understood that generally, the more IMUs there are, the more motion feedback data there is, and the more accurate the posture recognition result is, but it also causes a large amount of data to be processed, requiring high computing power, which is not conducive to real-time motion capture. Therefore, while meeting the requirements for posture recognition accuracy, selecting a smaller number of IMUs can improve the real-time motion capture capability.

[0087] Step 202 , performing posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator; wherein the current posture includes the current postures of M parts of the operator, where M is an integer greater than N.

[0088] In this embodiment, the controller inputs N groups of current motion feedback data of N parts of the operator into a preset posture recognition model.

[0089] In one embodiment, the operator's posture can be expressed by the rotation angle of each joint of the human body. Here, we can refer to the standard human body model, such as the Skinned Multi-Person Linear SMPL human body model. In order to define the action of a human body, we need to parameterize each movable joint point of the human body. This SMPL model stipulates 24 joints for the human body. In one embodiment, the robot usually mainly performs specific operations with the upper body. We can do some cutting and remove 8 joints of the lower body, leaving 16 joints. The rotation angle of each joint is three-dimensional, expressed by a 3*3 matrix (for example: [[0,0,1], [0,1,0], [-1,0,0]]), so the output of the posture recognition model is a 16*3*3 matrix, which represents the posture of the upper body of the human body.

[0090] The preset gesture recognition model may refer to a pre-trained gesture recognition model, which may be trained based on various existing or future-developed neural network model training methods such as supervised, semi-supervised or unsupervised.

[0091] Specifically, the above-mentioned posture recognition model can be various posture recognition models that exist now or will be developed in the future, such as transformer, LSTM, RNN, etc.

[0092] In an embodiment of the present application, the motion feedback data of the joints collected by a smaller number of posture sensors are used to identify the posture of the operator through a posture recognition model. Since the posture recognition model is trained based on a large amount of motion feedback data, a more accurate posture recognition result of the operator can be obtained based on the motion feedback data of the joints collected by a smaller number of posture sensors.

[0093] Continue as Figure 1 As shown, in some optional implementations of the present embodiment, N is equal to 4, then the N groups of current motion feedback data of N parts of the operator in step S201 include: motion feedback data corresponding to the left wrist of the operator; motion feedback data corresponding to the right wrist of the operator; feedback data corresponding to the waist of the operator; and motion feedback data corresponding to the head of the operator.

[0094] Specifically, the four IMUs can be respectively set on the operator's left wrist, right wrist, head and waist, and the controller obtains four sets of motion feedback data from the left wrist, right wrist, head and waist each time.

[0095] The embodiment of the present application sets IMUs at four places, namely the left wrist, right wrist, head and waist, to collect motion feedback data of key positions that determine the operator's posture. The controller can subsequently obtain more accurate current postures of more parts of the operator based on the four sets of motion feedback data. This can achieve more accurate operator posture recognition while meeting the minimum number of IMUs.

[0096] In some optional implementations of this embodiment, the current motion feedback data in step S201 is the motion feedback data at the current moment that satisfies the preset time interval and within a preset time window before the current moment.

[0097] Since this solution uses a small number of inertial sensors, the motion feedback data within the current time window is particularly important for judging the human body posture at that moment. Combining the motion feedback data within the current time window can improve the problem of underdetermined motion under the setting of a small number of sensors and improve the prediction accuracy of the posture recognition model.

[0098] In some optional implementations of this embodiment, the above-mentioned posture recognition model adopts a transformer network model structure. Because the transformer structure using multi-layer attention has strong expressive power, and there is no long-term memory loss and other prominent problems in traditional RNN networks and LSTM networks, it can better improve the posture prediction accuracy. Further, in a preferred embodiment, the current motion feedback data is the motion feedback data at the current moment that meets the preset time interval and the preset time window before the current moment. Since the number of inertial sensors used in this solution is small, the transformer model combined with the motion feedback data at the current moment and the preset time window before the current moment is particularly important for judging the human body posture at this moment, which can further improve the problem of underdetermined motion under a small number of sensor settings and improve the prediction accuracy of the posture recognition model.

[0099] like Figure 3 As shown, Figure 3 The flowchart of another embodiment of the gesture recognition method for teleoperation of the present application is shown in FIG.

[0100] In some optional implementations of this embodiment, before step S202 performs operator posture recognition on the current motion feedback data through a preset posture recognition model, the controller may further perform the following steps:

[0101] Step S203 constructs N sets of training sample sets of motion feedback data under multiple postures of the human body, wherein posture representation based on the SMPL upper body model is used as an annotation of the training sample set.

[0102] Based on the 16 joints of the upper body model of the SMPL model, when constructing training samples, IMUs are set for the 16 joints of the operator to collect motion feedback data of the 16 joints. Among them, N groups of motion feedback data of N preset parts of the operator in each posture are used as a group of training samples, and the posture of the human body represented by the matrix of the 16 joints in each posture of the operator is used as the annotation of each group of training samples.

[0103] Step S205 sets the hyperparameters of the preset posture recognition model.

[0104] According to the posture prediction accuracy and real-time requirements that the posture recognition model needs to achieve, the hyperparameters of the neural network model are adjusted and determined. Taking the transformer model as an example, the word embedding vector dimension, the number of self-attention layers, the number of multi-head attention heads and other parameters that affect the network size and performance, as well as the number of training cycles, learning rate, data batch size and other control parameters that affect the training process are adjusted, so that the posture recognition model can ensure a certain posture prediction accuracy, and at the same time the inference time of the network model can meet the real-time requirements.

[0105] Step S207 trains the gesture recognition model based on the training sample set of the motion feedback data.

[0106] The training sample set can be input into the posture recognition model with set hyperparameters, the output results of the model can be compared with the labeled results of the training sample set, and the parameters of the model can be continuously updated according to the difference until the preset conditions are met, thereby completing the model training.

[0107] The embodiment of the present application is based on a training sample set of N groups of motion feedback data under multiple postures of the human body. Using the posture representation based on the SMPL upper body model as the annotation of the training sample set can better achieve more accurate posture recognition results of the operator with fewer IMUs.

[0108] like Figure 4 As shown, Figure 4 The flowchart of another embodiment of the gesture recognition method for teleoperation of the present application is shown in FIG.

[0109] In some optional implementations of this embodiment, after performing the operator's posture recognition on the current motion feedback data through a preset posture recognition model in step 202, the controller may further perform the following steps:

[0110] Step 204 , based on the human body prior model, tracking optimization is performed with the current posture as the target to obtain the optimized current posture.

[0111] Among them, the human body prior model is a physical model defined based on the spatial positioning of human body parts or joints to meet certain prior distribution requirements. It has certain regulations, such as the wrist cannot rotate more than a certain angle, the upper arm cannot swing faster than a certain speed, etc. These regulations are more in line with the actual human physiological conditions and can reasonably reflect the movements of the human body. In other words, the human body prior model can also obtain posture recognition results for the 16 joints of the human body, but these posture recognition results need to meet some specific prior distribution requirements.

[0112] In this embodiment, when the preset posture recognition model outputs a posture recognition result each time, the result can often only be regarded as an expected posture recognition result. The expected posture recognition result is tracked using the human body prior model, that is, the recognition result output by the human body prior model is made to keep up with the expected posture recognition result as much as possible. If the result of the posture recognition model is more in line with reality, the tracking will be smoother. If the result output by the posture recognition model is unreasonable, the tracking result will be limited to a reasonable range based on the human body prior model itself, so as to obtain a reasonable posture recognition result overall.

[0113] Further, in some optional implementations of this embodiment, in step 204, based on the human body prior model, tracking optimization is performed with the current posture as the target, and in the optimized current posture, the above controller can specifically perform the following steps:

[0114] Step 2041, taking the current posture as the optimization target.

[0115] Step 2042, taking the torque of each joint and the force on the foot of the human body prior model as the control vector.

[0116] Step 2043, define the objective function and constraint conditions of the human body prior model tracking the current posture, perform joint optimization on the objective function and the constraint conditions, and obtain the current optimal control vector.

[0117] Step 2044, executing the current optimal control vector on the human body prior model to obtain the current posture of the human body prior model.

[0118] Step 2045, taking the current posture of the human body prior model as the optimized current posture.

[0119] The embodiment of the present application takes the current posture given by the posture recognition model as the optimization target; takes the torque of each joint of the human body prior model and the force on the foot as the control vector; defines the objective function and constraint conditions for the human body prior model to track the target posture, and jointly optimizes and solves the objective function and the constraint conditions to obtain the optimal control vector at each time (t0, t1, ...); executes the optimal control vector at time t0 for the human body prior model, and repeats the above optimization tracking process at each time; and uses the posture of the human body prior model obtained in this process as the final optimized current posture at each time.

[0120] It should be noted that the objective function can be set as needed. In one embodiment, the objective function is as follows:

[0121]

[0122] Among them, Q and R are both weight matrices, N1 represents the prediction time domain, p(k+it) represents the human body prior model motion posture vector predicted forward to the value at time k+i based on the value at time t, pr(k+it) represents the target motion posture vector output by the neural network predicted forward to the value at time k+i based on the value at time t, t represents the current time, N2 represents the control time domain, Δu(k+it) represents the control vector increment predicted forward to the value at time k+i based on the value at time t.

[0123] The constraint formula is:

[0124] φmi n≤φ≤φmax (2.1);

[0125] Vφ_mi n≤Vφ≤Vφ_max (2.2);

[0126] u_mi n≤u≤u_max (2.3);

[0127] Formula (2.1) indicates that the rotation angle φ of each joint of the human body prior model must be constrained within a certain range; Formula (2.2) indicates that the rotation angular velocity Vφ of each joint of the human body prior model must be constrained within a certain range; Formula (2.3) indicates that the torque of each joint and the force on the foot of the human body prior model, that is, the size of the control vector u must be constrained within a certain range.

[0128] In the embodiment of the present application, by solving the optimization problem that satisfies the objective function (1) and the constraints (2.1, 2.2, 2.3), the optimal control vectors at each time (t0, t1, t2...) required by the human body prior model to track the target posture output by the posture recognition model are obtained, and the control vector at time t0 is executed on the human body prior model. The above optimization tracking process is repeated at each time, and the posture of the human body prior model under this process is used as the final optimized posture at each time.

[0129] The embodiment of the present application optimizes the target posture by setting the above-mentioned objective function and combining various constraints, so that the optimized current posture can keep up with the target posture recognition result output by the posture recognition model as much as possible, while satisfying the operator's human body prior model and speed and other restrictions.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0131] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0132] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a gesture recognition device for remote operation, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various controllers.

[0133] like Figure 5 As shown, the gesture recognition device 300 for remote operation described in this embodiment includes: an acquisition module 301 and a recognition module 302. Among them:

[0134] The acquisition module 301 acquires N groups of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2.

[0135] The recognition module 302 recognizes the operator's posture on the current motion feedback data through a preset posture recognition model to obtain the operator's current posture; wherein the current posture includes the current postures of M parts of the operator, where M is an integer greater than N.

[0136] In some optional implementations of this embodiment, N is equal to 4, and the N groups of current motion feedback data of N parts of the operator include: motion feedback data corresponding to the left wrist of the operator; motion feedback data corresponding to the right wrist of the operator; feedback data corresponding to the waist of the operator; and motion feedback data corresponding to the head of the operator.

[0137] In some optional implementations of this embodiment, the above-mentioned posture recognition model adopts a transformer network structure.

[0138] In some optional implementations of this embodiment, the current motion feedback data is motion feedback data at the current moment that satisfies the preset time interval and within a preset time window before the current moment.

[0139] In an embodiment of the present application, the motion feedback data of the joints collected by a smaller number of posture sensors are used to identify the posture of the operator through a posture recognition model. Since the posture recognition model is trained based on a large amount of motion feedback data, a more accurate posture recognition result of the operator can be obtained based on the motion feedback data of the joints collected by a smaller number of posture sensors.

[0140] In some optional implementations of this embodiment, the apparatus 300 further includes: a sample construction module 303, a parameter setting module 305 and a model training module 307. Among them:

[0141] The sample construction module 303 is used to construct N sets of training sample sets of motion feedback data under multiple postures of the human body, wherein the posture representation based on the SMPL upper body model is used as the annotation of the training sample set.

[0142] The parameter setting module 305 is used to set the hyperparameters of the preset posture recognition model.

[0143] The model training module 307 is used to train the gesture recognition model based on the training sample set of motion feedback data.

[0144] The embodiment of the present application is based on a training sample set of N groups of motion feedback data under multiple postures of the human body. Using the posture representation based on the SMPL upper body model as the annotation of the training sample set can better achieve more accurate posture recognition results about the operator with fewer IMUs.

[0145] In some optional implementations of this embodiment, the apparatus 300 further includes: a posture optimization module 304, which is used to perform tracking optimization based on a human body prior model and with the current posture as a target, to obtain an optimized current posture.

[0146] In this embodiment, when the preset posture recognition model outputs a posture recognition result each time, the result can only be regarded as the expected target posture recognition result due to the limitations of human body prior conditions, speed, etc. The human body prior model is used to track the expected posture recognition result output by the posture recognition model, that is, the movement of the human body model is made to keep up with the posture recognition result of the posture recognition model as much as possible. If the result of the posture recognition model is more in line with reality, the tracking will be smoother. If the result output by the posture recognition model is unreasonable, the tracking result will be limited to a reasonable range, so as to obtain a reasonable posture recognition result overall.

[0147] Further, in some optional implementations of this embodiment, the posture optimization module 304 includes a target setting submodule 3041, a vector setting submodule 3042, a vector optimization submodule 3043, a priori optimization submodule 3044 and a result determination submodule 3045. Among them:

[0148] The target setting submodule 3041 is used in step 2041 to set the current posture as the optimization target.

[0149] The vector setting submodule 3042 is used to use the torque of each joint and the force on the foot of the human body prior model as the control vector.

[0150] The vector optimization submodule 3043 is used to define the objective function and constraint conditions of the human body prior model tracking the current posture, and perform joint optimization on the objective function and the constraint conditions to obtain the current optimal control vector.

[0151] The prior optimization submodule 3044 executes the current optimal control vector on the human body prior model to obtain the current posture of the human body prior model.

[0152] The result determination submodule 3045 uses the current posture of the human body prior model as the optimized current posture.

[0153] Furthermore, in some optional implementations of this embodiment, the objective function is as follows:

[0154]

[0155] The constraint formula is:

[0156]

[0157]

[0158] u_min≤u≤u_max(2.3).

[0159] The embodiment of the present application optimizes the target posture by setting the above-mentioned objective function and combining various constraints, so that the optimized current posture can keep up with the target posture recognition result output by the posture recognition model as much as possible, while satisfying the operator's human body prior model and speed and other restrictions.

[0160] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0161] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0162] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0163] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the gesture recognition method for remote operation, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0164] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the gesture recognition method for teleoperation.

[0165] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0166] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a gesture recognition program for remote operation, and the gesture recognition program for remote operation can be executed by at least one processor so that the at least one processor performs the steps of the gesture recognition method for remote operation as described above.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0168] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A gesture recognition method for remote operation, used to recognize the gesture of an operator, characterized in that: The steps include: Obtaining N sets of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2; and Performing posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator, wherein the current posture includes the current postures of M parts of the operator, where M is an integer greater than N; and taking the current posture as an optimization target; The torque of each joint and the force of the foot of the human body prior model are used as control vectors; Define the objective function and constraint conditions of the human body prior model tracking the current posture, and perform joint optimization solution on the objective function and the constraint conditions; Executing the current optimal control vector on the human body prior model to obtain a current posture of the human body prior model; The current posture of the human body prior model is used as the optimized current posture; wherein, The objective function is: Among them, Q and R are both weight matrices; N1 represents the prediction time domain; p(k+it) represents the human body prior model motion posture vector predicted forward to the value at time k+i based on the value at time t; pr(k+it) represents the target motion posture vector output by the neural network predicted forward to the value at time k+i based on the value at time t; t represents the current time; N2 represents the control time domain; Δu(k+it) represents the control vector increment predicted forward to the value at time k+i based on the value at time t; The constraint formula is: φmin≤φ≤φmax(2.1); Vφ_min≤Vφ≤Vφ_max(2.2); u_min≤u≤u_max(2.3); Among them, formula (2.1) indicates that the rotation angle φ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.2) indicates that the rotation angular velocity Vφ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.3) indicates the torque of each joint and the force on the foot of the human body prior model, that is, the size of the control vector u is constrained within a certain range.

2. The gesture recognition method for remote operation according to claim 1, characterized in that: N is equal to 4; the N groups of current motion feedback data of N parts of the operator include: The motion feedback data corresponding to the operator's left wrist; The motion feedback data corresponding to the operator's right wrist; Feedback data corresponding to the operator's waist; Corresponding to the operator's head movement feedback data.

3. The gesture recognition method for remote operation according to claim 1 or 2, characterized in that: Obtaining N sets of current motion feedback data of N parts of the operator includes: Acquire motion feedback data at the current moment and within a preset time window before the current moment that meets a preset time interval; and / or The posture recognition model is a Transformer model.

4. The gesture recognition method for remote operation according to claim 1 or 2, characterized in that: Before the operator's posture recognition is performed on the current motion feedback data by using a preset posture recognition model, the method further includes: Constructing N sets of training sample sets of motion feedback data under multiple human postures, wherein the posture representation based on the SMPL upper body model is used as the annotation of the training sample set; Set the hyperparameters of the preset gesture recognition model; The posture recognition model is trained based on a training sample set of motion feedback data.

5. A gesture recognition device for remote operation, used to recognize the gesture of an operator, characterized in that: include: An acquisition module, used to acquire N groups of current motion feedback data of N parts of the operator, where N is an integer greater than or equal to 2; A recognition module, used to perform posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator, wherein the current posture includes the current postures of M parts of the operator, where M is an integer greater than N; A target setting submodule, used for taking the current posture as an optimization target; A vector setting submodule is used to use the torque of each joint and the force of the foot of the human body prior model as the control vector; A vector optimization submodule, used for defining the objective function and constraint conditions of the human body prior model tracking the current posture, and performing joint optimization solution on the objective function and the constraint conditions; A priori optimization submodule, used for executing the current optimal control vector on the human body priori model to obtain the current posture of the human body priori model; The result determination submodule is used to use the current posture of the human body prior model as the optimized current posture; wherein, The objective function is: Among them, Q and R are both weight matrices; N1 represents the prediction time domain; p(k+it) represents the human body prior model motion posture vector predicted forward to the value at time k+i based on the value at time t; pr(k+it) represents the target motion posture vector output by the neural network predicted forward to the value at time k+i based on the value at time t; t represents the current time; N2 represents the control time domain; Δu(k+it) represents the control vector increment predicted forward to the value at time k+i based on the value at time t; The constraint formula is: φmin≤φ≤φmax(2.1); Vφ_min≤Vφ≤Vφ_max(2.2); u_min≤u≤u_max(2.3); Among them, formula (2.1) indicates that the rotation angle φ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.2) indicates that the rotation angular velocity Vφ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.3) indicates the torque of each joint and the force on the foot of the human body prior model, that is, the size of the control vector u is constrained within a certain range.

6. A remote operation system, characterized in that: include: N posture sensors, controllers and robots, where N is an integer greater than or equal to 2; The N posture sensors and the robot are respectively connected to the controller for communication; The N posture sensors are used to respectively collect N groups of current motion feedback data of N parts of the operator; The controller is used to obtain N groups of current motion feedback data of N parts of the operator; and perform posture recognition of the operator on the current motion feedback data through a preset posture recognition model to obtain the current posture of the operator, wherein the current posture includes the current postures of M parts of the operator, and M is an integer greater than N; Taking the current posture as an optimization target; The torque of each joint and the force of the foot of the human body prior model are used as control vectors; Define the objective function and constraint conditions of the human body prior model tracking the current posture, and perform joint optimization solution on the objective function and the constraint conditions; Executing the current optimal control vector on the human body prior model to obtain a current posture of the human body prior model; The current posture of the human body prior model is used as the optimized current posture; wherein, The objective function is: Among them, Q and R are both weight matrices; N1 represents the prediction time domain; p(k+it) represents the human body prior model motion posture vector predicted forward to the value at time k+i based on the value at time t; pr(k+it) represents the target motion posture vector output by the neural network predicted forward to the value at time k+i based on the value at time t; t represents the current time; N2 represents the control time domain; Δu(k+it) represents the control vector increment predicted forward to the value at time k+i based on the value at time t; The constraint formula is: φmin≤φ≤φmax(2.1); Vφ_min≤Vφ≤Vφ_max(2.2); u_min≤u≤u_max(2.3); Among them, formula (2.1) indicates that the rotation angle φ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.2) indicates that the rotation angular velocity Vφ of each joint of the human body prior model needs to be constrained within a certain range; formula (2.3) indicates the torque of each joint of the human body prior model and the force on the foot, that is, the size of the control vector u is constrained within a certain range; The robot is used to move according to the indication of the current posture.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the gesture recognition method for remote operation as claimed in any one of claims 1 to 4 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition method for remote operation as claimed in any one of claims 1 to 4 are implemented.

Images