Motion estimation method and device based on multi-modal perception, and electronic device

Abstract

The present disclosure discloses a motion estimation method and device based on multi-modal perception and electronic equipment, relates to the field of embodied intelligence technology, and particularly relates to the field of motion estimation based on multi-modal perception. The specific implementation scheme is: obtaining motion evaluation related data of a target part region of a target object includes IMU sensor data and flexible deformation sensor data, the pose information of the part where the IMU sensor is arranged can be determined based on the IMU sensor data, then the pose information of the part where the flexible deformation sensor is arranged is calculated according to the position relationship between the arranged IMU sensor and the flexible deformation sensor and the obtained flexible deformation sensor data, so that the pose information of the position where the IMU sensor is not arranged is obtained in the case that the IMU sensor is arranged.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of embodied intelligence, and particularly to the field of motion estimation based on multi-modal perception. BACKGROUND

[0002] In the field of embodied intelligence, humanoid robots, etc., pose estimation of a target object is usually required. Currently, pose estimation is mainly performed through an inertial sensor (IMU, Inertial Measurement Unit) and visual recognition. The principle of using an inertial sensor is as follows: an inertial motion capture system mainly consists of several measurement nodes, each of which has a nine-axis inertial measurement unit that returns measurement information of angular velocity, acceleration, and a magnetometer during movement; then, a quaternion is obtained through an algorithm, and pose recognition (solving) is performed to obtain pose information of the target object.

[0003] However, in a humanoid robot and other devices, especially in a small space such as a hand, too many IMU sensors cannot be configured, otherwise the limb movement of the relevant part will be affected. Therefore, how to obtain the pose information of the part without an IMU sensor in the case of limited IMU sensor configuration has become a problem. SUMMARY

[0004] The present disclosure provides a motion estimation method and device based on multi-modal perception and an electronic device, which are used to obtain the pose information of a part without an IMU sensor in the case of limited IMU sensor configuration.

[0005] In a first aspect, the present disclosure provides a motion estimation method based on multi-modal perception, which comprises:

[0006] obtaining motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data;

[0007] determining motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, wherein the motion information comprises pose information.

[0008] Optionally, an IMU sensor is configured at a key part of each target part region of the target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; and the determining of the motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object comprises:

[0009] determine first pose information of the key part of the any target part region of the target object based on the IMU sensor data of the key part of the any target part region of the target object collected;

[0010] determine second pose information of the first adjacent part corresponding to the key part of the any target part region of the target object based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the any target part region of the target object collected and the first pose information of the key part of the any target part region of the target object;

[0011] determine pose information of the any target part region of the target object based on the first pose information and the second pose information.

[0012] Optionally, a flexible pressure sensor is arranged at the second adjacent part corresponding to the key part of the any target part region of the target object; and the determination of the second pose information of the first adjacent part corresponding to the key part of the any target part region of the target object based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the any target part region of the target object collected and the first pose information of the key part of the any target part region of the target object includes:

[0013] determine a confidence deformation variable of the first adjacent part corresponding to the key part of the any target part region of the target object relative to the key part based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the any target part region of the target object collected and the flexible pressure sensor data collected by the flexible pressure sensor arranged at the second adjacent part corresponding to the key part of the any target part region of the target object;

[0014] determine the second pose information of the first adjacent part corresponding to the key part of the any target part region of the target object based on the confidence deformation variable and the first pose information of the key part of the any target part region of the target object.

[0015] Optionally, the key part of the any target part region of the target object is the back of the hand, the key part of the any target part region of the target object is the back of the hand, the first adjacent part corresponding to the key part of the any target part region of the target object includes the back of the hand, and the second adjacent part corresponding to the key part of the any target part region of the target object includes the palm of the hand.

[0016] Optionally, an EMG sensor is arranged at a third adjacent part corresponding to the key part of the any target part region of the target object; and the motion information further includes motion intensity information determined based on EMG sensor data collected by the EMG sensor arranged at the third adjacent part corresponding to the key part of the any target part region of the target object.

[0017] Optionally, the method further includes:

[0018] Based on the posture information and the motion intensity information of the target object at the target part region, the next action of the target object at the target part is determined.

[0019] Optionally, the method further comprises:

[0020] determining posture information and motion intensity information of the target object at multiple target regions;

[0021] Based on the posture information and the motion intensity information of the target object at the target part region, the next action of the target object at the target part is determined.

[0022] In a second aspect, the present disclosure provides a multi-modal perception-based motion estimation device, which comprises:

[0023] An acquisition module is configured to acquire motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data.

[0024] A first determination module is configured to determine motion information of at least one target part region of a target object based on motion evaluation related data of the at least one target part region of the target object; the motion information comprises posture information.

[0025] Optionally, IMU sensors are configured at key parts of each target part region of a target object, and flexible deformation sensors are configured at first adjacent parts corresponding to the key parts; the first determination module comprises:

[0026] A first determination unit is configured to determine first posture information of a key part of any target part region of a target object based on IMU sensor data of the key part of the target object.

[0027] A second determination unit is configured to determine second posture information of a first adjacent part corresponding to a key part of any target part region of a target object based on flexible deformation sensor data of the first adjacent part corresponding to the key part of the target object and the first posture information of the key part of the target object.

[0028] A third determination unit is configured to determine posture information of a target part region of a target object based on the first posture information and the second posture information.

[0029] Optionally, the second adjacent part corresponding to the key part of the any target part region of the target object is configured with a flexible pressure sensor; the second determination unit is specifically configured to determine a confidence deformation variable of the first adjacent part corresponding to the key part of the any target part region of the target object relative to the key part based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the any target part region of the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of the any target part region of the target object; and determine second posture information of the first adjacent part corresponding to the key part of the any target part region based on the confidence deformation variable and the first posture information of the key part of the any target part region.

[0030] Optionally, the key part of the any target part region is the back of the hand, the back of the hand nail part, the first adjacent part corresponding to the key part of the any target part region includes the back of the hand finger part, and the second adjacent part corresponding to the key part of the any target part region includes the palm of the hand finger part.

[0031] Optionally, the third adjacent part corresponding to the key part of the any target part region is configured with an EMG sensor; and the motion information further includes motion intensity information; the motion intensity information is determined based on the EMG sensor data collected by the EMG sensor configured at the third adjacent part corresponding to the key part of the any target part region.

[0032] Optionally, the apparatus further includes:

[0033] The first formulation module is configured to formulate a next action of the any target part of the target object based on the posture information of the any target part region of the target object and the motion intensity information.

[0034] Optionally, the apparatus further includes:

[0035] The second determination module is configured to determine the posture information and the motion intensity information of the multiple target regions of the target object.

[0036] The second formulation module is configured to formulate a next action of the any target part of the target object based on the determined posture information and the motion intensity information of the multiple target regions of the target object.

[0037] According to a third aspect of the present disclosure, an electronic device is provided, which includes at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method shown in the first aspect of the present disclosure.

[0038] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method according to the first aspect of the present disclosure.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to the first aspect of the present disclosure.

[0040] The present disclosure provides a multi-modal perception based motion estimation method, comprising: obtaining motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data; determining motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, the motion information comprising attitude information. That is, the motion evaluation related data of the target part region of the target object comprises IMU sensor data and flexible deformation sensor data, the pose information of the part where the IMU sensor is configured can be determined based on the IMU sensor data, then the pose of the part where the flexible deformation sensor is configured is obtained according to the position relationship between the configured IMU sensor and the flexible deformation sensor, and the flexible deformation sensor data, so as to realize the case that the pose information of the position where the IMU sensor is not configured is obtained in the case that the IMU sensor is configured.

[0041] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0043] Figure 1 is a flowchart of the multi-modal perception based motion estimation method provided by the embodiments of the present disclosure;

[0044] Figure 2 is a structural schematic diagram of the multi-modal perception based motion estimation device provided by the embodiments of the present disclosure;

[0045] Figure 3 is a block diagram of an electronic device for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0046] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Thus, it should be apparent to those skilled in the art that various modifications and changes can be made in the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, it should be apparent to those skilled in the art that the disclosed subject matter is not limited to what is described herein, but can be practiced with a wide range of modifications and variations, as will be apparent to those skilled in the art.

[0047] In the following description, descriptions of well-known functions and structures are omitted.

[0048] Embodiment One

[0049] Figure 1 A multi-modal perception-based motion estimation method provided by an embodiment of the present disclosure is shown, which includes:

[0050] In step S101, motion evaluation related data of at least one target part region of a target object is obtained; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data.

[0051] As described above, the IMU sensor has the disadvantage of zero bias error, and in the application scenario of part-body intelligent robots (target objects), the operation precision requirement of the target object is relatively high (such as the hand and fingers), which leads to the fact that too many IMU sensors cannot be configured, otherwise the flexibility of the operation of the hand and fingers will be affected.

[0052] Specifically, the IMU sensor data and the flexible deformation sensor data of the target part region can be obtained through the limited IMU sensors and flexible deformation sensors configured in the target part region of the target object. The target object can be a humanoid robot, a real person performing a simulation experiment, or a robot or device capable of applying the method of the present disclosure. The target part region can be a corresponding part of the target object. Taking a humanoid robot as an example, the target part region can be a hand region composed of the back of the hand, the back of the hand and the fingernail, the fingertip, the palm of the hand and the palm of the hand, such as a head region composed of the head, the neck, and the like.

[0053] Among them, the traditional strain sensor based on metal and semiconductor is rigid and cannot measure flexible and stretchable objects. In general, flexible materials have the characteristics of being bendable and deformable. The flexible deformation (strain) sensor has the characteristics of being bendable and deformable, and can measure the degree of deformation of an object.

[0054] The IMU sensor data can be data processed by the flexible deformation sensor. In the application process, the flexible deformation sensor data collected by the configured flexible deformation sensor can be compared with the flexible deformation sensor data corresponding to each preset standard action in the pre-recorded preset standard action library. If there is consistent data, it is determined that the current action of the target object target part is a preset standard action. Then, the current IMU sensor data is calibrated based on the pre-recorded IMU sensor parameter information corresponding to the preset standard action. Specifically, the pre-recorded IMU sensor parameter information corresponding to the preset standard action and the current IMU sensor data can be calibrated when the error exceeds a predetermined range.

[0055] In step S102, the motion information of at least one target part region of the target object is determined based on the motion evaluation related data of the at least one target part region of the target object. The motion information includes attitude information.

[0056] Specifically, the motion information of the corresponding target part region of the target object can be determined based on the obtained motion evaluation related data of the target part region of the target object. Specifically, the pose information of the part where the IMU sensor is configured can be determined based on the IMU sensor data. Then, the pose of the part where the flexible deformation sensor is configured is obtained by solving the position relationship between the configured IMU sensor and the flexible deformation sensor and the flexible deformation sensor data, so that the pose information of the position where the IMU sensor is not configured is obtained in the case that the IMU sensor is configured. In addition, the pose information of the region where the IMU sensor and the flexible sensor are not configured can be obtained based on the position relationship between the region where the IMU sensor and the flexible sensor are configured and the region where the IMU sensor and the flexible sensor are not configured. The motion information of the target object can also be determined by a pre-trained neural network model.

[0057] Specifically, the motion evaluation related data of multiple target part regions of the target object can also be obtained, and the motion information of each target part region can be determined respectively. In addition, since the target regions of the target object have a correlation relationship, the motion information of a certain target part region can also be determined based on the motion evaluation related data of multiple target part regions with a correlation relationship. For example, a neural network model can be trained by using multiple positive and negative sample data (motion evaluation related information of multiple target part regions) and the motion information label of a certain target part region. Then, the motion information of a certain target part region can be determined by using the pre-trained neural network model and the motion evaluation related information of multiple target part regions obtained in the application process.

[0058] The motion information includes pose information, wherein the pose represents a position and an attitude of the target part.

[0059] The motion estimation method based on multi-modal perception provided by the present disclosure comprises: acquiring motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data; determining motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, wherein the motion information comprises attitude information. That is, the motion evaluation related data of the target part region of the target object comprises IMU sensor data and flexible deformation sensor data, the pose information of the part where the IMU sensor is arranged can be determined based on the IMU sensor data, then the pose of the part where the flexible deformation sensor is arranged is obtained according to the positional relationship between the arranged IMU sensor and the flexible deformation sensor and the flexible deformation sensor data, so that the pose information of the position where the IMU sensor is not arranged is obtained in the case that the IMU sensor is arranged in a limited manner.

[0060] The present disclosure provides a possible implementation, wherein the IMU sensor is arranged at a key part of each target part region of the target object, and the flexible deformation sensor is arranged at a first adjacent part corresponding to the key part; and the step S102 of determining the motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object comprises:

[0061] The step S1021 (not shown in the figure) comprises: determining the first attitude information of the key part of any target part region of the target object based on the IMU sensor data of the key part of the target part region.

[0062] Specifically, the IMU sensor data is arranged at the key part of the target part region of the target object, wherein the key part can be a part where the IMU sensor is arranged and does not affect the flexibility of the target object. Through the IMU sensor of the key part of the target part region of the target object, the pose information of the key part can be obtained through a corresponding algorithm.

[0063] The step S1022 (not shown in the figure) comprises: determining the second attitude information of the first adjacent part corresponding to the key part of any target part region of the target object based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the target part region of the target object and the first attitude information of the key part of the target part region.

[0064] Specifically, based on the positional relationship between the key part of the any target part region and the corresponding first adjacent part, the confidence deformation variable of the first adjacent part relative to the key part of the any target part region can be obtained according to the collected flexible deformation sensor data of the corresponding first adjacent part, and the second attitude information of the first adjacent part can be calculated according to the first attitude information of the key part of the any target part region.

[0065] In step S1023 (not shown in the figure), the attitude information of the any target part region part of the target object is determined based on the first attitude information and the second attitude information.

[0066] Specifically, the attitude information of the any target part region part includes the key part of the any target part region and the attitude information of the adjacent part of the key part. The attitude information of the any target part region part of the target object can be determined according to the first attitude information and the second attitude information.

[0067] The embodiments of the present disclosure solve the problem of how to determine the attitude information of the target part region according to the flexible deformation sensor data and the IMU sensor data.

[0068] The present disclosure provides a possible implementation, and the second adjacent part corresponding to the key part of the any target part region of the target object is configured with a flexible pressure sensor. The second attitude information of the first adjacent part corresponding to the key part of the any target part region is determined based on the first adjacent part corresponding to the key part of the any target part region of the target object, the first attitude information of the key part of the any target part region, and the flexible deformation sensor data collected by the flexible pressure sensor, and the first attitude information of the key part of the any target part region, including:

[0069] Based on the flexible deformation sensor data collected by the first adjacent part corresponding to the key part of the any target part region of the target object, and the flexible pressure sensor data collected by the flexible pressure sensor configured in the second adjacent part corresponding to the key part of the any target part region of the target object, the confidence deformation variable of the first adjacent part corresponding to the key part of the any target part region relative to the key part is determined.

[0070] Specifically, the confidence deformation variable of the first adjacent part corresponding to the key part of the any target part region relative to the key part can be determined based on the positional relationship between the first adjacent part configured with the flexible deformation sensor and the second adjacent part configured with the flexible pressure sensor, and the flexible deformation sensor data of the first adjacent part and the flexible pressure sensor data of the second adjacent part collected.

[0071] determine second pose information of the first adjacent part corresponding to the key part of the any target part region based on the confidence deformation variable and the first pose information of the key part of the any target part region.

[0072] Specifically, the first pose information of the key part of the any target part region is converted according to the confidence deformation variable, and second pose information of the first adjacent part is obtained.

[0073] In the embodiment of the present disclosure, the flexible pressure sensor data is used to compensate the flexible deformation sensor data, so that the confidence deformation variable obtained is more accurate, and the determined pose information is more accurate.

[0074] The present disclosure provides a possible implementation, the key part of the any target part region is the back of the hand, the back of the hand nail part, the first adjacent part corresponding to the key part of the any target part region includes the back of the hand finger part, and the second adjacent part corresponding to the key part of the any target part region includes the palm of the hand finger part.

[0075] Specifically, since the hand region is limited, too many IMU sensors will affect the flexibility of hand operation, in addition, the single deformation sensor and the pressure sensor cannot reflect the change of the target 3D space. The IMU sensor can be configured at the key part of the hand (such as the back of the hand and the back of the hand nail part) which does not affect the flexibility of finger operation, the flexible deformation sensor is configured at the first adjacent part including the back of the hand finger part, and the flexible pressure sensor is configured at the second adjacent part including the palm of the hand finger part. Through the IMU sensor data, the flexible deformation sensor data and the flexible pressure sensor data collected, the pose information of the hand of the target object is determined through the foregoing corresponding method.

[0076] In addition, the EMG sensor can also be configured at the corresponding region of the hand, and the motion intensity information of the target object is obtained through the EMG sensor data collected and the flexible pressure sensor data collected.

[0077] The embodiment of the present disclosure solves the problem of how to obtain the hand pose information while ensuring the flexibility of hand operation.

[0078] The embodiment of the present disclosure provides a possible implementation, the third adjacent part corresponding to the key part of the any target part region is configured with an EMG sensor; the motion information further includes motion intensity information; and the motion intensity information is determined based on the EMG sensor data collected by the EMG sensor configured at the third adjacent part corresponding to the key part of the any target part region.

[0079] The embodiment of the present disclosure can be applied to the scene of embodied intelligence through collecting human motion information to simulate the experimental activities of robots, human wearing artificial limbs, etc. For example, in the simulation experiment scene, specifically, the EMG sensor data can be configured at the third adjacent part corresponding to the key part of the target part region of the target object, the muscle information and contraction characteristics reflected by the EMG sensor data can be used to analyze the motion intensity of the target part region. In addition, the corresponding flexible pressure sensor can also be configured, and through the collected flexible pressure sensor cooperating with the EMG data, more accurate motion intensity information can be obtained. In addition, combined with the pose information of the target part of the target object, the motion speed information of the target part of the target object can also be obtained. The corresponding information obtained by analysis is provided for subsequent embodied intelligent robot research and development.

[0080] For the embodiment of the present disclosure, the pose information of the target object can be obtained, and the motion intensity, speed and other information of the target object can also be obtained.

[0081] The embodiment of the present disclosure provides a possible implementation manner, and the method further includes:

[0082] Step S103 (not shown in the figure), based on the pose information and the motion intensity information of the target part region of the target object, the next action of the target part of the target object is formulated.

[0083] Specifically, based on the pose information and the motion intensity information of the target part region of the target object, the current motion of the target object can be determined, and the next action of the target part of the target object can be determined according to the task situation of the target part of the target object.

[0084] Specifically, the next action of the target part of the target object can also be determined directly through the pre-trained neural network model according to one or more of the collected IMU sensor data, deformation sensor data, flexible pressure sensor data and EMG sensor data of the target part region of the target object.

[0085] The embodiment of the present disclosure solves the problem of how the target object acts on the corresponding target part.

[0086] The embodiment of the present disclosure provides a possible implementation manner, and the method further includes:

[0087] Step S104 (not shown in the figure), the pose information and the motion intensity information of the target region of the target object are determined.

[0088] Step S105 (not shown in the figure), based on the determined pose information and the motion intensity information of the target region of the target object, the next action of the target part of the target object is formulated.

[0089] Specifically, since there are corresponding association relationships among multiple parts of the target object, such as the coordinated work of multiple parts (arms, legs, feet, etc.) in the process of human running, the posture information and the motion intensity information of multiple target regions of the target object can be determined through the foregoing corresponding method, and based on the posture information and the motion intensity information of the multiple target part regions of the target object, the current overall motion condition of the target object can be determined, and the next action of the target object can be determined according to the target task condition of the target object, which can be a coordinated action including multiple target parts.

[0090] Specifically, the next action of the multiple parts of the target object can also be determined directly through a pre-trained neural network model according to one or more of the IMU sensor data, the deformation sensor data, the flexible pressure sensor data and the EMG sensor data collected from the multiple target part regions of the target object.

[0091] Embodiments of the present disclosure solve the problem of how the target object acts.

[0092] A motion estimation device based on multi-modal perception is provided, and the device 30 comprises:

[0093] The acquisition module 301 is configured to acquire motion evaluation related data of at least one target part region of a target object, wherein the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data.

[0094] The first determination module 302 is configured to determine motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, wherein the motion information comprises posture information.

[0095] The present disclosure provides a possible implementation, wherein a key part of each target part region of a target object is configured with an IMU sensor, and a first adjacent part corresponding to the key part is configured with a deformation sensor.

[0096] The first determination unit is configured to determine first posture information of a key part of any target part region of a target object based on IMU sensor data of the key part.

[0097] The second determination unit is configured to determine second posture information of a first adjacent part corresponding to the key part of the any target part region of the target object based on flexible deformation sensor data of the first adjacent part corresponding to the key part of the any target part region of the target object and the first posture information of the key part of the any target part region.

[0098] The third determining unit is configured to determine the pose information of the target part region of the target object based on the first pose information and the second pose information.

[0099] The disclosure provides a possible implementation, and a flexible pressure sensor is arranged at the second adjacent part corresponding to the key part of the target part region of the target object. The second determining unit is specifically configured to determine a confidence deformation variable of the first adjacent part corresponding to the key part of the target part region of the target object relative to the key part based on the flexible deformation sensor data of the first adjacent part corresponding to the key part of the target part region of the target object and the flexible pressure sensor data collected by the flexible pressure sensor arranged at the second adjacent part corresponding to the key part of the target part region of the target object, and determine second pose information of the first adjacent part corresponding to the key part of the target part region of the target object based on the confidence deformation variable and the first pose information of the key part of the target part region.

[0100] The disclosure provides a possible implementation, the key part of the target part region is a back of hand, and the first adjacent part corresponding to the key part of the target part region includes a back of hand finger part.

[0101] The disclosure provides a possible implementation, and an EMG sensor is arranged at the third adjacent part corresponding to the key part of the target part region. The motion information further includes motion intensity information, and the motion intensity information is determined based on EMG sensor data collected by the EMG sensor arranged at the third adjacent part corresponding to the key part of the target part region.

[0102] The disclosure provides a possible implementation, and the apparatus further includes:

[0103] The first formulating module is configured to formulate a next action of the target part of the target object based on the pose information of the target part region of the target object and the motion intensity information.

[0104] Optionally, the apparatus further includes:

[0105] The second determining module is configured to determine the pose information and the motion intensity information of the multiple target regions of the target object.

[0106] The second formulating module is configured to formulate a next action of the target part of the target object based on the determined pose information and the motion intensity information of the multiple target regions of the target object.

[0107] For the embodiments of the present application, the beneficial effects achieved are the same as those of the above method embodiments, which will not be repeated here.

[0108] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0109] The electronic device comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by the embodiments of the present disclosure.

[0110] Compared with the prior art, the electronic device of the present disclosure can obtain the motion evaluation related data of the target part region of the target object, including the IMU sensor data and the flexible deformation sensor data, determine the pose information of the part where the IMU sensor is configured based on the IMU sensor data, and then obtain the pose information of the part where the flexible deformation sensor is configured based on the position relationship between the configured IMU sensor and the flexible deformation sensor and the obtained flexible deformation sensor data, so as to obtain the pose information of the position where the IMU sensor is not configured in the case of limited configuration of the IMU sensor.

[0111] The readable storage medium is a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the method provided by the embodiments of the present disclosure.

[0112] Compared with the prior art, the readable storage medium of the present disclosure can obtain the motion evaluation related data of the target part region of the target object, including the IMU sensor data and the flexible deformation sensor data, determine the pose information of the part where the IMU sensor is configured based on the IMU sensor data, and then obtain the pose information of the part where the flexible deformation sensor is configured based on the position relationship between the configured IMU sensor and the flexible deformation sensor and the obtained flexible deformation sensor data, so as to obtain the pose information of the position where the IMU sensor is not configured in the case of limited configuration of the IMU sensor.

[0113] The computer program product comprises a computer program, and the computer program is executed by a processor to implement the method shown in the first aspect of the present disclosure.

[0114] Compared with the prior art, the computer program product acquires motion evaluation related data of the target part region of the target object, including IMU sensor data and flexible deformation sensor data, can determine the pose information of the part where the IMU sensor is arranged based on the IMU sensor data, and then according to the position relationship between the arranged IMU sensor and the flexible deformation sensor and the acquired flexible deformation sensor data, solves to obtain the pose information of the part where the flexible deformation sensor is arranged, so that the pose information of the position where the IMU sensor is not arranged is obtained in the case that the IMU sensor is arranged.

[0115] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0116] As shown in Figure 3 The electronic device 300 includes a computing unit 301 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0117] Various components in the electronic device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, etc., an output unit 307, such as various types of displays, speakers, etc., a storage unit 308, such as a magnetic disk, an optical disk, etc., and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0118] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running artificial intelligence model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above, such as the method for motion estimation based on multi-modal perception. For example, in some embodiments, the method for motion estimation based on multi-modal perception can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 305. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded onto the RAM 303 and executed by the computing unit 301, one or more steps of the method for motion estimation based on multi-modal perception described above can be performed. Alternatively, in other embodiments, the computing unit 301 can be configured to perform the method for motion estimation based on multi-modal perception by any other appropriate means, such as by means of firmware.

[0119] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0120] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0123] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0125] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0126] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for motion estimation based on multi-modal perception, the method comprising: The method comprises: acquiring motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data; determining motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, wherein the motion information comprises attitude information; IMU sensors are arranged at key parts of each target part region of the target object, and flexible deformation sensors are arranged at first adjacent parts corresponding to the key parts; the determination of the motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object comprises: determining first attitude information of a key part of any target part region of the target object based on IMU sensor data collected by the IMU sensor of the key part; determining second attitude information of a first adjacent part corresponding to the key part of the any target part region of the target object based on flexible deformation sensor data collected by the flexible deformation sensor of the first adjacent part corresponding to the key part of the any target part region of the target object and the first attitude information of the key part; and determining attitude information of the any target part region of the target object based on the first attitude information and the second attitude information. A flexible pressure sensor is arranged at a second adjacent part corresponding to the key part of the any target part region of the target object; and the determination of the second attitude information of the first adjacent part corresponding to the key part of the any target part region of the target object based on the flexible deformation sensor data collected by the flexible deformation sensor of the first adjacent part corresponding to the key part of the any target part region of the target object and the first attitude information of the key part comprises: determining a confidence deformation variable of the first adjacent part corresponding to the key part of the any target part region of the target object relative to the key part based on the flexible deformation sensor data collected by the flexible deformation sensor of the first adjacent part corresponding to the key part of the any target part region of the target object and flexible pressure sensor data collected by the flexible pressure sensor arranged at the second adjacent part corresponding to the key part of the any target part region of the target object; and determining the second attitude information of the first adjacent part corresponding to the key part of the any target part region of the target object based on the confidence deformation variable and the first attitude information of the key part.

2. The method of claim 1, wherein, The key part of the any target part region of the target object is the back of the hand, and the key part of the any target part region of the target object comprises a finger part on the back of the hand; the first adjacent part corresponding to the key part of the any target part region of the target object comprises a finger part on the back of the hand; and the second adjacent part corresponding to the key part of the any target part region of the target object comprises a finger part on the palm.

3. The method of claim 1, wherein, A third adjacent part corresponding to the key part of the any target part region of the target object is arranged with an EMG sensor; the motion information further comprises motion intensity information; and the motion intensity information is determined based on EMG sensor data collected by the EMG sensor arranged at the third adjacent part corresponding to the key part of the any target part region of the target object.

4. The method of claim 3, wherein, The method further comprises: Formulate the next step action of the target object at any target part based on the posture information and the motion intensity information of the target object at the target part.

5. The method of claim 4, wherein, The method further comprises: determining posture information and motion intensity information of multiple target regions of the target object; formulate the next step action of the target object at any target part based on the determined posture information and motion intensity information of the target object at the multiple target regions.

6. A multi-modal perception based motion estimation apparatus, comprising: The device comprises: an acquisition module configured to acquire motion evaluation related data of at least one target part region of a target object; the motion evaluation related data comprises IMU sensor data and flexible deformation sensor data; a first determination module configured to determine motion information of the at least one target part region of the target object based on the motion evaluation related data of the at least one target part region of the target object, the motion information comprising posture information; key parts of each target part region of the target object are configured with IMU sensors, and first adjacent parts corresponding to the key parts are configured with flexible deformation sensors; the first determination module comprises: a first determination unit configured to determine first posture information of a key part of any target part region of the target object based on IMU sensor data of the key part of the target part region; a second determination unit configured to determine second posture information of a first adjacent part corresponding to the key part of any target part region of the target object based on flexible deformation sensor data of the first adjacent part corresponding to the key part of the target part region of the target object and the first posture information of the key part of the target part region; and a third determination unit configured to determine posture information of the target object at the target part region based on the first posture information and the second posture information. The second adjacent part corresponding to the key part of the target part region of the target object is configured with a flexible pressure sensor; the second determination unit is specifically configured to determine a confidence deformation variable of the first adjacent part corresponding to the key part of the target part region of the target object relative to the key part based on flexible deformation sensor data of the first adjacent part corresponding to the key part of the target part region of the target object and flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of the target part region of the target object; and determine the second posture information of the first adjacent part corresponding to the key part of the target part region of the target object based on the confidence deformation variable and the first posture information of the key part of the target part region. 7.An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5. 8.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5.

Images