Motion estimation method and device based on multi-modal perception and electronic equipment

Abstract

The invention discloses a motion estimation method and device based on multi-modal perception and electronic equipment, and relates to the technical field of intelligent technology, in particular to the technical field of motion estimation based on multi-modal perception. According to the specific implementation scheme, motion evaluation related data, including I MU sensor data and flexible deformation sensor data, of a target part area of a target object are obtained, and pose information of a part configured with an I MU sensor can be determined based on the I MU sensor data; and according to the position relationship between the configured I MU sensor and the flexible deformation sensor and the acquired data of the flexible deformation sensor, the pose information of the position where the flexible deformation sensor is configured is calculated, so that the pose information of the position where the I MU sensor is not configured is obtained under the condition that the configuration of the I MU sensor is limited.

Description

Technical Field

[0001] The present disclosure relates to the field of embodied intelligence, and in particular to the technical field of motion estimation based on multimodal perception. Background Art

[0002] In the fields of embodied intelligence and humanoid robots, it is usually necessary to estimate the posture of the target object. Currently, posture estimation is mainly performed through inertial sensors (I MU, Inertial Measurement Unit) and visual recognition methods. Among them, the principle of using the inertial sensor method is: the inertial motion capture system is mainly composed of several measurement nodes, each of which has a nine-axis inertial measurement unit, which will return the measurement information of angular velocity, acceleration and magnetometer during the motion process; then the quaternion is obtained through the algorithm, and then posture recognition (solution) is performed to obtain the position and posture information of the target object.

[0003] However, on devices such as humanoid robots, especially in small areas such as hands, it is not possible to configure too many IMU sensors, otherwise it will affect the limb movement of related parts. How to obtain the posture information of parts without IMU sensors when configuring limited IMU sensors has become a problem. Summary of the invention

[0004] The present disclosure provides a motion estimation method, device and electronic device based on multimodal perception, which are used to obtain the posture information of the position where the IMU sensor is not configured when the IMU sensor is limited.

[0005] In a first aspect of the present disclosure, a motion estimation method based on multimodal perception is provided, the method comprising:

[0006] Acquire motion evaluation related data of at least one target part area of ​​the target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data;

[0007] Based on the motion evaluation related data of at least one target part area of ​​the target object, motion information of the at least one target part area of ​​the target object is determined, where the motion information includes posture information.

[0008] Optionally, an IMU sensor is configured at a key part of each target part region of the target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; and determining the motion information of at least one target part region of the target object based on the motion evaluation related data of at least one target part region of the target object includes:

[0009] Determine first posture information of the key part of any target part area based on the collected IMU sensor data of the key part of any target part area of ​​the target object;

[0010] Determine second posture information of the first adjacent part corresponding to the key part of any target part region based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part region of the target object and the first posture information of the key part of any target part region;

[0011] The posture information of any target part area of ​​the target object is determined based on the first posture information and the second posture information.

[0012] Optionally, a flexible pressure sensor is configured at a second adjacent portion corresponding to a key portion of any target portion region of the target object; and the second posture information of the first adjacent portion corresponding to the key portion of any target portion region of the target object is determined based on the collected flexible deformation sensor data of the first adjacent portion corresponding to the key portion of any target portion region of the target object and the first posture information of the key portion of any target portion region, including:

[0013] Determine the confidence deformation amount of the first adjacent part corresponding to the key part of any target part region of the target object relative to the key part based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part region of the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of any target part region of the target object;

[0014] Based on the confidence deformation amount and the first posture information of the key part of the any target part region, the second posture information of the first adjacent part corresponding to the key part of the any target part region is determined.

[0015] Optionally, the key parts of any target part area are the back of the hand and the nails on the back of the hand, the first adjacent parts corresponding to the key parts of any target part area include the back of the hand and the fingers, and the second adjacent parts corresponding to the key parts of any target part area include the palm of the hand and the fingers.

[0016] Optionally, an EMG sensor is configured at a third adjacent part corresponding to the key part of any target part area; the motion information also includes motion intensity information; the motion intensity information is determined based on EMG sensor data collected by the EMG sensor configured at a third adjacent part corresponding to the key part of any target part area.

[0017] Optionally, the method further comprises:

[0018] Based on the posture information of the target part area of ​​the target object and the motion intensity information, the next action of the target part of the target object is formulated.

[0019] Optionally, the method further comprises:

[0020] Determine posture information and motion intensity information of multiple target areas of the target object;

[0021] Based on the determined posture information and motion intensity information of multiple target areas of the target object, the next action of any target part of the target object is formulated.

[0022] According to a second aspect of the present disclosure, a motion estimation device based on multimodal perception is provided, the device comprising:

[0023] An acquisition module, used to acquire motion evaluation related data of at least one target part area of ​​a target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data;

[0024] The first determination module is used to determine motion information of at least one target part area of ​​the target object based on motion evaluation related data of at least one target part area of ​​the target object, wherein the motion information includes posture information.

[0025] Optionally, an IMU sensor is configured at a key part of each target part area of ​​the target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; and the first determining module includes:

[0026] A first determining unit, configured to determine first posture information of a key part of any target part region of the target object based on the collected IMU sensor data of the key part of the target part region;

[0027] A second determination unit is used to determine the second posture information of the first adjacent part corresponding to the key part of any target part area based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part area of ​​the target object and the first posture information of the key part of any target part area;

[0028] The third determining unit is used to determine the posture information of any target part area of ​​the target object based on the first posture information and the second posture information.

[0029] Optionally, a flexible pressure sensor is configured at a second adjacent part corresponding to a key part of any target part area of ​​the target object; the second determination unit is specifically used to determine a confidence deformation amount of the first adjacent part corresponding to the key part of any target part area of ​​the target object relative to the key part based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part area of ​​the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of any target part area of ​​the target object; and is used to determine the second posture information of the first adjacent part corresponding to the key part of any target part area based on the confidence deformation amount and the first posture information of the key part of any target part area.

[0030] Optionally, the key parts of any target part area are the back of the hand and the nails on the back of the hand, the first adjacent parts corresponding to the key parts of any target part area include the back of the hand and the fingers, and the second adjacent parts corresponding to the key parts of any target part area include the palm of the hand and the fingers.

[0031] Optionally, an EMG sensor is configured at a third adjacent part corresponding to the key part of any target part area; the motion information also includes motion intensity information; the motion intensity information is determined based on EMG sensor data collected by the EMG sensor configured at a third adjacent part corresponding to the key part of any target part area.

[0032] Optionally, the device further comprises:

[0033] The first formulation module is used to formulate the next action of any target part of the target object based on the posture information of the region of any target part of the target object and the motion intensity information.

[0034] Optionally, the device further comprises:

[0035] A second determination module is used to determine the posture information and motion intensity information of multiple target areas of the target object;

[0036] The second formulation module is used to formulate the next action of any target part of the target object based on the determined posture information and movement intensity information of multiple target areas of the target object.

[0037] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method shown in the first aspect of the present disclosure.

[0038] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as shown in the first aspect of the present disclosure.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method shown in the first aspect of the present disclosure.

[0040] The present disclosure provides a motion estimation method based on multimodal perception, including: acquiring motion evaluation related data of at least one target part area of ​​a target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data; based on the motion evaluation related data of at least one target part area of ​​the target object, determining the motion information of the at least one target part area of ​​the target object, the motion information includes posture information. That is, the motion evaluation related data of the target part area of ​​the target object includes IMU sensor data and flexible deformation sensor data, and the posture information of the part configured with the IMU sensor can be determined based on the IMU sensor data, and then according to the positional relationship between the configured IMU sensor and the flexible deformation sensor, the flexible deformation sensor data is obtained, and the posture of the part configured with the flexible deformation sensor is solved, thereby achieving the posture information of the position where the IMU sensor is not configured when the number of configured IMU sensors is limited.

[0041] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0043] Figure 1 is a flowchart of a motion estimation method based on multimodal perception provided by an embodiment of the present disclosure;

[0044] Figure 2 is a schematic diagram of the structure of a motion estimation device based on multimodal perception provided by an embodiment of the present disclosure;

[0045] Figure 3 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to aid understanding, and they should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness,

[0047] In the following description, description of well-known functions and structures is omitted.

[0048] Embodiment 1

[0049] Figure 1 A motion estimation method based on multimodal perception provided by an embodiment of the present disclosure is shown, and the method includes:

[0050] Step S101, obtaining motion evaluation related data of at least one target part area of ​​a target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data;

[0051] As mentioned earlier, the IMU sensor has the disadvantage of zero bias error. In some embodied intelligent robot (target object) application scenarios, high precision requirements are placed on the operation of the target object (such as at the hand fingers), which makes it impossible to configure too many IMU sensors, otherwise the flexibility of the operation of the hand fingers will be affected.

[0052] Specifically, the IMU sensor data and flexible deformation sensor data of the target part area can be obtained through the limited IMU sensors and flexible deformation sensors configured in the target part area of ​​the target object. Among them, the target object can be a humanoid robot, a real person performing a simulation experiment, or a robot or device to which the disclosed method can be applied. The target part area can be the corresponding part of the target object. Taking a humanoid robot as an example, the target part area can be a hand area composed of the back of the hand, the back of the hand, the fingernails, the fingertips, the palm of the hand, the palm of the hand, etc., such as a head area composed of the head, the neck, the neck, etc.

[0053] Among them, traditional strain sensors based on metals and semiconductors are rigid and cannot measure flexible and stretchable objects. Generally speaking, flexible materials have the characteristics of being bendable and deformable. Flexible deformation (strain) sensors are sensors that have the characteristics of being bendable and deformable and can measure the degree of deformation of an object.

[0054] Among them, the IMU sensor data may be data after calibration processing by the flexible deformation sensor. During the application process, the flexible deformation sensor data collected by the configured flexible deformation sensor can be compared with the flexible deformation sensor data corresponding to each preset standard action in the pre-recorded preset standard action library for consistency; if there is consistent data compared, it is determined that the current action of the target part of the target object is the preset standard action. Then, the current IMU sensor data is calibrated based on the IMU sensor parameter information corresponding to the pre-recorded preset standard action. Specifically, the calibration process can be performed when the error between the IMU sensor parameter information corresponding to the pre-recorded preset standard action and the current IMU sensor data exceeds a predetermined range.

[0055] Step S102: determining motion information of at least one target part region of the target object based on motion evaluation related data of the at least one target part region of the target object, wherein the motion information includes posture information.

[0056] Specifically, the motion information of the target part area corresponding to the target object can be determined based on the motion evaluation related data of a target part area of ​​the target object. Specifically, the position information of the part configured with the IMU sensor can be determined based on the IMU sensor data, and then the position information of the part configured with the flexible deformation sensor can be solved based on the positional relationship between the configured IMU sensor and the flexible deformation sensor, thereby achieving the position information of the position where the IMU sensor is not configured when the IMU sensor is limited. In addition, the position information of the area where the IMU sensor and the flexible sensor are not configured can also be solved based on the positional relationship between other areas of the target area that are not configured with IMU sensors and flexible sensors and the areas where the IMU sensors and / or flexible sensors are configured. The motion information of the target object can also be determined through a pre-trained neural network model.

[0057] Specifically, it is also possible to obtain motion evaluation related data of multiple target part areas of the target object, and then determine the motion information of each target part area respectively. In addition, since there is an association relationship between the relevant target area parts of the target object, the motion information of a certain target part area can also be determined based on the motion evaluation related data of multiple target part areas with an association relationship. For example, a neural network model can be trained through multiple positive and negative sample data (motion evaluation related information of multiple target part areas) and the corresponding motion information label of a certain target part area, and then the motion information of a certain target part area can be determined through the pre-trained neural network model and the motion evaluation related information of multiple target part areas obtained during the application process.

[0058] The motion information includes posture information, wherein the posture expresses the position and posture of the target part.

[0059] The present disclosure provides a motion estimation method based on multimodal perception, including: acquiring motion evaluation related data of at least one target part area of ​​a target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data; based on the motion evaluation related data of at least one target part area of ​​the target object, determining the motion information of the at least one target part area of ​​the target object, the motion information includes posture information. That is, the motion evaluation related data of the target part area of ​​the target object includes IMU sensor data and flexible deformation sensor data, and the posture information of the part configured with the IMU sensor can be determined based on the IMU sensor data, and then according to the positional relationship between the configured IMU sensor and the flexible deformation sensor, the flexible deformation sensor data is obtained, and the posture of the part configured with the flexible deformation sensor is solved, thereby achieving the posture information of the position where the IMU sensor is not configured when the number of configured IMU sensors is limited.

[0060] The present disclosure provides a possible implementation method, wherein an IMU sensor is configured at a key part of each target part region of a target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; and step S102 determines motion information of at least one target part region of the target object based on motion evaluation related data of at least one target part region of the target object, including:

[0061] Step S1021 (not shown in the figure), determining first posture information of the key part of any target part area based on the collected IMU sensor data of the key part of any target part area of ​​the target object;

[0062] Specifically, the key part of any target part area of ​​the target object is configured with IMU sensor data, wherein the key part can be a part where the IMU sensor is configured and does not affect the flexibility of the target object. By collecting the IMU sensors of the key part of any target part area of ​​the target object, the position and posture information of the key part can be calculated through the corresponding algorithm.

[0063] Step S1022 (not shown in the figure), based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part area of ​​the target object and the first posture information of the key part of any target part area, determine the second posture information of the first adjacent part corresponding to the key part of any target part area;

[0064] Specifically, based on the positional relationship between the key part of any target part area and the corresponding first adjacent part, according to the collected flexible deformation sensor data of the corresponding first adjacent part, the confidence deformation amount of the key part relative to any target part area up to the first adjacency can be obtained, and according to the first posture information of the key part of any target part area, the second posture information of the first adjacent part can be calculated.

[0065] Step S1023 (not shown in the figure), determining the posture information of any target part area of ​​the target object based on the first posture information and the second posture information.

[0066] Specifically, the posture information of any target part region includes the posture information of the key part of the target part region and the adjacent parts of the key part. The posture information of any target part region of the target object can be determined according to the first posture information and the second posture information.

[0067] The embodiments of the present disclosure solve the problem of how to determine the posture information of the target area based on the flexible deformation sensor data and the IMU sensor data.

[0068] The present disclosure provides a possible implementation method, wherein a flexible pressure sensor is configured at a second adjacent part corresponding to a key part of any target part region of the target object; and based on the collected flexible deformation sensor data of a first adjacent part corresponding to a key part of any target part region of the target object and the first posture information of the key part of any target part region, the second posture information of the first adjacent part corresponding to the key part of any target part region is determined, including:

[0069] Determine the confidence deformation amount of the first adjacent part corresponding to the key part of any target part region of the target object relative to the key part based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part region of the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of any target part region of the target object;

[0070] Specifically, based on the positional relationship between the first adjacent part configured with the flexible deformation sensor and the second adjacent part configured with the flexible pressure sensor, combined with the collected data of the flexible deformation sensor of the first adjacent part and the flexible pressure sensor of the second adjacent part, the confidence deformation amount of the key part of any target part area corresponding to the first adjacent part relative to the key part can be determined.

[0071] Based on the confidence deformation amount and the first posture information of the key part of the any target part region, the second posture information of the first adjacent part corresponding to the key part of the any target part region is determined.

[0072] Specifically, according to the confidence deformation amount, the first posture information of the key part of any target part area is converted accordingly to obtain the second posture information of the first adjacent part.

[0073] In the disclosed embodiment, the flexible deformation sensor data is compensated by the flexible pressure sensor data, so that the calculated confidence deformation amount is more accurate, thereby making the determined posture information more accurate.

[0074] The present disclosure provides a possible implementation method, wherein the key parts of any target part area are the back of the hand and the nail part of the back of the hand, the first adjacent parts corresponding to the key parts of any target part area include the back of the hand and the fingers, and the second adjacent parts corresponding to the key parts of any target part area include the palm of the hand and the fingers.

[0075] Specifically, due to the limited area of ​​the hand, configuring too many IMU sensors will affect the flexibility of hand operation. In addition, a single deformation sensor and pressure sensor cannot reflect the changes in the target 3D space. IMU sensors can be configured at key parts of the hand that do not affect the flexibility of finger operation (such as the back of the hand and the nails on the back of the hand), including configuring a flexible deformation sensor at the first adjacent part of the back of the hand finger part, and configuring a flexible pressure sensor at the second adjacent part of the palm finger part. The posture information of the target object's hand is determined by the collected IMU sensor data, flexible deformation sensor data, and flexible pressure sensor data through the aforementioned corresponding methods.

[0076] In addition, EMG sensors may be configured in corresponding areas of the hand, and the motion intensity information of the target object may be calculated through the collected EMG sensor data and the collected flexible pressure sensor data.

[0077] The disclosed embodiments solve the problem of how to obtain hand posture information while ensuring the flexibility of hand operation.

[0078] An embodiment of the present disclosure provides a possible implementation method, wherein an EMG sensor is configured at a third adjacent part corresponding to the key part of any target part area; the motion information also includes motion intensity information; the motion intensity information is determined based on EMG sensor data collected by the EMG sensor configured at the third adjacent part corresponding to the key part of any target part area.

[0079] The disclosed embodiments can be applied to scenarios such as embodied intelligence simulating robot experimental activities by collecting human motion information, and human body wearing prostheses. For example, in a simulated experimental scenario, specifically, EMG sensor data can be configured at the third adjacent part corresponding to the key part of any target part area of ​​the target object, and the motion intensity of the target part area can be analyzed based on the muscle information and contraction characteristics reflected by the EMG sensor data. In addition, corresponding flexible pressure sensors can also be configured. By combining the collected flexible pressure sensors with EMG data, more accurate motion intensity information can be obtained. In addition, combined with the posture information of the target part of the target object, the motion speed information of the target part of the target object can also be obtained. The corresponding information obtained by analysis is provided to the subsequent development and use of embodied intelligent robots.

[0080] According to the embodiment of the present disclosure, while being able to obtain the position information of the target object, it is also possible to obtain information such as the motion intensity and speed of the target object.

[0081] The present disclosure provides a possible implementation method, wherein the method further includes:

[0082] Step S103 (not shown in the figure), based on the posture information of the any target part area of ​​the target object and the motion intensity information, the next action of the any target part of the target object is formulated.

[0083] Specifically, based on the posture information of any target part area of ​​the target object and the motion intensity information, the current motion status of the target object can be determined, and according to the task status of the target part of the target object, the next action for any target part of the target object can be determined.

[0084] Specifically, the next action for any target part of the target object can be determined directly through a pre-trained neural network model based on one or more of the collected IMU sensor data, deformation sensor data, flexible pressure sensor data, and EMG sensor data of any target part area of ​​the target object.

[0085] The disclosed embodiments solve the problem of how the target object should act in accordance with the target part.

[0086] The present disclosure provides a possible implementation method, wherein the method further includes:

[0087] Step S104 (not shown in the figure), determining the posture information and motion intensity information of multiple target areas of the target object;

[0088] Step S105 (not shown in the figure), based on the determined posture information and motion intensity information of the multiple target areas of the target object, formulate the next action of any target part of the target object.

[0089] Specifically, since there are corresponding associations between multiple parts of the target object, such as when the human body is running, the coordinated work of multiple parts (arms, legs, feet, etc.) is required, the posture information and motion intensity information of multiple target areas of the target object can be determined through the aforementioned corresponding methods. Based on the posture information and motion intensity information of multiple target part areas of the target object, the current overall motion status of the target object can be determined, and according to the target task status of the target object, the next action for the target object can be determined, which can be a coordinated action involving multiple target parts.

[0090] Specifically, the next action for multiple parts of the target object can be determined directly through a pre-trained neural network model based on one or more of the collected IMU sensor data, deformation sensor data, flexible pressure sensor data, and EMG sensor data of multiple target areas of the target object.

[0091] The disclosed embodiments solve the problem of how the target object acts.

[0092] A motion estimation device based on multimodal perception is provided, the device 30 comprising:

[0093] The acquisition module 301 is used to acquire motion evaluation related data of at least one target part area of ​​the target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data;

[0094] The first determination module 302 is configured to determine motion information of at least one target part region of the target object based on motion evaluation related data of the at least one target part region of the target object, wherein the motion information includes posture information.

[0095] The present disclosure provides a possible implementation method, wherein an IMU sensor is configured at a key part of each target part area of ​​the target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; the first determination module includes:

[0096] A first determining unit, configured to determine first posture information of a key part of any target part region of the target object based on the collected IMU sensor data of the key part of the target part region;

[0097] A second determination unit is used to determine the second posture information of the first adjacent part corresponding to the key part of any target part area based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part area of ​​the target object and the first posture information of the key part of any target part area;

[0098] The third determining unit is used to determine the posture information of any target part area of ​​the target object based on the first posture information and the second posture information.

[0099] The present disclosure provides a possible implementation method, wherein a flexible pressure sensor is configured at a second adjacent part corresponding to a key part of any target part area of ​​the target object; the second determining unit is specifically used to determine a confidence deformation amount of a first adjacent part corresponding to a key part of any target part area of ​​the target object relative to the key part based on the collected flexible deformation sensor data of a first adjacent part corresponding to the key part of any target part area of ​​the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of any target part area of ​​the target object; and is used to determine second posture information of the first adjacent part corresponding to the key part of any target part area based on the confidence deformation amount and the first posture information of the key part of any target part area.

[0100] The present disclosure provides a possible implementation method, wherein the key parts of any target part area are the back of the hand and the nail part of the back of the hand, the first adjacent parts corresponding to the key parts of any target part area include the back of the hand and the fingers, and the second adjacent parts corresponding to the key parts of any target part area include the palm of the hand and the fingers.

[0101] The present disclosure provides a possible implementation method, wherein an EMG sensor is configured at a third adjacent part corresponding to the key part of any target part area; the motion information also includes motion intensity information; the motion intensity information is determined based on EMG sensor data collected by the EMG sensor configured at the third adjacent part corresponding to the key part of any target part area.

[0102] The present disclosure provides a possible implementation method, wherein the device further includes:

[0103] The first formulation module is used to formulate the next action of any target part of the target object based on the posture information of the region of any target part of the target object and the motion intensity information.

[0104] Optionally, the device further comprises:

[0105] A second determination module is used to determine the posture information and motion intensity information of multiple target areas of the target object;

[0106] The second formulation module is used to formulate the next action of any target part of the target object based on the determined posture information and movement intensity information of multiple target areas of the target object.

[0107] For the embodiments of the present application, the beneficial effects achieved are the same as those of the above-mentioned method embodiments and will not be repeated here.

[0108] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0109] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the embodiment of the present disclosure.

[0110] Compared with the prior art, the electronic device can determine the posture information of the part equipped with the IMU sensor based on the IMU sensor data by acquiring motion evaluation related data of the target part area of ​​the target object, including IMU sensor data and flexible deformation sensor data, and then calculate the posture information of the part equipped with the flexible deformation sensor according to the positional relationship between the configured IMU sensor and the flexible deformation sensor and the acquired flexible deformation sensor data, thereby achieving the posture information of the position without the IMU sensor configured when the number of configured IMU sensors is limited.

[0111] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method provided by the embodiment of the present disclosure.

[0112] Compared with the prior art, the readable storage medium can determine the posture information of the part equipped with the IMU sensor based on the IMU sensor data by acquiring motion evaluation-related data of the target part area of ​​the target object, including IMU sensor data and flexible deformation sensor data, and then calculate the posture information of the part equipped with the flexible deformation sensor according to the positional relationship between the configured IMU sensor and the flexible deformation sensor and the acquired flexible deformation sensor data, thereby achieving the posture information of the position without the IMU sensor configured when the number of configured IMU sensors is limited.

[0113] The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the method as shown in the first aspect of the present disclosure.

[0114] Compared with the prior art, the computer program product can obtain motion assessment related data of the target part area of ​​the target object, including IMU sensor data and flexible deformation sensor data, and can determine the position and posture information of the part configured with the IMU sensor based on the IMU sensor data, and then calculate and obtain the position and posture information of the part configured with the flexible deformation sensor according to the positional relationship between the configured IMU sensor and the flexible deformation sensor and the obtained flexible deformation sensor data, thereby achieving the position and posture information of the position not configured with the IMU sensor when the number of configured IMU sensors is limited.

[0115] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 3 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 303 is also connected to the bus 304.

[0117] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running artificial intelligence model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a motion estimation method based on multimodal perception. For example, in some embodiments, the motion estimation method based on multimodal perception may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 305. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the motion estimation method based on multimodal perception described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the motion estimation method based on multimodal perception in any other appropriate manner (for example, by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A motion estimation method based on multimodal perception, characterized in that: The method includes: Acquire motion evaluation related data of at least one target part area of ​​the target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data; Based on the motion evaluation related data of at least one target part area of ​​the target object, motion information of the at least one target part area of ​​the target object is determined, where the motion information includes posture information.

2. The method according to claim 1, characterized in that An IMU sensor is configured at a key part of each target part area of ​​the target object, and a deformation sensor is configured at a first adjacent part corresponding to the key part; The step of determining the motion information of at least one target part region of the target object based on the motion evaluation related data of at least one target part region of the target object comprises: Determine first posture information of the key part of any target part area based on the collected IMU sensor data of the key part of any target part area of ​​the target object; Determine second posture information of the first adjacent part corresponding to the key part of any target part region based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part region of the target object and the first posture information of the key part of any target part region; The posture information of any target part area of ​​the target object is determined based on the first posture information and the second posture information.

3. The method according to claim 2, characterized in that The second adjacent part corresponding to the key part of any target part area of ​​the target object is configured with a flexible pressure sensor; the second posture information of the first adjacent part corresponding to the key part of any target part area of ​​the target object is determined based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part area of ​​the target object and the first posture information of the key part of any target part area, including: Determine the confidence deformation amount of the first adjacent part corresponding to the key part of any target part region of the target object relative to the key part based on the collected flexible deformation sensor data of the first adjacent part corresponding to the key part of any target part region of the target object and the flexible pressure sensor data collected by the flexible pressure sensor configured at the second adjacent part corresponding to the key part of any target part region of the target object; Based on the confidence deformation amount and the first posture information of the key part of the any target part region, the second posture information of the first adjacent part corresponding to the key part of the any target part region is determined.

4. The method according to claim 3, characterized in that The key parts of any target part area are the back of the hand and the nails on the back of the hand, the first adjacent parts corresponding to the key parts of any target part area include the back of the hand and the fingers, and the second adjacent parts corresponding to the key parts of any target part area include the palm of the hand and the fingers.

5. The method according to claim 2, characterized in that: An EMG sensor is configured at the third adjacent part corresponding to the key part of any target part area; the motion information also includes motion intensity information; the motion intensity information is determined based on EMG sensor data collected by the EMG sensor configured at the third adjacent part corresponding to the key part of any target part area.

6. The method according to claim 5, characterized in that The method further includes: Based on the posture information of the target part area of ​​the target object and the motion intensity information, the next action of the target part of the target object is formulated.

7. The method according to claim 6, characterized in that The method further includes: Determine posture information and motion intensity information of multiple target areas of the target object; Based on the determined posture information and motion intensity information of multiple target areas of the target object, the next action of any target part of the target object is formulated.

8. A motion estimation device based on multimodal perception, characterized in that: The device includes: An acquisition module, used to acquire motion evaluation related data of at least one target part area of ​​the target object; the motion evaluation related data includes IMU sensor data and flexible deformation sensor data; The first determination module is used to determine motion information of at least one target part area of ​​the target object based on motion evaluation related data of at least one target part area of ​​the target object, wherein the motion information includes posture information.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images