Motion capture method and device

By combining the local information obtained by the wearable device and the global information obtained by the camera device, the problem of the balance of accuracy and convenience in the prior art is solved, and the accuracy and noise resistance of the motion capture system are improved.

CN119007277AInactive Publication Date: 2024-11-22SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310572499.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing real-time human motion capture system is difficult to balance between motion capture accuracy and convenience, especially wearable devices pay more attention to local information and cannot accurately reflect the global information of the human body. The camera is susceptible to external environment interference, resulting in local errors. big.

Method used

By combining local information and global information related to the object's pose, the wearable device obtains local information and the imaging device obtains global information, and weighted combinations are performed to determine the object's pose information, improving the accuracy and convenience of motion capture.

Benefits of technology

It realizes a more accurate reflection of the object's posture and motion conditions, reduces the impact of external interference, and improves the overall performance of the motion capture system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007277A_ABST
    Figure CN119007277A_ABST
Patent Text Reader

Abstract

The invention relates to a motion capture method and equipment. An electronic device for motion capture, the device comprising: a processing circuit configured to acquire local information related to a specific part attitude of a motion capture object; global information related to the posture of the specific part of the motion capture object is obtained; and combining the local information and the global information to determine attitude information of the motion capture object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to motion capture, and more particularly to a motion capture method and device. Background Art

[0002] Motion capture is widely used in the field of image processing and analysis, and with the development of technology, motion capture can be used in more and more fields, such as computer animation, television, movies, video games, sports monitoring and medical rehabilitation. Usually, the user is equipped with appropriate sensors so that the action can be recognized through the information provided by the sensors. Summary of the invention

[0003] This summary is provided to introduce the concepts of the present disclosure in a simplified form, and these concepts will be described in detail in the following detailed description.

[0004] The present disclosure provides an improved motion capture solution.

[0005] In one aspect of the present disclosure, an electronic device for motion capture is provided, the device comprising a processing circuit configured to: obtain local information related to the posture of a specific part of a motion capture object; obtain global information related to the posture of a specific part of the motion capture object; and combine the local information and the global information to determine the posture information of the motion capture object.

[0006] In another aspect of the present disclosure, a motion capture method is provided, comprising: obtaining local information related to the posture of a specific part of a motion capture object; obtaining global information related to the posture of the specific part of the motion capture object; and combining the local information and the global information to determine the posture information of the motion capture object.

[0007] In yet another aspect of the present disclosure, a motion capture system is provided, comprising a wearable device, which is configured at a specific part of a motion capture object and is configured to obtain local information related to the posture of the specific part of the motion capture object; a camera device, which is configured to obtain global information related to the posture of the specific part of the motion capture object; and a processor, which is configured to combine the local information and the global information to determine the posture information of the motion capture object.

[0008] In yet another aspect of the present disclosure, a device is provided, comprising at least one processor and at least one storage device, wherein the at least one storage device stores instructions thereon, which, when executed by the at least one processor, enable the at least one processor to perform a method according to an embodiment of the present disclosure.

[0009] In still another aspect of the present disclosure, a storage medium storing instructions is provided, which, when executed by a processor, can enable the execution of a method according to an embodiment of the present disclosure.

[0010] In still another aspect of the present disclosure, a program product is provided. The program product includes instructions. When the instructions are executed by a processor, the processor can execute the method according to the embodiment of the present disclosure.

[0011] In still another aspect of the present disclosure, a computer program is provided. The computer program includes instructions. When the instructions are executed by a processor, the processor can execute the method according to the embodiment of the present disclosure.

[0012] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure, and each drawing together with the following detailed description is included in this specification and forms a part of the specification for explaining the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure.

[0014] Figure 1 A conceptual diagram of a motion capture solution according to an embodiment of the present disclosure is shown.

[0015] Figure 2 A schematic block diagram of an electronic device for motion capture according to an embodiment of the present disclosure is shown.

[0016] Figure 3A and 3B A flow chart of a motion capture method according to an embodiment of the present disclosure is shown.

[0017] Figure 4A : shows a schematic block diagram of a motion capture system according to an embodiment of the present disclosure,

[0018] Figure 4B A schematic operation diagram of a motion capture system according to an embodiment of the present disclosure is shown.

[0019] Figure 5 A schematic block diagram of a motion capture system according to an embodiment of the present disclosure is shown.

[0020] Figure 6 A schematic block diagram of motion capture preprocessing according to an embodiment of the present disclosure is shown.

[0021] Figure 7A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing an embodiment of the present disclosure is shown.

[0022] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the present disclosure, but it should be understood that the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The drawings and the description of the embodiments below are actually only illustrative and do not serve as any limitation on the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein.

[0024] In addition, when describing the exemplary embodiments of the present disclosure in conjunction with the accompanying drawings below, for the sake of clarity and conciseness, not all features of the embodiments are described in the specification. It should be noted that in order to avoid obscuring the present disclosure due to unnecessary details, only the processing steps and / or device structures closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not closely related to the present disclosure are omitted.

[0025] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0026] In the present disclosure, the terms "first", "second", etc. are used merely to distinguish elements or steps, but are not intended to indicate time sequence, priority or importance.

[0027] With the development of the technology itself and the improvement of the technical level of related application fields, motion capture technology, especially real-time motion capture technology, has been used more and more widely. The inventors have found that real-time motion capture technology can capture the motion of an object by using a small number of cameras and / or wearable devices. Wearable devices can be worn on specific joints of a moving object, such as a specific person, to capture the motion of the object in real time as the object moves, and can be applied to various purposes. For example, motion capture technology has very broad application prospects in fields such as animation production, ergonomics research, simulation training, rehabilitation medicine, biomechanics research, human-computer interaction, virtual reality systems, robot remote control, interactive games, etc.

[0028] However, the inventors have found that the current real-time human motion capture system has problems in balancing the accuracy and convenience of motion capture. Although reducing the use of wearable devices for motion capture can make users feel simple and convenient, it will also cause a significant reduction in the accuracy of the motion capture system. In particular, wearable devices are more focused on the information of various parts of the human body and cannot more accurately reflect the global information of the human body. Cameras that take images and / or videos of the human body cannot accurately and comprehensively obtain high-accuracy human motion capture. Moreover, since the camera is easily affected by the external environment, the local error is too large.

[0029] Therefore, an improved motion capture solution is needed to improve the accuracy and / or convenience of motion capture.

[0030] In view of this, the present disclosure proposes an improved motion capture solution. In particular, the present disclosure proposes to combine multiple information related to the object posture, especially to combine local information and global information related to the object posture, so as to improve the accuracy and / or convenience of motion capture. It should be noted that the information related to the object posture can include various types of information. In some embodiments, it includes at least one of the spatial position information of the object or a specific part of the object, and motion-related information, wherein the motion-related information includes at least one of the motion direction, displacement, speed, and motion acceleration.

[0031] The embodiments according to the present disclosure are described in detail below in conjunction with illustrative examples. Figure 1 A schematic conceptual diagram of a motion capture solution according to an embodiment of the present disclosure is shown.

[0032] On the one hand, global information related to the posture of the object is obtained. Global information may refer to information that characterizes the posture of the object in a global sense, for example, it may include information that can characterize the posture of the object in the case of reflecting the relative relationship between a specific part of the object and the object as a whole, the situation in which the object is located, the environment, and the space. In some embodiments, the global information can be obtained based on the image / video of the object collected by a camera device that is separated from the object, for example, it can be achieved by various appropriate feature extraction and analysis methods, and can be in various appropriate formats and represented by various appropriate forms. As an example, the global information may include two-dimensional information or three-dimensional information that characterizes the posture of the object or the posture of a specific part of the object. For example, it can be represented by a vector, a vector, a matrix, etc.

[0033] As an example, the camera device can be any appropriate image / video acquisition device, and can include various appropriate types of camera devices, such as cameras, mobile camera devices, etc., and can be set at various appropriate positions relative to the object. For example, the front, side, etc. of the object, as long as it is suitable for capturing the overall image of the object. In addition, one or more camera devices can be set relative to the object, so that appropriate images / videos of the object can be captured. As an example, the camera device can be a camera set in front of the user, which can capture the image of the whole body or part of the user to obtain a two-dimensional image frame. As another example, the camera can also be set around the object to capture the image of the whole body or part of the user to obtain a three-dimensional image.

[0034] On the other hand, local information related to the posture of the object is obtained. Local information may refer to information that characterizes the posture of the object in a local sense, such as only considering the posture-related information of a specific part of the object, which is often determined by the local condition of the specific part. In some embodiments, the local information can be obtained based on a sensing device set at a specific part of the object, and can be in various appropriate formats and represented by various appropriate forms. As an example, the global information may include three-dimensional spatial information that characterizes the posture of the object or the posture of a specific part of the object. For example, it can be represented by a vector, a vector, a matrix, etc.

[0035] As an example, the sensing device may be a sensor-type device that can be configured on a specific part of an object and sense the posture, motion status, etc. of the specific part of the object, and may also be referred to as a wearable device. For example, the sensing device may be a device that senses based on inertia, such as an inertial sensor. In some embodiments, an appropriate number of sensing devices may be provided, such as the same number of sensing devices as the specific parts, and each of the sensing devices may be worn on the specific parts to obtain local information of each part.

[0036] Then, a combination of global information and local information may be performed to obtain improved action information. In a sense, the combination may be equivalent to modifying the local information by utilizing the global information. The combination may be performed in various suitable ways. In some embodiments, a weighted combination of global information and local information may be included.

[0037] Thus, improved information about the object's posture can be obtained, which can more accurately reflect the object's posture or motion. In addition, in addition to wearable sensing devices, a small number of camera devices can be used to obtain appropriate global information to achieve information correction and optimization, so that more accurate object posture or motion information can be easily obtained.

[0038] Various exemplary implementations of the motion capture solution according to the embodiments of the present disclosure will be described below with reference to the accompanying drawings. Figure 2 A block diagram of an electronic device for motion capture according to an embodiment of the present disclosure is shown. The electronic device 200 includes a processing circuit 202, which is configured to: obtain local information related to the posture of a specific part of the motion capture object; obtain global information related to the posture of the specific part of the motion capture object; combine the local information and the global information to determine the posture information of the motion capture object.

[0039] In some embodiments of the present disclosure, the specific part may include a part of the object appropriately selected or set from all parts of the object. Preferably, the specific part may include a representative part or a basic part that can characterize or derive the motion posture of the object. In some embodiments, the object is a human body whose motion is captured, and the specific part includes a basic or representative joint part of the human body, including at least one of the head, waist, wrist, ankle, etc.

[0040] Local information

[0041] In some embodiments of the present disclosure, local information related to the posture of a specific part of the motion capture object may indicate posture-related information of the specific part in a local sense, which may include posture-related information or spatial information of the object in a specific or local space / coordinate system, such as characterizing the spatial position, motion state, etc. of the object in the specific or local space / coordinate system. As an example, the specific or local space / coordinate system may include a coordinate system set by a wearable device for acquiring local information, or other appropriate coordinate systems.

[0042] According to an embodiment of the present disclosure, the acquisition of local information may be implemented in various appropriate ways. As an example, the local information may be obtained based on a specific sensing device / wearable device. As an example, the specific sensing device / wearable device may be an inertial sensor, a motion capture sensor, a gyroscope, etc.

[0043] The local information may include various appropriate types of information and may be expressed in various appropriate forms. In some embodiments of the present disclosure, the local information may include or derive information related to the posture of the specific part, which may also be referred to as the first spatial information, which may include at least one of the posture of the part, the spatial position, and the motion-related information. As an example, the motion-related information may include the moving direction, moving speed, moving acceleration, moving frequency, displacement, etc. of the part. As an example, the local information may be in a three-dimensional form, referred to as three-dimensional spatial information, and include at least one of the three-dimensional spatial position information and motion-related information of the specific part. In particular, the motion-related information of the specific part includes at least one of the motion acceleration, angular velocity, velocity, and displacement of the specific part.

[0044] In some embodiments, additionally or alternatively, the local information may also include other information, such as an index indicating local information related to the posture of a specific part. The index may be set in any appropriate manner. For example, a unique index is set for each part, and the information of each part may be set as a sub-index according to a time sequence, for example, time may be used as a sub-index, or a corresponding numerical value may be set as a sub-index according to a time sequence, and so on. As an example, the local information related to the posture of a specific part and the index may be pre-associated and stored, and the corresponding local information may be called according to the index during the application process. In some embodiments, the local information may include only the index, only posture related information, both the index and posture related information, and so on.

[0045] In still other embodiments, the local posture information may be subjected to specific processing, such as filtering, denoising, and the like.

[0046] Global Information

[0047] In some embodiments of the present disclosure, the global information related to the posture of a specific part of the motion capture object may indicate information that characterizes the posture of the object in a global sense, which may include posture-related information or spatial information of the object in a specific or global space / coordinate system, for example, information that characterizes / reflects the spatial position, motion-related information, etc. of a specific part in a specific or global space / coordinate system. In particular, the specific or global space / coordinate system may include a coordinate system set by a camera device for acquiring global information, a coordinate system of the space in which the object is located, or other appropriate coordinate systems. It should be noted that the specific or global space / coordinate system is often different from the aforementioned local space / coordinate system.

[0048] According to some embodiments of the present disclosure, global information can be obtained based on the image frames of the object captured by the camera device in the scene or scene where the object is located. Global information can be obtained by various appropriate methods. In some embodiments, an image / video of the entire object can be captured, and then relevant information about the posture of a specific part can be analyzed or extracted therefrom. The information extraction here can be achieved by methods known in the art and will not be described in detail here. In some embodiments, a specific part of the object can be directly captured to obtain relevant information about its posture.

[0049] In some examples, the user's movements can be obtained through a camera / camera, and then the user's posture analysis can be performed from the images or videos of the user's movements obtained to obtain information / data related to the user's posture. In other examples, a specific color or a specific label can be set on a specific part of the user, and then the movement of the corresponding part can be obtained through camera color recognition. For example, a patch of a specific color can be attached to at least one finger of the user, and then the information / data related to the user's finger posture / movement can be captured through camera color recognition. In still other examples, the movement state of the contour, skeleton, etc. of a specific part can be detected by a projector or other equipment through a specific algorithm, so as to obtain the posture of the corresponding part through the movement of the overall contour, skeleton, etc. of the part.

[0050] According to some embodiments of the present disclosure, the global information may include various appropriate types of information and may be expressed in various appropriate forms. In particular, similar to the local information, the global information may also include or derive information related to the posture of the specific part, which may also be referred to as the second spatial information. The posture information may include at least one of the posture of the part, the spatial position, and the motion related information. In some embodiments, the global information may be two-dimensional information or three-dimensional information.

[0051] In some embodiments, the global information may be two-dimensional information, especially when only a single image / video camera is used. As an example, when only a single image / video camera is used, the camera may be generally set in front of the object to obtain a two-dimensional image / video of the object, from which the two-dimensional global information is analyzed.

[0052] In some embodiments, the global information may be three-dimensional information. As an example, when multiple cameras are arranged around an object, a three-dimensional image of the object may be obtained or combined, and the three-dimensional global information may be obtained by analysis. As another example, even when a single image / video camera is arranged, two-dimensional information may be obtained by analysis, and then three-dimensional information may be obtained by expansion of the two-dimensional information. The analysis, extraction or expansion of the three-dimensional information may be implemented in various appropriate ways in the art, and will not be described in detail here.

[0053] In some embodiments, similar to the local information, the global information may also include a corresponding index, and / or be subjected to various appropriate preprocessing, which will not be described in detail here.

[0054] Information combination

[0055] In the present disclosure, local information and global information can be combined in various appropriate ways, so that the posture information or motion information of the object can be determined based on the combination result. In particular, when both global information and local information contain various types of information, a combination of at least one type of information corresponding to the global information and the local information can be included. As an example, a combination of at least one of the corresponding spatial position information and motion-related information in the global information and the local information can be included.

[0056] In some embodiments, the local information of a specific part and the global information can be combined, and then the information related to the posture of the object can be obtained based on the combination result. In particular, the local information of a specific part of the object and the global information can be combined to obtain the information related to the posture of the specific part, and based on the posture related information of the specific part of the motion capture object obtained by the combination, the posture information or motion information of the object as a whole can be inferred, thereby realizing the motion capture of the object. In some embodiments, the inference can be performed based on the correlation between the parts of the object, the relative relationship between the object part and the object as a whole, etc. Here, the correlation between the parts of the object, the relative relationship between the object part and the object as a whole, etc. can be obtained or set in advance, for example, before the information is obtained; or it can be obtained based on the global information, for example, it can be obtained by collecting the image or video of the object as a whole. The inference can be achieved by various appropriate inference techniques in the art, especially with the object motion inference technique, which will not be described in detail here.

[0057] In some embodiments of the present disclosure, the combination of local information and global information, especially the combination of local information of a specific part and global information may include a weighted combination of local information and global information, especially a weighted combination of local information of a specific part and global information. The weight may be appropriately set, and in particular, the weight may include a weight set based on at least one of the speed-related information of the specific part from the local information and the speed-related information of the specific part from the global information. Unless otherwise specified, the local information and global information described below may generally refer to information of a specific part.

[0058] In some embodiments, a weighted combination of spatial position information contained in or derivable from local information and global information may be included, and the weights may also be set in various appropriate ways. In particular, the weights may depend on motion-related information contained in or derivable from at least one of the local information and the global information. As an example, the weights may be set depending on motion-related information contained in or derivable from the global information. For example, the weights may be set based on the variance of motion-related information contained in or derivable from the global information, and in particular, may be set based on the variance of speed information contained in or derivable from the global information.

[0059] In some embodiments of the present disclosure, the weighted combination of local information and global information may also include a weighted combination of motion-related information contained in or derivable from the local information and the global information. For example, a weighted combination of speed information contained in or derivable from the local information and the global information may be included. In particular, for the combination of information at a specific moment, a weighted combination of at least one of the speed information at a specific moment contained in or derivable from the local information and the global information and the historical speed information may be included. As an example, the historical speed information may include the average speed in a specific time period adjacent to the specific moment, especially in a specific time period before the specific moment. In particular, the weights used for the weighted combination of speed information may be appropriately set, such as various empirical setting values, which will not be described in detail here.

[0060] According to some embodiments of the present disclosure, the speed-related information at a specific part in the local information includes the speed at a specific moment at a specific part of the object in the local information, and the average speed in a previous specific period, while the speed-related information at a specific part in the global information includes the speed at a specific moment at a specific part of the object in the global information, and the average speed in a previous specific period. Therefore, in some embodiments, the combination of the local information and the global information may include weighted combination of the speed information at a specific moment at a specific part in the local information, the spatial position information, and the average speed information in a specific period adjacent to the specific moment, and the speed information at a specific moment at a specific part in the global information, the spatial position information, and the average speed information in a specific period adjacent to the specific moment. In particular, it may include weighted combination of the speed information at a specific moment at a specific part in the local information, the spatial position information, the spatial position information at a specific moment at a specific part in the global information, and the difference between the average speed information in a specific period adjacent to the specific moment in the global information and the local information, wherein the weight may include a weight dependent on the speed information at a specific moment at a specific part in the global information.

[0061] In particular, considering that the speed-related information contained in or derivable from the global information may correspond to the inter-frame difference information between multiple image frames used to obtain the global information, such as the changes between the position information in the global information, the speed-related information contained in or derivable from the global information may correspond to the inter-group difference information between multiple groups of three-dimensional spatial information of the local information, and the combination based on the difference information, especially by utilizing the inter-frame information of the global information to compensate for the differences between the local three-dimensional spatial information, can effectively improve the accuracy and smoothness of the three-dimensional information of the object and the noise resistance.

[0062] In some embodiments of the present disclosure, depending on the formats of the global information and the local information, the combination of the global information and the local information may further include a transformation step, which includes performing a format transformation on at least one of the global information and the local information so that the information to be combined has the same or corresponding format or representation. Such a transformation step may also be referred to as a calibration step, i.e., calibrating both the global information and the local information, especially the spatial information therein, including at least one of the position information, motion-related information, etc.

[0063] In some embodiments, the information transformation may include transformation of at least one of the position information and the motion-related information contained in the information. Here, the aforementioned combination operation may be performed based on the transformed information, for example, the transformed position information may be combined based on the transformed motion-related information, especially the changed speed information.

[0064] For the implementation of information transformation, in some embodiments, at least one of the local information and the global information can be transformed into the same format, and / or transformed into the same space or coordinate system, and / or transformed into the same space or coordinate system to be expressed as the same format. As an example, when the global information is two-dimensional information, the local information can be transformed into three-dimensional information. Additionally or alternatively, when the local information and the global information have different coordinate systems, the global information can be transformed into the coordinate system of the local information, or both can be transformed into the same space or coordinate system. For example, in a virtual application space or coordinate system, both the global information and the local information can be transformed into the space or coordinate system, so as to be appropriately combined.

[0065] In some embodiments, as an example, the two-dimensional information of the global information can be integrated into vector information in its own coordinate system; the three-dimensional information of the local information can also be integrated into vector information in its own coordinate system; then the coordinate systems of the two are mapped or transformed to transform them into information in the same coordinate system, and then combined. Of course, they can also be mapped or transformed into other public coordinate systems and then combined.

[0066] In some embodiments according to the present disclosure, local information and global information can be converted to scalar information of their own coordinate systems, such as spatial positions, and then the coordinate system is transformed accordingly, for example, the coordinate system of the global information is transformed to the coordinate system of the local information, and the scalar is transformed accordingly, and then combined. In some embodiments according to the present disclosure, vectors of local information and global information can be converted to vectors in the same coordinate system, and then their respective scalars are obtained, and then combined.

[0067] It should be noted that the above transformation step can also be performed before the combination operation. In particular, the transformation operation can be performed after the global information and the local information are acquired as described above, and then the combination is performed based on the transformed information. Here, in some embodiments, the acquired information can also be considered to include the transformed information. For example, the acquired global information and local information respectively include the global information and the local information after the transformation, that is, the acquisition operation of the global information and the local information includes the respective transformation operations, and the transformation results are provided for combination.

[0068] In some embodiments of the present disclosure, the combination of global information and local information may also consider time information. In particular, the combination of global information and local information may be performed in a time-aligned manner, for example, the combination is performed based on global information and local information at the same time.

[0069] For example, the acquisition frequency or time interval of local information and global information may be the same or different. In particular, the frequency of the wearable device for acquiring local information may be the same or different from the frequency of the camera for capturing the image / video of the object. As another example, the local information may be acquired during the acquisition of global information, but at the same or different time intervals. In this way, when combining, the global information and local information at the same time should be selected for combination.

[0070] According to some embodiments of the present disclosure, in addition to combining the time-aligned local information and global information, other time-unaligned information may be processed in an appropriate manner. As an example, it may be discarded directly. As another example, it may be combined for combination. In some embodiments, information adjacent to the moment may be combined, such as information in a specific time period before and after the moment or information in a specific time period before the moment, to construct global information equivalent to the moment for combination. This is particularly suitable for situations where the global information may not contain information at the same time as the local information. For example, the time period may be appropriately set, such as based on historical experience, etc., which will not be described in detail here.

[0071] In some embodiments of the present disclosure, optionally, in the combination operation of local information and global information or after the combination operation, the historical posture related information of the object can also be corrected using the combination result to obtain improved object posture related information. As an example, the historical spatial position information of the object, such as the spatial position information of a specific part at the previous moment or the spatial position information obtained by the combination at the previous moment, can be corrected. In some embodiments, the combination result at the current moment can be weightedly combined with the historical posture related information at the previous moment, and the weight here can be set in various appropriate ways, for example, it can be set according to historical experience.

[0072] In an exemplary implementation, according to some embodiments of the present disclosure, speed information at a specific moment at a specific part of local information and global information, spatial position information at a specific moment, the difference between the average speed at the specific part in the local information in a previous specific time period and the average speed at the specific part in the global information in a previous specific time period are weighted combined, and the weighted combination result is further weighted combined with the position information at the previous moment at the specific part in the local information, especially the combined position information, to serve as the corrected position information at the specific part at the current moment.

[0073] Therefore, based on the combination of local information and global information, and further based on historical posture information, correction is performed, thereby further improving the accuracy and smoothness of the object posture information.

[0074] In the structural example of the above-mentioned device, the processing circuit 202 can be in the form of a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the processing circuit 202 can be constructed by a circuit (hardware) or a central processing device (such as a central processing unit (CPU)). In addition, the processing circuit 302 can carry a program (software) for making the circuit (hardware) or the central processing device work. The program can be stored in a memory (such as arranged in the memory) or an external storage medium connected from the outside, and downloaded via a network (such as the Internet).

[0075] According to an embodiment of the present disclosure, the processing circuit 202 may include various units for implementing the above-mentioned functions, for example, a first acquisition unit 204, configured to acquire local information related to the posture of a specific part of the motion capture object; a second acquisition unit 206, configured to acquire global information related to the posture of a specific part of the motion capture object; and a combination unit 208, configured to combine the local information and the global information to determine the posture information of the motion capture object.

[0076] In some embodiments, as an example, the first acquisition unit 204 may acquire data related to the posture of a specific part provided by a wearable device set at a specific part of the object, and optionally perform appropriate processing to serve as the local information. In some embodiments, as an example, the second acquisition unit 206 may acquire an object image / video from a shooting device, and extract / analyze it to obtain data related to the posture of a specific part as global information; or data related to the posture of a specific part may be extracted or analyzed by other devices from the object image / video of the shooting device, and provided to the second acquisition unit as global information.

[0077] According to an embodiment of the present disclosure, the combination unit 208 may be further configured to combine the local information and the global information of the specific part of the motion capture object to obtain posture-related information of the specific part, and determine the motion information of the motion capture object based on the posture-related information of the specific part of the motion capture object obtained by combination.

[0078] According to an embodiment of the present disclosure, before the information is combined, the global information and the local information may be appropriately transformed or calibrated. In some embodiments, the transformation or calibration may include information format transformation or calibration, and / or time alignment.

[0079] According to an embodiment of the present disclosure, such transformation or calibration may be implemented by a combination unit. In some embodiments, the combination unit 208 may be further configured to time-align the local information with the global information, and then combine the time-aligned local information and the global information. In particular, as an example, the combination unit 208 may include an alignment unit configured to time-align the local information with the global information.

[0080] In some embodiments, the combination unit or alignment unit may be further configured to, for local information at a specific moment, select global information at the same moment as the global information aligned with the local information for combination; and / or combine the global information in a specific time period adjacent to the specific moment, and then use the combination result as the global information aligned with the local information at the specific moment for combination.

[0081] In some embodiments, the combining unit 208 may be further configured to transform at least one of the local information and the global information so that the two are represented in the same format in the same coordinate system, and to combine the information based on the at least one of the transformed local information and the global information. In particular, as an example, the combining unit 208 may include a transform unit configured to transform at least one of the local information and the global information.

[0082] It should be noted that the above-mentioned transformation unit and / or alignment unit may also be located in the device 200 and outside the combination unit 208. Alternatively, they may even be implemented by any other suitable device or equipment outside the device 200. In this case, the transformed and / or aligned data may be acquired by the first acquisition unit and the second acquisition unit as local information and / or global information, respectively.

[0083] According to an embodiment of the present disclosure, the combining unit 208 may be further configured to perform various appropriate information combinations, for example, may include an information combining unit.

[0084] In some embodiments, the combination unit or the information combination unit may be configured to perform a weighted combination of the local information and the global information at a specific part of the object, wherein the weight includes a weight set based on at least one of the speed-related information of the specific part from the local information and the speed-related information of the specific part from the global information.

[0085] In some embodiments, the combination unit or the information combination unit may be configured to perform a weighted combination of the spatial position information contained in or derived from the local characteristic information at the characteristic part of the object and the spatial position information contained in or derived from the global information, wherein the weight is based on a variance-related value of the velocity-related information at a specific part in the global information.

[0086] In some embodiments, the combination unit or the information combination unit may be configured to perform a weighted combination of the speed-related information contained in or derived from the local characteristic information at the characteristic part of the object and the speed-related information contained in or derived from the global information.

[0087] In some embodiments, the combination unit or the information combination unit may be configured to perform a weighted combination of the speed information at a specific moment at a specific part of the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment, with the speed information at a specific moment at a specific part of the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.

[0088] According to some embodiments of the present disclosure, the combining unit 208 may be further configured to determine the object posture information based on the combined data. As an example, the combining unit 208 may include a determining unit for determining the object posture information based on the combined data.

[0089] In some embodiments, the combination unit 208 or the determination unit may be configured to infer information about the overall posture of the object based on the correlation between the various parts of the object and / or the various parts of the object, based on the posture-related information of the specific parts of the combined motion capture object.

[0090] In some embodiments, the combination unit 208 or the determination unit may be configured to use the combination result of the global information and local information obtained at a specific moment to correct the position information of the specific part of the object in the local information at the previous moment of the specific moment, as the position information of the specific part of the object in the corrected local information.

[0091] In some embodiments, the combination unit 208 or the determination unit may be configured to perform a weighted combination of the combination result of the global information and the local information at a specific part at a specific moment with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific part at the specific moment from the local information and the speed-related information at the specific part at the specific moment from the global information.

[0092] Alternatively, the determination of the object posture information may also be performed outside the combination unit. For example, the determination unit or correction unit may also be located in the device 200 and outside the combination unit 208, or even outside the device 200, and implemented by any other suitable device or equipment.

[0093] It should be noted that although Figure 2 In the figure, each unit is shown as a discrete unit, but one or more of these units may be combined into one unit, or split into multiple units. In addition, some units may not be included in the processing circuit or even the receiving side device, so they may be shown with dotted lines. As an example, at least one of the transform unit 210 and the alignment unit, or even both, may even be outside the processing circuit 202, so they may also be shown with dotted lines. In addition, as another implementation, the alignment unit may also be included in the transform unit, or the transform unit and the alignment unit may be included together in the calibration unit.

[0094] It should be noted that the above-mentioned units are only logical modules divided according to the specific functions implemented by them, and are not used to limit the specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations / functions implemented by them can be implemented by the processing circuit itself.

[0095] It should be understood that Figure 2It is only a schematic structural configuration of an electronic device for motion capture according to an embodiment of the present disclosure. The device 200 may also include other possible components, such as memory, network interface, controller, communication unit, etc., which are not shown for clarity. In particular, the processing circuit may be associated with the memory. For example, the processing circuit may be directly or indirectly (for example, other components may be connected in the middle) connected to the memory to access image processing related data. The memory may store various data and / or information generated by the processing circuit 202. The memory may also be located in the optimization device but outside the processing circuit, or even outside the optimization device. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory.

[0096] The following will refer to Figure 3A 1 is a flowchart of an object motion capture method according to an embodiment of the present disclosure. In method 300, in step S301 (first acquisition step), local information related to the posture of a specific part of the motion capture object is acquired; in step S302 (second acquisition step), global information related to the posture of a specific part of the motion capture object is acquired; in step S303 (combination step), the local information and the global information are combined to determine the posture information of the motion capture object.

[0097] It should be noted that the order shown in the first acquisition step and the second acquisition step is only to illustrate that the method includes these two steps, and does not limit the execution order of the two steps. Without limitation, the two acquisition steps can be executed in parallel, for example, at respective frequencies and intervals. In some embodiments, generally, the acquisition frequency of local information in the first acquisition step can be equal to or higher than the acquisition frequency of global information in the second acquisition step, that is, compared with local information, global information (e.g., image / video frames used to extract or analyze global information) can be acquired at equal or greater time intervals.

[0098] According to an embodiment of the present disclosure, each step in method 300 may be performed as described above, and in particular, various appropriate operations described herein may be performed. In particular, Figure 3BA flowchart of an exemplary implementation of the combination step according to an embodiment of the present disclosure is shown. In step S313 (transformation step), the transformation / calibration described in the text may be performed, including format conversion, time alignment, etc., in step S323 (information combination step), the combination operations of various global information and local information described in the text may be performed, and in step S333 (inference step), the inference operations of the object posture information and motion information described in the text may be performed, and in particular, may include correction operations. It will not be described in detail here. It should be noted that these steps may also be outside the method of the present disclosure.

[0099] It should be noted that these steps can be performed by any appropriate device or device element, such as the aforementioned receiving side device, the processing circuit in the receiving side device, the corresponding element in the processing circuit, etc. It should be noted that the audio presentation method according to the embodiment of the present disclosure may also include other steps, such as the various further processing described above. Moreover, these further processing can also be performed by appropriate devices or device elements, which will not be described in detail here.

[0100] According to an embodiment of the present disclosure, a motion capture system is also provided. Figure 4A As shown, the system 400 includes a camera device 401, which can be configured to capture an image or video of an object or a specific part of the object. As an example, the camera device 401 may include a camera device separated from the object, such as a camera, etc.; a wearable device 402, which can be configured at the object or a specific part of the object and obtain three-dimensional spatial information of a specific position. As an example, the wearable device 402 may include a device based on an inertial sensor; and a processing device 403, which can be configured to receive information provided by the camera device 401 and the wearable device 402, and determine the posture information of the object based on the received information. In particular, the information from the camera device 401 may correspond to the aforementioned global information, which may be a graphic / video captured by the camera device 401, and the processor analyzes or extracts the global information therefrom, or the global information extracted or analyzed by the camera device 401 itself or analyzed by other devices; the information provided by the wearable device 402 may correspond to the aforementioned local information, so that the processing device 403 may correspond to the aforementioned electronic device for motion capture, and perform the operations described above.

[0101] Figure 4BA conceptual flow chart of object motion capture according to an embodiment of the present disclosure is shown. On the one hand, an image or video of the object or a specific part of the object is captured by a camera device that is separated from the object, thereby obtaining global information, and on the other hand, local information is obtained by a wearable device configured on the object or the specific part of the object, and then, optionally, at least one of the global information and the local information is transformed, and then, the global information and the local information, or the optionally transformed global information and the local information, are combined by a processing device to obtain the posture information of the object.

[0102] The following will describe an exemplary implementation of a motion capture system according to an embodiment of the present disclosure with reference to the accompanying drawings. In the following description, the human body is described as an example of a motion capture object, but it should be noted that the motion capture scheme of the present disclosure can be equally applied to other objects, especially other movable objects, such as other objects held by or associated with the human body, other movable organisms, etc.

[0103] Figure 5 A dynamic capture system according to some embodiments of the present disclosure is shown as an example. The system includes: a camera for collecting multiple image frames containing two-dimensional image information of the wearer's limbs, in particular, the camera can collect continuous image frames; multiple wearable devices for collecting multiple groups of three-dimensional spatial information of key parts of the wearer's limbs within the time of the multiple image frames; a processor for receiving the three-dimensional spatial information and the multiple image frames, and compensating the inter-group differences of the multiple groups of three-dimensional spatial information using the inter-frame information between the multiple image frames to obtain three-dimensional spatial information with smooth transition between groups.

[0104] The aforementioned camera is preferably an ordinary webcam, or it can be a mobile phone camera, etc. It can communicate with the outside world through data line communication, integrated data transmission (such as a mobile phone and its camera), or through wireless network transmission.

[0105] Figure 5Six wearable devices that can be worn on the human body are shown. The wearable devices can be worn on the six joints of the human body, preferably the head, waist, wrists, and ankles. It should be noted that this is only exemplary, and the position can be increased, decreased, and adjusted as needed. The wearable device is preferably a device including an inertial sensor (IMU) and a communication unit and can send three-dimensional space information to the outside world, such as motion acceleration, angular velocity, and speed, displacement, and other information calculated therefrom. As an example, the appearance style of the wearable device is preferably lightweight, and a high-convenience style such as a bracelet, an anklet, a hairpin, etc. meets the requirements of the human body for burden-free wearing. The internal data acquisition and calculation processing is preferably a known inertial sensor data processing method. The method of communication with the outside world and data transmission is preferably a communication method based on the Bluetooth communication protocol, or it can be a data transmission method based on a WIFI wireless network.

[0106] The aforementioned processor is preferably a device with a computing unit such as a computer or a mobile phone, which can process the received three-dimensional spatial information and multiple image frames to determine the movement information of the human body. Such processing may correspond to the aforementioned combination step. Preferably, the processing steps of the processor may include: a preprocessing step, an information combination step, and a post-processing step, which correspond to the aforementioned transformation step, information combination step, and inference step, respectively.

[0107] Figure 6 The schematic implementation of the combined processing, especially the preprocessing step, in the dynamic capture system according to the embodiment of the present disclosure is shown. In this step, the processor receives continuous image frames or video frames from the camera, detects six joints of the human body frame by frame, and outputs the two-dimensional spatial information of the joints, including two-dimensional position information, motion-related information, etc. The human joint detection algorithm is preferably a joint detection algorithm based on a deep learning network, or it can be an algorithm based on traditional image processing technology. Note that the above is only an example. In other implementations of the present disclosure, the camera can also directly detect the human joints through edge computing, and only send the two-dimensional spatial information of the human joints in each frame to the processor.

[0108] In addition, the processor also receives motion information of six joints of the human body sent by six wearable devices, such as acceleration and angular velocity information per hertz, and calculates the three-dimensional position information of the six wearable devices projected in a unified virtual space, and then obtains the three-dimensional spatial information of the six joints. The position information calculation is preferably the inertial sensor integral calculation in the prior art, and the projection calculation of multiple inertial sensors in the virtual space is preferably a position prediction calculation based on a deep learning network. Note that the above is only an example. In other implementations of the present disclosure, the six wearable devices can also directly perform position calculations of the human joints in a unified virtual space through edge computing, thereby obtaining the spatial information of each joint, including position information, motion-related information, etc., and only send the spatial information of the human joints per hertz to the processor.

[0109] The information combination step includes a spatial calibration step and a fusion smoothing step. The spatial calibration step preferably corresponds the two-dimensional human joint point position of the camera to the three-dimensional human joint point position in the virtual space, and the two-dimensional spatial position of the camera is also correspondingly expanded to a three-dimensional spatial position. Such a calibration operation can be performed in various appropriate ways in the art, for example, based on specific constraints.

[0110] The fusion smoothing step is preferably a fusion method based on the following formulas (1) and (2) for each joint point.

[0111]

[0112] K t =tanh(VAR(V CAM )) (2)

[0113] in, K is the fused 3D position of each joint point at the current time t and the previous time t-1, t is the dynamic fusion weight, X CAM , X IMU are the 3D position from the camera and the 3D position from the wearable device at the current moment after spatial calibration, V CAM ,V IMU , The speeds from the camera and the wearable device at the current moment after spatial calibration, and the average speeds from the camera at the previous moments and the average speeds from the wearable device at the previous moments are respectively. The speed is preferably obtained by using the difference between the position information at the current moment and the position information at the previous moment. VAR is the variance calculation function, tanh is the inverse tangent function, and α, β, γ are weight coefficients. The dynamic fusion weight is calculated and dynamic fusion is performed using the relevant speed information obtained by the camera and the relevant speed information obtained by the wearable device. The α weight is a smoothing weight, which is preferably a fixed coefficient, or a dynamic coefficient.

[0114] It should be noted that this is only exemplary. In other implementations of the present disclosure, for example It can also be used as the result of the combination operation without further correction of the historical combination information, that is, no longer as in the above formula Further corrections.

[0115] The post-processing step is to update the original joint point positions obtained from multiple wearable devices using the fused and smoothed joint point positions, and to infer the motion information of all human joint points using the positions of preferably 6 human joint points to drive the human virtual model to move in the virtual three-dimensional space. All human joint points are preferably all joint points in the human kinematic chain including shoulders, necks, necks, knees, etc., and the motion information of the joint points is preferably a motion matrix.

[0116] The position information provided by the wearable device after fusion and smooth update can more accurately simulate the position of human body movement, thereby achieving high-precision human motion capture.

[0117] In this way, the camera is greatly disturbed by external conditions, and the wearer's limb information obtained between consecutive frames has high accuracy in the global range but contains a lot of local noise. Wearable devices have the problem of low local noise but low global three-dimensional information accuracy. The inter-frame information between multiple image frames compensates the inter-group differences of the multiple groups of three-dimensional spatial information, which can effectively improve the accuracy, smoothness and noise resistance of the wearer's limb three-dimensional information.

[0118] According to an embodiment of the present disclosure, the improved object posture / motion information can be further processed or applied in various appropriate ways. For example, it can be presented to a viewer through an appropriate presentation device such as various types of display devices, or it can be provided to a processing device for further processing to be applied to various appropriate scenarios, such as fitting into a virtual object in the production of movies, animations, etc., performing video / audio interaction, etc. As an example, the obtained object motion information can be applied to an appropriate interactive application, where interaction can be performed based on the posture of the object.

[0119] As an example, in an interactive application, the user posture information can be accurately obtained by accurately detecting the user's actions, and then the user's reaction, emotion, etc. can be judged based on the user posture information, and then the presented content can be adaptively adjusted. For example, the emotion and reaction corresponding to the user's posture / action can be pre-set, and then the user's reaction, emotion, etc. can be judged based on the detected user posture, and the content can be adaptively adjusted. As an example, in an application scenario of audio interaction, the user's reaction to the audio can be judged based on the user's posture, so as to adjust the audio content. For example, in some embodiments, the adjustment includes at least one of the following: increasing or decreasing the volume of the audio content; adjusting the rhythm of the audio content; enhancing the effect of the audio content; adding additional effects to the audio content. It should be noted that the motion capture scheme according to the embodiment of the present disclosure can also be effectively applied to other applications.

[0120] In some embodiments, the scheme of the present disclosure can be implemented by a software algorithm, so that it can be conveniently integrated into various types of devices including presentation devices, such as devices including various presentation devices, such as finger sleeves. In particular, the method of the present disclosure can be executed as a computer program, instructions, etc. by a processor of a portable device to capture object motion, such as human motion capture and fitting, for various appropriate applications.

[0121] In addition, it should be understood that the above series of processes and devices can also be implemented by software and / or firmware. In the case of being implemented by software and / or firmware, from a storage medium or a network to a computer with a dedicated hardware structure, such as Figure 7 The general-purpose personal computer 700 shown installs the programs constituting the software, and when the various programs are installed, the computer can execute various functions and the like. Figure 7 1 is a block diagram showing an example structure of a personal computer of an optimization device that can be adopted in an embodiment of the present disclosure. In one example, the personal computer can correspond to the above-mentioned exemplary optimization device according to the present disclosure.

[0122] exist Figure 7 In the embodiment, a central processing unit (CPU) 701 executes various processes according to a program stored in a read only memory (ROM) 702 or a program loaded from a storage section 708 to a random access memory (RAM) 703. In the RAM 703, data required when the CPU 701 executes various processes and the like is also stored as needed.

[0123] The CPU 701, the ROM 702, and the RAM 703 are connected to one another via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0124] The following components are connected to the input / output interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet.

[0125] A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0126] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 711 .

[0127] Those skilled in the art should understand that such storage media is not limited to Figure 7 The removable medium 711 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 711 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 702, a hard disk included in the storage portion 708, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0128] It should be noted that the methods and devices described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application specific integrated circuit.

[0129] In addition, the method and system of the present disclosure may be implemented in a variety of ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination thereof. The order of the steps of the method described above is illustrative only, and unless otherwise specifically stated, the steps of the method of the present disclosure are not limited to the order specifically described above. In addition, in some embodiments, the present disclosure may also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also encompasses a recording medium storing a program for implementing the method according to the present disclosure. Such storage media may include, but are not limited to, a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0130] Those skilled in the art will appreciate that the boundaries between the above operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of specific operations, and the order of operations may be changed in other various embodiments. However, other modifications, variations, and replacements are equally possible. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.

[0131] Additionally, embodiments of the present disclosure may also include the following exemplary embodiment implementations (EEE).

[0132] EEE 1. An electronic device for motion capture, the device comprising a processing circuit configured to:

[0133] Acquire local information related to the posture of a specific part of the motion capture object;

[0134] Obtaining global information related to the posture of specific parts of the motion capture object;

[0135] The local information and the global information are combined to determine the posture information of the motion capture object.

[0136] EEE 2. An electronic device according to EEE 1, wherein the local information includes first spatial information related to the posture of the specific part expressed in a local space or a local coordinate system, which may include at least one of the three-dimensional spatial position information and motion-related information of the specific part.

[0137] EEE 3. The electronic device according to EEE 1, wherein the motion-related information of the specific part includes at least one of motion acceleration, angular velocity, speed, and displacement of the specific part.

[0138] EEE 4. The electronic device according to EEE 1, wherein the global information includes second spatial information related to the posture of the specific part expressed in a global space or a global coordinate system.

[0139] EEE 5. The electronic device according to EEE 4, wherein:

[0140] The second spatial information is obtained by analyzing or extracting from an overall image and / or video of the motion capture object, or by analyzing or extracting from an image and / or video of a specific part of the motion capture object.

[0141] EEE 6. The electronic device according to EEE 4, wherein:

[0142] The second spatial information includes:

[0143] The two-dimensional spatial position information and motion-related information of the specific part obtained by the camera device, and / or

[0144] Three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information.

[0145] EEE 7. The electronic device according to EEE 6, wherein the three-dimensional spatial position information and motion-related information converted based on the two-dimensional position information are in the same format as the first spatial information or in the same coordinate system.

[0146] EEE 8. The electronic device according to EEE 1, wherein the local information and the global information are combined to determine the posture information of the motion capture object, comprising:

[0147] combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and

[0148] Based on the combined posture-related information of the specific part of the motion capture object, the motion information of the motion capture object is determined.

[0149] EEE 9. The device according to EEE 8, wherein, based on the correlation between various parts of the object and / or the various parts of the object, information about the overall posture of the object is inferred based on the posture-related information of specific parts of the motion capture object obtained by combination, as motion information.

[0150] EEE 10. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:

[0151] The local information and the global information are time-aligned, and then

[0152] The time-aligned local information and the global information are combined.

[0153] EEE 11. The device according to EEE 10, wherein time aligning the local information with the global information comprises: for the local information at a specific moment,

[0154] Selecting global information at the same time instant as being aligned with local information for combination; and / or

[0155] The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.

[0156] EEE 12. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:

[0157] transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and

[0158] Information combining is performed based on the at least one of the transformed local information and the global information.

[0159] EEE 13. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:

[0160] The local information and the global information at the specific part of the object are weightedly combined, wherein the weight includes a weight set based on at least one of the velocity-related information of the specific part from the local information and the velocity-related information of the specific part from the global information.

[0161] EEE 14. The electronic device according to EEE 13, wherein weighted combination of the local information at a specific part of the object and the global information comprises:

[0162] The spatial position information contained in or derived from the local characteristic information at the characteristic part of the object is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on a variance correlation value of the speed-related information at a specific part in the global information.

[0163] EEE 15. The electronic device according to EEE 13 or 14, wherein weighted combination of the local information at a specific part of the object and the global information comprises:

[0164] The speed-related information contained in or obtained in the local characteristic information at the characteristic part of the object and the speed-related information contained in or obtained in the global information are weightedly combined.

[0165] EEE 16. The apparatus according to any one of EEE 13-15, wherein:

[0166] The speed-related information at the specific part in the local information includes the speed at the specific part of the object in the local information at a specific moment and the average speed in the previous specific period;

[0167] The speed-related information at the specific part in the global information includes the speed at the specific part of the object in the global information at a specific moment and the average speed in the previous specific period.

[0168] EEE 17. The apparatus according to EEE 13, wherein combining the local information and the global information comprises:

[0169] The speed information at a specific moment at a specific part in the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment are weightedly combined with the speed information at a specific moment at a specific part in the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.

[0170] EEE 18. The apparatus according to EEE 1, wherein combining the local information and the global information further comprises:

[0171] The combination result of the global information and the local information obtained at a specific moment is used to correct the position information of the specific part of the object in the local information at the previous moment of the specific moment, and use it as the position information of the specific part of the object in the corrected local information.

[0172] EEE 19. The apparatus according to EEE 18, wherein the modification comprises

[0173] The combination result of the global information and the local information at a specific part at a specific moment is weightedly combined with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific part at the specific moment from the local information and the speed-related information at the specific part at the specific moment from the global information.

[0174] EEE 20. The device according to EEE 1, wherein the local information is acquired during the acquisition of the global information, and / or the frequency of acquiring the local information is the same as or different from the frequency of acquiring the global information.

[0175] EEE 21. The device according to EEE 1, wherein the specific part comprises a representative part or a basic part that can characterize or derive the motion posture of the object.

[0176] EEE 22. The device according to EEE 1, wherein the object is a person whose motion is captured, and the specific parts include basic joints of the human body, including at least one of the head, waist, wrists, ankles, etc.

[0177] EEE 23. A method for motion capture, comprising:

[0178] Acquire local information related to the posture of a specific part of the motion capture object;

[0179] Obtaining global information related to the posture of specific parts of the motion capture object;

[0180] The local information and the global information are combined to determine the posture information of the motion capture object.

[0181] EEE 24. The method according to EEE 23, wherein the local information and the global information are combined to determine the posture information of the motion capture object, comprising:

[0182] combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and

[0183] Based on the combined posture-related information of the specific part of the motion capture object, the motion information of the motion capture object is determined.

[0184] EEE 25. The method according to EEE 23, wherein combining the local information and the global information comprises:

[0185] The local information and the global information are time-aligned, and then

[0186] The time-aligned local information and the global information are combined.

[0187] EEE 26. The method according to EEE 25, wherein time aligning the local information with the global information comprises: for the local information at a specific moment,

[0188] Selecting global information at the same time instant as being aligned with local information for combination; and / or

[0189] The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.

[0190] EEE 27. The method according to EEE 23, wherein combining the local information and the global information comprises:

[0191] transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and

[0192] Information combining is performed based on the at least one of the transformed local information and the global information.

[0193] EEE 28. The method according to EEE 23, wherein combining the local information and the global information comprises:

[0194] The local information and the global information at the specific part of the object are weightedly combined, wherein the weight includes a weight set based on at least one of the velocity-related information of the specific part from the local information and the velocity-related information of the specific part from the global information.

[0195] EEE 29. The method according to EEE 28, wherein weighted combination of the local information and the global information at a specific part of the object comprises:

[0196] The spatial position information contained in or derived from the local characteristic information at the characteristic part of the object is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on a variance correlation value of the speed-related information at a specific part in the global information.

[0197] EEE 30. The method according to EEE 23, wherein weighted combination of the local information and the global information at a specific part of the object comprises:

[0198] The speed-related information contained in or obtained in the local characteristic information at the characteristic part of the object and the speed-related information contained in or obtained in the global information are weightedly combined.

[0199] EEE 31. The method according to EEE 23, wherein combining the local information and the global information comprises:

[0200] The speed information at a specific moment at a specific part in the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment are weightedly combined with the speed information at a specific moment at a specific part in the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.

[0201] EEE 32. The method according to EEE 23, wherein combining the local information and the global information further comprises:

[0202] The combination result of the global information and the local information obtained at a specific moment is used to correct the position information of the specific part of the object in the local information at the previous moment of the specific moment, and use it as the position information of the specific part of the object in the corrected local information.

[0203] EEE 33. A method according to EEE 32, wherein the correction includes weighted combination of the combination result of global information and local information at a specific part at a specific moment with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific part at the specific moment from the local information and the speed-related information at the specific part at the specific moment from the global information.

[0204] EEE 34. A motion capture system comprising:

[0205] A wearable device, which is disposed on a specific part of a motion capture object and is configured to obtain local information related to a posture of the specific part of the motion capture object;

[0206] A camera device configured to obtain global information related to the posture of a specific part of the motion capture object;

[0207] A processor is configured to combine the local information and the global information to determine posture information of the motion capture object.

[0208] EEE 35. A device comprising at least one processor; and at least one storage device storing thereon instructions which, when executed by the at least one processor, cause the at least one processor to perform a method according to any one of EEE 23-33.

[0209] EEE 36. A storage medium storing instructions, which, when executed by a processor, enables the processor to perform the method according to any one of EEE23-33.

[0210] EEE 37. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of EEE 23-33.

[0211] EEE 38. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of EEE 23-33.

[0212] Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and transformations can be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. Moreover, the terms "including", "comprising" or any other variants of the embodiments of the present disclosure are intended to cover non-exclusive inclusions, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "including one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0213] Although some specific embodiments of the present disclosure have been described in detail, it should be understood by those skilled in the art that the above embodiments are merely illustrative and do not limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be combined, modified or replaced without departing from the scope and essence of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An electronic device for motion capture, the device comprising a processing circuit configured to: Acquire local information related to the posture of a specific part of the motion capture object; Obtaining global information related to the posture of specific parts of the motion capture object; The local information and the global information are combined to determine the posture information of the motion capture object.

2. The electronic device according to claim 1, wherein: The local information includes first spatial information related to the posture of the specific part expressed in a local space or a local coordinate system, which may include at least one of the three-dimensional spatial position information and motion-related information of the specific part.

3. The electronic device according to claim 1, wherein: The motion-related information of the specific part includes at least one of the motion acceleration, angular velocity, speed, and displacement of the specific part.

4. The electronic device according to claim 1, wherein: The global information includes second spatial information related to the posture of the specific part expressed in a global space or a global coordinate system.

5. The electronic device according to claim 4, wherein: The second spatial information is obtained by analyzing or extracting from an overall image and / or video of the motion capture object, or by analyzing or extracting from an image and / or video of a specific part of the motion capture object.

6. The electronic device according to claim 4, wherein: The second spatial information includes: The two-dimensional spatial position information and motion-related information of the specific part obtained by the camera device, and / or Three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information.

7. The electronic device according to claim 6, wherein: The three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information are in the same format as the first spatial information or in the same coordinate system.

8. The electronic device according to claim 1, wherein: Combining the local information with the global information to determine the posture information of the motion capture object includes: combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and Based on the combined posture-related information of the specific part of the motion capture object, the motion information of the motion capture object is determined.

9. The device according to claim 8, wherein: Based on the correlation between various parts in the object and / or various parts in the object, information about the overall posture of the object is inferred based on the posture-related information of the specific part of the motion capture object obtained by combination, as the motion information.

10. The electronic device according to claim 1, wherein: Combining the local information with the global information comprises: The local information and the global information are time-aligned, and then The time-aligned local information and the global information are combined.