Method and device for motion capture
By combining the local information obtained by the wearable device and the global information obtained by the camera device, the problem of the human real-time motion capture system in balancing accuracy and convenience is solved, and the accuracy and noise resistance of motion capture are improved.
Patent Information
- Application Number
- PCT/CN2024/092896
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-28
Smart Images

Figure CN2024092896_28112024_PF_FP_ABST
Abstract
Description
Motion capture method and device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims and is based on the priority of Chinese Patent Application No. 202310572499.2 filed on May 19, 2023, the entirety of which is incorporated herein by reference. Technical Field
[0003] The present disclosure relates to motion capture, and more particularly to a motion capture method and apparatus. Background Art
[0004] Motion capture is widely used in the field of image processing and analysis. As technology develops, motion capture can be used in more and more fields, such as computer animation, television, movies, video games, sports monitoring and medical rehabilitation. Usually, the user is equipped with appropriate sensors so that the action can be recognized through the information provided by the sensors.
[0005] Summary of the Invention
[0006] This Summary is provided to introduce the concepts of the present disclosure in a simplified form, which are described in detail later in the Detailed Description.
[0007] The present disclosure provides an improved motion capture solution.
[0008] In one aspect of the present disclosure, an electronic device for motion capture is provided, the device including a processing circuit configured to: obtain local information related to the posture of a specific part of a motion capture object; obtain global information related to the posture of a specific part of the motion capture object; and combine the local information and the global information to determine the posture information of the motion capture object.
[0009] In another aspect of the present disclosure, a motion capture method is provided, comprising: obtaining local information related to the posture of a specific part of a motion capture object; obtaining global information related to the posture of the specific part of the motion capture object; and combining the local information and the global information to determine the posture information of the motion capture object.
[0010] In yet another aspect of the present disclosure, a motion capture system is provided, comprising a wearable device, which is configured at a specific part of a motion capture object and is configured to obtain local information related to the posture of the specific part of the motion capture object; a camera device, which is configured to obtain global information related to the posture of the specific part of the motion capture object; and a processor, which is configured to combine the local information and the global information to determine the posture information of the motion capture object.
[0011] In yet another aspect of the present disclosure, a device is provided, comprising at least one processor and at least one storage device, wherein the at least one storage device stores instructions thereon, which, when executed by the at least one processor, enables the at least one processor to perform a method according to an embodiment of the present disclosure.
[0012] In still another aspect of the present disclosure, a storage medium storing instructions is provided. When the instructions are executed by a processor, the method according to an embodiment of the present disclosure may be performed.
[0013] In still another aspect of the present disclosure, a program product is provided. The program product includes instructions. When the instructions are executed by a processor, the processor can execute the method according to the embodiment of the present disclosure.
[0014] In still another aspect of the present disclosure, a computer program is provided. The computer program includes instructions. When the instructions are executed by a processor, the processor may execute the method according to the embodiment of the present disclosure.
[0015] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are intended to provide a further understanding of the present disclosure. Together with the following detailed description, each of the drawings is incorporated into and forms a part of this specification and is used to explain the present disclosure. It should be understood that the drawings described below relate only to some embodiments of the present disclosure and are not intended to limit the present disclosure.
[0017] FIG1 shows a conceptual diagram of a motion capture solution according to an embodiment of the present disclosure.
[0018] FIG2 shows a schematic block diagram of an electronic device for motion capture according to an embodiment of the present disclosure.
[0019] 3A and 3B show flowcharts of a motion capture method according to an embodiment of the present disclosure.
[0020] FIG4A shows a schematic block diagram of a motion capture system according to an embodiment of the present disclosure, and FIG4B shows a schematic operation diagram of the motion capture system according to an embodiment of the present disclosure.
[0021] FIG5 shows a schematic block diagram of a motion capture system according to an embodiment of the present disclosure.
[0022] FIG6 shows a schematic block diagram of motion capture preprocessing according to an embodiment of the present disclosure.
[0023] FIG7 illustrates a block diagram showing an exemplary hardware configuration of a computer system capable of implementing embodiments of the present disclosure.
[0024] It should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not necessarily drawn to scale. The same or similar reference numerals are used throughout the drawings to indicate the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. However, it should be understood that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The drawings and the following description of the embodiments are actually only illustrative and do not constitute any limitation on the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein.
[0026] In addition, when describing exemplary embodiments of the present disclosure below in conjunction with the accompanying drawings, for the sake of clarity and conciseness, not all features of the embodiments are described in the specification. It should be noted that in order to avoid obscuring the present disclosure with unnecessary details, only the processing steps and / or device structures that are closely related to the solutions according to the present disclosure are shown in the accompanying drawings, while other details that are not relevant to the present disclosure are omitted.
[0027] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.
[0028] In the present disclosure, the terms "first," "second," etc. are used merely to distinguish between elements or steps, but are not intended to indicate temporal order, priority, or importance.
[0029] With the development of the technology itself and the improvement of the technical level of related application fields, motion capture technology, especially real-time motion capture technology, has been used more and more widely. The inventors have found that real-time motion capture technology can capture the movement of an object by using a small number of cameras and / or wearable devices. Wearable devices can be worn on specific joints of a moving object, such as a specific person, to capture the movement of the object in real time as the object moves, and can be applied to various purposes. For example, motion capture technology has very broad application prospects in fields such as animation production, ergonomics research, simulation training, rehabilitation medicine, biomechanics research, human-computer interaction means, virtual reality systems, robot remote control, interactive games, etc.
[0030] However, the inventors have found that the current real-time human motion capture system has problems in balancing motion capture accuracy and convenience. Although reducing the use of wearable devices for motion capture can make users feel simple and convenient, it will also cause a significant reduction in the accuracy of the motion capture system. In particular, wearable devices are more focused on the information of various parts of the human body and cannot more accurately reflect the global information of the human body. Cameras that capture images and / or videos of the human body cannot accurately and comprehensively obtain high-accuracy human motion capture, and because the camera is easily affected by external environmental interference, it leads to excessive local errors.
[0031] Therefore, an improved motion capture solution is needed to improve the accuracy and / or convenience of motion capture.
[0032] In view of this, the present disclosure proposes an improved motion capture solution. In particular, the present disclosure proposes combining multiple information related to the object posture, especially combining local information and global information related to the object posture, so as to improve the accuracy and / or convenience of motion capture. It should be noted that the information related to the object posture can include various types of information. In some embodiments, it includes at least one of the spatial position information of the object or a specific part of the object, and motion-related information, wherein the motion-related information includes at least one of the motion direction, displacement, speed, and motion acceleration.
[0033] The following describes the embodiments of the present disclosure in detail with reference to exemplary examples. FIG1 shows a schematic conceptual diagram of a motion capture solution according to an embodiment of the present disclosure.
[0034] On the one hand, global information related to the posture of the object is obtained. Global information may refer to information that characterizes the posture of the object in a global sense, for example, it may include information that can characterize the posture of the object in the case of reflecting the relative relationship between a specific part of the object and the object as a whole, the situation in which the object is located, the environment, and space. In some embodiments, the global information can be obtained based on the image / video of the object captured by a camera device that is separated from the object. For example, it can be achieved through various appropriate feature extraction and analysis methods, and can be in various appropriate formats and represented by various appropriate forms. As an example, the global information may include two-dimensional information or three-dimensional information that characterizes the posture of the object or the posture of a specific part of the object. For example, it can be represented by a vector, a matrix, etc.
[0035] As an example, the camera device can be any appropriate image / video acquisition device, and can include various appropriate types of camera devices, such as cameras, mobile camera devices, etc., and can be set in various appropriate positions relative to the object. For example, the front, side, etc. of the object, as long as it is suitable for capturing the entire image of the object. In addition, one or more camera devices can be set relative to the object to obtain appropriate images / videos of the object. As an example, the camera device can be a camera set in front of the user, which can capture images of the entire body or a part of the user to obtain a two-dimensional image frame. As another example, the camera can also be set around the object to capture images of the entire body or a part of the user to obtain a three-dimensional image.
[0036] On the other hand, local information related to the object's posture is obtained. Local information may refer to information that represents the object's posture in a local sense, such as information related to the posture of only a specific part of the object, which is often determined based on the local conditions of the specific part. In some embodiments, the local information can be obtained based on a sensing device set at a specific part of the object, and can be in various appropriate formats and represented by various appropriate forms. As an example, global information can include three-dimensional spatial information that represents the object's posture or the posture of a specific part of the object. For example, it can be represented by a vector, a matrix, etc.
[0037] As an example, a sensing device can be a sensor-type device that can be placed on a specific part of a subject and sense the posture, motion, etc. of that specific part of the subject. This device can also be referred to as a wearable device. For example, the sensing device can be a device that performs inertial sensing, such as an inertial sensor. In some embodiments, an appropriate number of sensing devices, such as the same number of sensing devices as the specific parts, can be provided and each device can be worn on the specific part to obtain local information about each part.
[0038] Then, a combination of the global information and the local information can be performed to obtain improved action information. In a sense, the combination can be equivalent to modifying the local information by utilizing the global information. The combination can be performed in various suitable ways. In some embodiments, it can include a weighted combination of the global information and the local information.
[0039] This allows for improved information about the object's posture, more accurately reflecting the object's posture or motion. Furthermore, in addition to wearable sensing devices, a small number of camera devices can be used to obtain appropriate global information for information correction and optimization, making it easier to obtain more accurate information about the object's posture or motion.
[0040] Various exemplary implementations of motion capture solutions according to embodiments of the present disclosure will be described below with reference to the accompanying drawings. FIG2 illustrates a block diagram of an electronic device for motion capture according to an embodiment of the present disclosure. The electronic device 200 includes a processing circuit 202 configured to: obtain local information related to the posture of a specific part of a motion capture object; obtain global information related to the posture of a specific part of the motion capture object; and combine the local information and the global information to determine posture information of the motion capture object.
[0041] In some embodiments of the present disclosure, the specific part may include a part of the object appropriately selected or set from all parts of the object. Preferably, the specific part may include a representative part or a basic part that can characterize or derive the object's motion posture. In some embodiments, the object is a human body whose motion is being captured, and the specific part includes a basic or representative joint part of the human body, including at least one of the head, waist, wrist, ankle, etc.
[0042] Local information
[0043] In some embodiments of the present disclosure, local information related to the posture of a specific part of a motion capture object may indicate posture-related information of the specific part in a local sense, which may include posture-related information or spatial information of the object in a specific or local space / coordinate system, such as information representing the spatial position, motion state, etc. of the object in the specific or local space / coordinate system. As an example, the specific or local space / coordinate system may include a coordinate system set by a wearable device for obtaining local information, or other appropriate coordinate systems.
[0044] According to embodiments of the present disclosure, local information may be acquired in various suitable ways. For example, the local information may be obtained through a specific sensing device / wearable device. For example, the specific sensing device / wearable device may be an inertial sensor, a motion capture sensor, a gyroscope, or the like.
[0045] The local information may include various appropriate types of information and may be expressed in various appropriate forms. In some embodiments of the present disclosure, the local information may include or derive posture-related information of the specific part, which may also be referred to as first spatial information, which may include at least one of the part posture, spatial position, and motion-related information. As an example, the motion-related information may include the moving direction, moving speed, moving acceleration, moving frequency, displacement, and the like of the part. As an example, the local information may be in a three-dimensional form, referred to as three-dimensional spatial information, and include at least one of the three-dimensional spatial position information and motion-related information of the specific part. In particular, the motion-related information of the specific part includes at least one of the motion acceleration, angular velocity, velocity, and displacement of the specific part.
[0046] In some embodiments, additionally or alternatively, the local information may also include other information, such as an index indicating local information related to the posture of a specific part. The index may be set in any appropriate manner. For example, a unique index is set for each part, and the information of each part may be set as a sub-index according to a time sequence, for example, time may be used as a sub-index, or corresponding numerical values may be set as sub-indexes according to a time sequence, and so on. As an example, the local information related to the posture of a specific part and the index may be pre-associated and stored, and the corresponding local information may be called according to the index during the application process. In some embodiments, the local information may include only the index, only posture related information, both the index and posture related information, and so on.
[0047] In still other embodiments, the local pose information may be subjected to specific processing, such as filtering, denoising, and the like.
[0048] Global Information
[0049] In some embodiments of the present disclosure, global information related to the posture of a specific part of a motion capture object may indicate information that represents the posture of the object in a global sense, which may include posture-related information or spatial information of the object in a specific or global space / coordinate system, for example, information that may represent / reflect the spatial position, motion-related information, etc. of a specific part in a specific or global space / coordinate system. In particular, the specific or global space / coordinate system may include a coordinate system set by a camera device for acquiring global information, a coordinate system of the space in which the object is located, or other appropriate coordinate systems. It should be noted that the specific or global space / coordinate system is often different from the aforementioned local space / coordinate system.
[0050] According to some embodiments of the present disclosure, global information can be derived based on image frames of the object captured by a camera device in the scene or scenario in which the object is located. Global information can be derived through various appropriate methods. In some embodiments, an image / video of the entire object can be captured, and then relevant information about the posture of a specific part can be analyzed or extracted therefrom. The information extraction here can be achieved by methods known in the art and will not be described in detail here. In some embodiments, a specific part of the object can be directly captured to obtain relevant information about its posture.
[0051] In some examples, the user's movements can be acquired through a camera / camera, and then the user's posture analysis can be performed from the acquired images or videos of the user's movements to obtain information / data related to the user's posture. In other examples, a specific color or a specific label can be set on a specific part of the user, and then the movement of the corresponding part can be acquired through camera color recognition. For example, a patch of a specific color can be attached to at least one finger of the user, and then the information / data related to the user's finger posture / movement can be captured through camera color recognition. In still other examples, a projector or other device can be used to detect the motion state of the contour, skeleton, etc. of a specific part through a specific algorithm, so as to acquire the posture of the corresponding part through the movement of the overall contour, skeleton, etc. of the part.
[0052] According to some embodiments of the present disclosure, global information may include various appropriate types of information and may be expressed in various appropriate forms. In particular, similar to local information, the global information may also include or derive information related to the posture of the specific part, which may also be referred to as second spatial information. The posture information may include at least one of the part's posture, spatial position, and motion-related information. In some embodiments, the global information may be two-dimensional information or three-dimensional information.
[0053] In some embodiments, the global information may be two-dimensional information, particularly when only a single image / video camera is used. For example, when only a single image / video camera is used, the camera may be positioned in front of an object to obtain a two-dimensional image / video of the object, from which the two-dimensional global information is analyzed.
[0054] In some embodiments, the global information can be three-dimensional information. As an example, when multiple cameras are positioned around an object, a three-dimensional image of the object can be obtained or combined, and analyzed to obtain three-dimensional global information. As another example, even when a single image / video camera is used, two-dimensional information can be analyzed and then expanded to obtain three-dimensional information. The analysis, extraction, or expansion of three-dimensional information can be achieved using various suitable methods known in the art and will not be described in detail here.
[0055] In some embodiments, similar to the local information, the global information may also include corresponding indexes and / or be subjected to various appropriate preprocessing, which will not be described in detail here.
[0056] Information combination
[0057] In the present disclosure, local information and global information can be combined in various appropriate ways, so that the pose information or motion information of the object can be determined based on the combined result. In particular, when both global information and local information contain various types of information, the combined result can include a combination of at least one corresponding type of information from the global information and the local information. As an example, the combined result can include a combination of at least one of the corresponding spatial position information and motion-related information from the global information and the local information.
[0058] In some embodiments, local information and global information of a specific part of the object can be combined, and then object pose-related information can be derived based on the combined result. In particular, local information and global information of a specific part of the object can be combined to obtain pose-related information of the specific part. Based on the combined pose-related information of the specific part of the motion capture object, pose information or motion information of the object as a whole can be inferred, thereby achieving motion capture of the object. In some embodiments, the inference can be performed based on the correlation between various parts of the object, the relative relationship between the object parts and the object as a whole, etc. Here, the correlation between various parts of the object, the relative relationship between the object parts and the object as a whole, etc. can be obtained or set in advance, for example, before information acquisition; or can be obtained based on global information, for example, from a captured image or video of the object as a whole. The inference can be achieved using various appropriate inference techniques known in the art, especially object motion inference techniques, which will not be described in detail here.
[0059] In some embodiments of the present disclosure, the combination of local information and global information, in particular the combination of local information of a specific part and global information, may be implemented by a weighted combination of local information and global information, in particular the weighted combination of local information of a specific part and global information. The weights may be appropriately set, and in particular, the weights may include weights set based on at least one of the velocity-related information of the specific part from the local information and the velocity-related information of the specific part from the global information. Unless otherwise specified, the local information and global information described below may generally refer to information of the specific part.
[0060] In some embodiments, a weighted combination of spatial position information contained in or derivable from the local information and the global information may be included, and the weights may be set in various appropriate ways. In particular, the weights may depend on motion-related information contained in or derivable from at least one of the local information and the global information. As an example, the weights may be set based on motion-related information contained in or derivable from the global information. For example, the weights may be set based on the variance of motion-related information contained in or derivable from the global information, and in particular, may be set based on the variance of velocity information contained in or derivable from the global information.
[0061] In some embodiments of the present disclosure, the weighted combination of local information and global information may further include a weighted combination of motion-related information contained in or derivable from the local information and global information. For example, a weighted combination of speed information contained in or derivable from the local information and global information may be included. In particular, for the combination of information at a specific moment, a weighted combination of at least one of the speed information at a specific moment contained in or derivable from the local information and global information and historical speed information may be included. As an example, the historical speed information may include the average speed within a specific time period adjacent to the specific moment, especially within a specific time period before the specific moment. In particular, the weights used for the weighted combination of speed information may be appropriately set, such as various empirical setting values, which will not be described in detail here.
[0062] According to some embodiments of the present disclosure, the speed-related information at a specific location in the local information includes the speed of the specific location at a specific moment in the local information and the average speed over a previous specific time period, while the speed-related information at a specific location in the global information includes the speed of the specific location at a specific moment in the global information and the average speed over a previous specific time period. Therefore, in some embodiments, combining the local information and the global information may include weightedly combining the speed information, spatial position information, and average speed information over a specific time period adjacent to the specific moment at the specific location in the local information with the speed information, spatial position information, and average speed information over a specific time period adjacent to the specific moment at the specific location in the global information. Specifically, this may include weightedly combining the speed information, spatial position information, and average speed information over a specific time period adjacent to the specific moment at the specific location in the local information and the spatial position information of the specific location at a specific moment in the global information, as well as the difference between the average speed information over a specific time period adjacent to the specific moment in the global information and the local information, wherein the weight may include a weight dependent on the speed information at the specific location at a specific moment in the global information.
[0063] In particular, considering that the speed-related information contained in or derivable by the global information may correspond to the inter-frame difference information between multiple image frames used to obtain the global information, such as the changes between the position information in the global information, the speed-related information contained in or derivable by the global information may correspond to the inter-group difference information between multiple groups of three-dimensional spatial information of the local information, and the combination based on the difference information, especially by utilizing the inter-frame information of the global information to compensate for the differences between the local three-dimensional spatial information, can effectively improve the accuracy and smoothness of the three-dimensional information of the object and the noise resistance.
[0064] In some embodiments of the present disclosure, depending on the respective formats of the global information and the local information, the combination of the global information and the local information may further include a transformation step, wherein the transformation step includes performing a format transformation on at least one of the global information and the local information so that the combined information has the same or corresponding format or representation. Such a transformation step may also be referred to as a calibration step, i.e., calibrating both the global information and the local information, and in particular calibrating the spatial information therein, including at least one of position information and motion-related information.
[0065] In some embodiments, the information transformation may include a transformation of at least one of the position information and the motion-related information contained in the information. Here, the aforementioned combination operation may be performed based on the transformed information, for example, the transformed position information may be combined based on the transformed motion-related information, especially the changed speed information.
[0066] Regarding the implementation of information transformation, in some embodiments, at least one of the local information and the global information can be transformed into the same format, and / or transformed into the same space or coordinate system, and / or transformed into the same space or coordinate system and expressed as the same format. As an example, when the global information is two-dimensional information, the local information can be transformed into three-dimensional information. Additionally or alternatively, when the local information and the global information have different coordinate systems, the global information can be transformed into the coordinate system of the local information, or both can be transformed into the same space or coordinate system. For example, in a virtual application space or coordinate system, both the global information and the local information can be transformed into the space or coordinate system, so as to be appropriately combined.
[0067] In some embodiments, for example, the two-dimensional information of the global information can be integrated into vector information in its own coordinate system; the three-dimensional information of the local information can also be integrated into vector information in its own coordinate system; the coordinate systems of the two can then be mapped or transformed to transform them into information in the same coordinate system for combination. Of course, mapping or transformation to other common coordinate systems can also be used for combination.
[0068] In some embodiments of the present disclosure, local information and global information can be converted to scalar information in their own coordinate systems, such as spatial positions, and then a coordinate system corresponding transformation can be performed. For example, the coordinate system of the global information is transformed to the coordinate system of the local information, and the scalars are transformed accordingly, and then combined. In some embodiments of the present disclosure, the vectors of the local information and the global information can be converted to vectors in the same coordinate system, and then the respective scalars are obtained, and then combined.
[0069] It should be noted that the above-mentioned transformation step can also be performed before the combination operation. In particular, the transformation operation can be performed after obtaining the global information and local information as described above, and then the combination is performed based on the transformed information. Here, in some embodiments, it can also be considered that the acquired information includes transformed information. For example, the acquired global information and local information respectively include the global information and local information after transformation. In other words, the acquisition operation of the global information and the local information includes the respective transformation operations, and the transformation results are provided for combination.
[0070] In some embodiments of the present disclosure, the combination of global information and local information may also consider time information. In particular, the combination of global information and local information may be performed in a time-aligned manner, for example, combining global information and local information at the same time.
[0071] For example, the acquisition frequency or time interval of local information and global information may be the same or different. In particular, the frequency of the wearable device used to acquire local information may be the same as or different from the frequency of the camera used to capture the image / video of the object. As another example, local information may be acquired during the acquisition of global information, but at the same or different time intervals. In this way, when combining, global information and local information at the same time should be selected for combination.
[0072] According to some embodiments of the present disclosure, in addition to combining the time-aligned local information and global information, other time-unaligned information may be processed in an appropriate manner. As an example, it may be discarded directly. As another example, it may be combined for combination. In some embodiments, information adjacent to the moment may be combined, such as information in a specific time period before and after the moment or information in a specific time period before the moment, to construct global information equivalent to the moment for combination. This is particularly suitable for situations where the global information may not contain information at the same moment as the local information. For example, the time period may be appropriately set, such as based on historical experience, etc., and will not be described in detail here.
[0073] In some embodiments of the present disclosure, optionally, during or after the combination operation of local information and global information, the combination result can also be used to correct the historical posture-related information of the object to obtain improved object posture-related information. As an example, the historical spatial position information of the object, such as the spatial position information of a specific part at the previous moment or the spatial position information obtained by the combination at the previous moment, can be corrected. In some embodiments, the combination result at the current moment can be weightedly combined with the historical posture-related information at the previous moment, and the weights here can be set in various appropriate ways, for example, based on historical experience.
[0074] In an exemplary implementation, according to some embodiments of the present disclosure, speed information at a specific moment at a specific part of local information and global information, spatial position information at a specific moment, the difference between the average speed at the specific part in the local information in the previous specific time period and the average speed at the specific part in the global information in the previous specific time period are weighted combined, and the weighted combination result is further weighted combined with the position information at the previous moment at the specific part in the local information, especially the combined position information, to serve as the corrected position information at the specific part at the current moment.
[0075] Therefore, correction is performed based on the combination of local information and global information, and further based on historical posture information, thereby further improving the accuracy and smoothness of the object posture information.
[0076] In the structural example of the above-mentioned apparatus, the processing circuit 202 can be in the form of a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the processing circuit 202 can be constructed by a circuit (hardware) or a central processing device (such as a central processing unit (CPU)). In addition, the processing circuit 302 can carry a program (software) for operating the circuit (hardware) or the central processing device. The program can be stored in a memory (such as, arranged in the memory) or an external storage medium connected from the outside, and can be downloaded via a network (such as the Internet).
[0077] According to an embodiment of the present disclosure, the processing circuit 202 may include various units for implementing the above-mentioned functions, for example, a first acquisition unit 204, configured to acquire local information related to the posture of a specific part of the motion capture object; a second acquisition unit 206, configured to acquire global information related to the posture of a specific part of the motion capture object; and a combination unit 208, configured to combine the local information and the global information to determine the posture information of the motion capture object.
[0078] In some embodiments, as an example, the first acquisition unit 204 may acquire data related to the posture of a specific part provided by a wearable device installed at a specific part of the subject, and optionally perform appropriate processing to provide the data as the local information. In some embodiments, as an example, the second acquisition unit 206 may acquire an image / video of the subject from a capture device and extract / analyze it to obtain data related to the posture of the specific part as global information; or the data related to the posture of the specific part may be extracted or analyzed by another device from the image / video of the subject from the capture device and provided to the second acquisition unit as global information.
[0079] According to an embodiment of the present disclosure, the combination unit 208 can be further configured to combine the local information and the global information of the specific part of the motion capture object to obtain posture-related information of the specific part, and determine the motion information of the motion capture object based on the posture-related information of the specific part of the motion capture object obtained by combination.
[0080] According to an embodiment of the present disclosure, before information is combined, the global information and the local information may be appropriately transformed or aligned. In some embodiments, the transformation or alignment may include information format transformation or alignment, and / or time alignment.
[0081] According to an embodiment of the present disclosure, such transformation or calibration can be implemented by a combining unit. In some embodiments, the combining unit 208 can be further configured to time-align the local information with the global information, and then combine the time-aligned local information and the global information. In particular, as an example, the combining unit 208 can include an alignment unit configured to time-align the local information with the global information. Of course, the alignment unit can also be located outside the combining unit.
[0082] In some embodiments, the combination unit or alignment unit may be further configured to, for local information at a specific moment, select global information at the same moment to align with the local information for combination; and / or combine global information in a specific time period adjacent to the specific moment, and then use the combination result as global information aligned with the local information at the specific moment for combination.
[0083] In some embodiments, the combining unit 208 may be further configured to transform at least one of the local information and the global information so that the two are represented in the same coordinate system and in the same format, and to combine the information based on the at least one of the transformed local information and the global information. In particular, as an example, the combining unit 208 may include a transform unit configured to transform at least one of the local information and the global information.
[0084] It should be noted that the aforementioned transformation unit and / or alignment unit may also be located in the apparatus 200 and outside the combining unit 208. Alternatively, they may even be implemented outside the apparatus 200 by any other suitable device or apparatus. In this case, the transformed and / or aligned data may be acquired as local information and / or global information by the first acquisition unit and the second acquisition unit, respectively.
[0085] According to an embodiment of the present disclosure, the combining unit 208 may be further configured to perform various appropriate information combinations, for example, may include an information combining unit.
[0086] In some embodiments, the combination unit or information combination unit may be configured to perform a weighted combination of the local information and the global information at a specific part of the object, wherein the weight includes a weight set based on at least one of the speed-related information of the specific part from the local information and the speed-related information of the specific part from the global information.
[0087] In some embodiments, the combination unit or information combination unit can be configured to perform a weighted combination of the spatial position information contained in or derived from the local characteristic information at the characteristic part of the object and the spatial position information contained in or derived from the global information, wherein the weight is based on the variance correlation value of the speed-related information at a specific part in the global information.
[0088] In some embodiments, the combination unit or the information combination unit may be configured to perform a weighted combination of the speed-related information contained in or derived from the local characteristic information at the characteristic part of the object and the speed-related information contained in or derived from the global information.
[0089] In some embodiments, the combination unit or the information combination unit can be configured to perform a weighted combination of the speed information at a specific moment at a specific part of the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment, and the speed information at a specific moment at a specific part of the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.
[0090] According to some embodiments of the present disclosure, the combining unit 208 may be further configured to determine object pose information based on the combined data. As an example, the combining unit 208 may include a determining unit for determining object pose information based on the combined data.
[0091] In some embodiments, the combination unit 208 or the determination unit may be configured to infer information about the overall posture of the object based on the correlation between the various parts of the object and / or the posture-related information of the specific parts of the combined motion capture object.
[0092] In some embodiments, the combination unit 208 or the determination unit may be configured to use the combination result of the global information and local information obtained at a specific moment to correct the position information of the specific part of the object in the local information at the previous moment of the specific moment as the position information of the specific part of the object in the corrected local information.
[0093] In some embodiments, the combination unit 208 or the determination unit may be configured to perform a weighted combination of the combination result of the global information and the local information at a specific moment at a specific part with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at a specific moment at a specific part from the local information and the speed-related information at a specific moment at a specific part from the global information.
[0094] Alternatively, the determination of the object posture information may also be performed outside the combination unit. For example, the determination unit or correction unit may be located in the apparatus 200 and outside the combination unit 208. Alternatively, the determination unit or correction unit may be located outside the apparatus 200 and implemented by any other suitable device or apparatus.
[0095] It should be noted that although FIG2 shows the various units as separate units, one or more of these units may be combined into one unit or split into multiple units. In addition, some units may not be included in the processing circuit or even the receiving-side device, and thus may be shown with dashed lines. As an example, at least one of the transformation unit 210 and the alignment unit, or even both, may even be outside the processing circuit 202, and thus may also be shown with dashed lines. In addition, as another implementation, the alignment unit may also be included in the transformation unit, or the transformation unit and the alignment unit may be included together in the calibration unit.
[0096] It should be noted that the above-mentioned units are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the accompanying drawings to indicate that these units may not actually exist, and the operations / functions they implement can be implemented by the processing circuit itself.
[0097] It should be understood that Figure 2 is only a schematic structural configuration of an electronic device for motion capture according to an embodiment of the present disclosure. The device 200 may also include other possible components, such as memory, network interface, controller, communication unit, etc., which are not shown for the sake of clarity. In particular, the processing circuit can be associated with the memory. For example, the processing circuit can be directly or indirectly (for example, other components may be connected in the middle) connected to the memory to access image processing related data. The memory can store various data and / or information generated by the processing circuit 202. The memory can also be located within the optimization device but outside the processing circuit, or even outside the optimization device. The memory can be a volatile memory and / or a non-volatile memory. For example, the memory can include but is not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory.
[0098] The following describes a flow chart of a method for capturing motion of an object according to an embodiment of the present disclosure with reference to FIG3A . In method 300 , in step S301 (a first acquisition step), local information related to the posture of a specific part of the motion capture object is acquired; in step S302 (a second acquisition step), global information related to the posture of the specific part of the motion capture object is acquired; and in step S303 (a combination step), the local information and the global information are combined to determine the posture information of the motion capture object.
[0099] It should be noted that the order shown in the first acquisition step and the second acquisition step is only to illustrate that the method includes these two steps, and does not limit the execution order of the two steps. Without limitation, the two acquisition steps can be performed in parallel, for example, at their respective frequencies and intervals. In some embodiments, generally, the acquisition frequency of the local information in the first acquisition step can be equal to or higher than the acquisition frequency of the global information in the second acquisition step, that is, compared with the local information, the global information (for example, the image / video frame used to extract or analyze the global information) can be acquired at equal or greater time intervals.
[0100] According to an embodiment of the present disclosure, the various steps in method 300 can be performed as described above, and in particular, the various appropriate operations described herein can be performed. In particular, FIG3B shows a flowchart of an exemplary implementation of the combination step according to an embodiment of the present disclosure. In step S313 (transformation step), the transformation / calibration described herein can be performed, including format conversion, time alignment, etc., in step S323 (information combination step), the various global information and local information combination operations described herein can be performed, and in step S333 (inference step), the inference operation of the object posture information and motion information described herein can be performed, and in particular, may include correction operations. It will not be described in detail here. It should be noted that these steps may also be outside the method of the present disclosure.
[0101] It should be noted that these steps can be performed by any appropriate device or device component, such as the aforementioned receiving-side device, the processing circuit in the receiving-side device, the corresponding components in the processing circuit, etc. It should be noted that the audio presentation method according to the embodiments of the present disclosure may also include other steps, such as the various further processing described above. Moreover, these further processing can also be performed by appropriate devices or device components and will not be described in detail here.
[0102] According to an embodiment of the present disclosure, a motion capture system is also provided. As shown in Figure 4A, the system 400 includes a camera device 401, which can be configured to capture images or videos of an object or a specific part of an object. As an example, the camera device 401 may include a camera device set separately from the object, such as a camera; a wearable device 402, which can be configured at the object or a specific part of the object and obtain three-dimensional spatial information of a specific position. As an example, the wearable device 402 may include a device based on an inertial sensor; and a processing device (such as, which can be implemented as various appropriate processors) 403, which can be configured to receive information provided by the camera device 401 and the wearable device 402, and determine the posture information of the object based on the received information. In particular, the information from the camera device 401 may correspond to the aforementioned global information, which may be graphics / videos captured by the camera device 401, and the global information is analyzed or extracted by the processor, or global information is extracted or analyzed by the camera device 401 itself or analyzed by other devices; the information provided from the wearable device 402 may correspond to the aforementioned local information, so that the processing device 403 may correspond to the aforementioned electronic device for motion capture and perform the operations as described above.
[0103] 4B shows a conceptual flow chart of object motion capture according to an embodiment of the present disclosure. On the one hand, an image or video of the object or a specific part of the object is captured by a camera device set separately from the object, thereby obtaining global information. On the other hand, local information is obtained by a sensing device (such as various appropriate wearable devices capable of performing sensing) configured on the object or the specific part of the object. Then, optionally, at least one of the global information and the local information is transformed, for example, into spatial information within the same space / coordinate system. Then, the global information and the local information, or the optional transformed global information and the local information, are combined by a processing device to obtain the posture information of the object.
[0104] An exemplary implementation of a motion capture system according to an embodiment of the present disclosure will be described below with reference to the accompanying drawings. In the following description, the human body is used as an example of a motion capture object. However, it should be noted that the motion capture solution of the present disclosure can be equally applied to other objects, particularly other movable objects, such as other objects held by or associated with a human body, other movable organisms, and the like.
[0105] Figure 5 illustrates an exemplary motion capture system according to some embodiments of the present disclosure. The system includes: a camera configured to capture multiple image frames containing two-dimensional image information of a wearer's limbs, particularly continuous image frames; multiple wearable devices configured to capture multiple sets of three-dimensional spatial information of key parts of the wearer's limbs within the timeframe of the multiple image frames; and a processor configured to receive the three-dimensional spatial information and the multiple image frames and utilize inter-frame information between the multiple image frames to compensate for differences between the multiple sets of three-dimensional spatial information, thereby obtaining three-dimensional spatial information with smooth transitions between the sets.
[0106] The aforementioned camera is preferably an ordinary webcam, or a mobile phone camera, etc. It can communicate with the outside world through data lines, integrated data transmission (such as a mobile phone and its camera), or through wireless network transmission.
[0107] Figure 5 shows six wearable devices that can be worn on the human body. The wearable devices can be worn on six joints of the human body, preferably the head, waist, wrists, and ankles. It should be noted that this is only exemplary and the positions can be increased, decreased, and adjusted as needed. The wearable device is preferably a device that includes an inertial sensor (IMU) and a communication unit and can send three-dimensional spatial information to the outside world, such as motion acceleration, angular velocity, and the speed and displacement information calculated therefrom. As an example, the appearance style of the wearable device is preferably lightweight, and highly convenient styles such as bracelets, anklets, hairpins, etc. meet the needs of the human body without burden. The internal data acquisition and calculation processing is preferably a well-known inertial sensor data processing method. The method of communication with the outside world and data transmission is preferably a communication method based on the Bluetooth communication protocol, or it can be a data transmission method based on a WIFI wireless network.
[0108] The aforementioned processor is preferably a device with a computing unit, such as a computer or mobile phone, capable of processing the received three-dimensional spatial information and multiple image frames to determine the motion information of the human body. Such processing may correspond to the aforementioned combination step. Preferably, the processing steps of the processor may include: a pre-processing step, an information combination step, and a post-processing step, corresponding to the aforementioned transformation step, information combination step, and inference step, respectively.
[0109] FIG6 shows a schematic implementation of combined processing, especially the pre-processing step, in a motion capture system according to an embodiment of the present disclosure. In this step, the processor receives continuous image frames or video frames from the camera and performs image frame data processing. For example, it can detect six joints of the human body frame by frame and output two-dimensional spatial information of the joints, including two-dimensional position information, motion-related information, etc. This detection can be performed using various appropriate detection algorithms. In particular, the human joint detection algorithm is preferably a joint detection algorithm based on a deep learning network, or an algorithm based on traditional image processing technology. Note that the above is only an example. In other implementations of the present disclosure, the camera can also directly detect human joints through edge computing, and only send the two-dimensional spatial information of the human joints in each frame to the processor.
[0110] In addition, the processor also receives motion information of the six joints of the human body sent by the six wearable devices, including but not limited to acceleration, angular velocity and other information, such as continuous motion acceleration, angular velocity, such as acceleration per hertz, angular velocity information, and then performs wearable device data processing, for example, by calculating the three-dimensional position information of the six wearable devices projected in a unified virtual space, and then obtaining the three-dimensional spatial information of the six joints. The position information calculation is preferably the inertial sensor integral calculation in the prior art, and the projection calculation of multiple inertial sensors in the virtual space is preferably a position prediction calculation based on a deep learning network. Note that the above is only an example. In other implementations of the present disclosure, the six wearable devices can also directly perform position calculations of the human joints in a unified virtual space through edge computing, thereby obtaining the spatial information of each joint, including position information, motion-related information, etc., and only send the spatial information of the human joints per hertz to the processor.
[0111] The information combination step includes a spatial calibration step and a fusion smoothing step. The spatial calibration step preferably maps the camera's two-dimensional human joint positions to the three-dimensional human joint positions in virtual space, and the camera's two-dimensional spatial position is correspondingly expanded to a three-dimensional spatial position. This calibration operation can be performed using various appropriate methods known in the art, such as based on specific constraints.
[0112] The fusion smoothing step is preferably a fusion method based on the following formulas (1) and (2) for each joint point. K t =tanh(VAR(V CAM )) (2)
[0113] in, K is the fused 3D position of each joint point at the current time t and the previous time t-1, t is the dynamic fusion weight, X CAM , X IMU are the 3D positions from the camera and the wearable device at the current moment after spatial calibration, V CAM ,V IMU , The speeds from the camera and the wearable device at the current moment after spatial calibration are respectively the average speeds from the camera and the wearable device at the previous few moments. The speed is preferably obtained by taking the difference between the current position information and the position information at the previous moment. VAR is the variance function, tanh is the inverse tangent function, and α, β, and γ are weight coefficients. The dynamic fusion weight is calculated using the relevant speed information obtained from the camera and the relevant speed information obtained from the wearable device and then dynamic fusion is performed. The α weight is a smoothing weight and is preferably a fixed coefficient, but it can also be a dynamic coefficient.
[0114] It should be noted that this is only exemplary. In other implementations of the present disclosure, for example It can also be used as the result of the combination operation without further correction of the historical combination information, that is, no longer as in the above formula Further corrections.
[0115] The post-processing step involves using the fused and smoothed joint positions to update the original joint positions obtained from multiple wearable devices. The positions of preferably six human joints are then used to infer the motion information of all human joints, thereby driving the human virtual model to move in virtual 3D space. The total number of human joints is preferably all joints in the human kinematic chain, including the shoulders, neck, neck, and knees. The joint motion information is preferably a motion matrix.
[0116] The position information provided by the wearable device after fusion and smooth update can more accurately simulate the position of human body movement, thereby achieving high-precision human motion capture.
[0117] Because the camera is subject to significant interference from external conditions, the wearer's limb information captured between consecutive frames is highly accurate globally but contains significant local noise. This creates a problem for wearable devices, resulting in low local noise but low global 3D accuracy. Compensating for the differences between the multiple sets of 3D spatial information by using inter-frame information between multiple image frames can effectively improve the accuracy, smoothness, and noise resistance of the wearer's limb 3D information.
[0118] According to embodiments of the present disclosure, the improved object pose / motion information can be further processed or applied in various appropriate ways. For example, it can be presented to a viewer via an appropriate presentation device, such as various types of display devices, or it can be provided to a processing device for further processing and application in various appropriate scenarios, such as fitting into virtual objects in film and animation production, performing video / audio interaction, and the like. As an example, the obtained object motion information can be applied to appropriate interactive applications, where interaction can be performed based on the object's pose.
[0119] As an example, in an interactive application, the user's posture information can be accurately obtained by accurately detecting the user's movements, and then the user's reaction, emotion, etc. can be judged based on the user's posture information, and then the presented content can be adaptively adjusted. For example, the emotion and reaction corresponding to the user's posture / movement can be pre-set, and then the user's reaction, emotion, etc. can be judged based on the detected user posture, and the content can be adaptively adjusted. As an example, in an audio interactive application scenario, the user's reaction to the audio can be judged based on the user's posture, so as to adjust the audio content. For example, in some embodiments, the adjustment includes at least one of the following: increasing or decreasing the volume of the audio content; adjusting the rhythm of the audio content; enhancing the effect of the audio content; adding additional effects to the audio content. It should be noted that the motion capture scheme according to the embodiment of the present disclosure can also be effectively applied to other applications.
[0120] In some embodiments, the disclosed methods can be implemented via software algorithms, thereby enabling convenient integration into various types of devices including presentation devices, such as devices including various presentation devices, such as finger cuffs. Specifically, the disclosed methods can be implemented as computer programs, instructions, and the like, and executed by a processor of a portable device to facilitate object motion capture, such as human motion capture and fitting, for various appropriate applications.
[0121] In addition, it should be understood that the above series of processes and devices can also be implemented by software and / or firmware. In the case of implementation by software and / or firmware, the program constituting the software is installed from a storage medium or a network to a computer with a dedicated hardware structure, such as a general-purpose personal computer 700 shown in Figure 7. When various programs are installed, the computer can perform various functions, etc. Figure 7 is a block diagram showing an example structure of a personal computer of an optimization device that can be used in an embodiment of the present disclosure. In one example, the personal computer can correspond to the above-mentioned exemplary optimization device according to the present disclosure.
[0122] 7 , a central processing unit (CPU) 701 executes various processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 to a random access memory (RAM) 703. In the RAM 703, data required when the CPU 701 executes various processes and the like is also stored as needed.
[0123] The CPU 701, the ROM 702, and the RAM 703 are connected to one another via a bus 704. An input / output interface 705 is also connected to the bus 704.
[0124] The following components are connected to the input / output interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including various suitable storage devices such as a magnetic storage device, an optical storage device, and the like, such as a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet.
[0125] A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 710 as needed so that a computer program read therefrom is installed in the storage section 708 as needed.
[0126] In the case of realizing the above-described series of processing by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 711 .
[0127] Those skilled in the art will appreciate that such storage media are not limited to the removable medium 711 shown in FIG7 , which stores the program therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 711 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memories (CD-ROMs) and digital versatile disks (DVDs)), magneto-optical disks (including minidiscs (MDs) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 702, a hard disk included in the storage section 708, or the like, in which the program is stored and distributed to the user together with the device containing the program.
[0128] It should be noted that the methods and apparatus described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application-specific integrated circuit.
[0129] In addition, the methods and systems of the present disclosure may be implemented in a variety of ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination thereof. The order of the steps of the method described above is merely illustrative, and unless otherwise specifically stated, the steps of the method of the present disclosure are not limited to the order specifically described above. In addition, in some embodiments, the present disclosure may also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for implementing the method according to the present disclosure. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.
[0130] Those skilled in the art will appreciate that the boundaries between the above-mentioned operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed among additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in various other embodiments. However, other modifications, variations, and replacements are also possible. Therefore, this specification and the accompanying drawings should be considered illustrative, not restrictive.
[0131] Additionally, embodiments of the present disclosure may also include the following exemplary embodiment implementations (EEEs).
[0132] EEE 1. An electronic device for motion capture, the device comprising a processing circuit configured to:
[0133] Obtaining local information related to the posture of a specific part of the motion capture object;
[0134] Obtaining global information related to the posture of specific parts of the motion capture object;
[0135] The local information and the global information are combined to determine the posture information of the motion capture object.
[0136] EEE 2. An electronic device according to EEE 1, wherein the local information includes first spatial information related to the posture of the specific part expressed in a local space or a local coordinate system, which may include at least one of three-dimensional spatial position information and motion-related information of the specific part.
[0137] EEE 3. The electronic device according to EEE 1, wherein the motion-related information of the specific part includes at least one of motion acceleration, angular velocity, speed, and displacement of the specific part.
[0138] EEE 4. The electronic device according to EEE 1, wherein the global information includes second spatial information related to a posture of a specific part expressed in a global space or a global coordinate system.
[0139] EEE 5. The electronic device according to EEE 4, wherein:
[0140] The second spatial information is obtained by analyzing or extracting from an overall image and / or video of the motion capture object, or by analyzing or extracting from an image and / or video of a specific part of the motion capture object.
[0141] EEE 6. The electronic device according to EEE 4, wherein:
[0142] The second spatial information includes:
[0143] The two-dimensional spatial position information and motion-related information of the specific part obtained by the camera device, and / or
[0144] Three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information.
[0145] EEE 7. The electronic device according to EEE 6, wherein the three-dimensional spatial position information and motion-related information converted based on the two-dimensional position information are in the same format as the first spatial information or in the same coordinate system.
[0146] EEE 8. The electronic device according to EEE 1, wherein combining the local information and the global information to determine the posture information of the motion capture object includes:
[0147] combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and
[0148] Based on the posture related information of the specific part of the motion capture object obtained by combining, the motion information of the motion capture object is determined.
[0149] EEE 9. The device according to EEE 8, wherein, based on the correlation between various parts of the object and / or the various parts of the object, the posture-related information of the specific parts of the motion capture object obtained by combination is inferred as the motion information.
[0150] EEE 10. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:
[0151] The local information and the global information are time-aligned, and then
[0152] The time-aligned local information and the global information are combined.
[0153] EEE 11. The device according to EEE 10, wherein time-aligning the local information with the global information comprises: for the local information at a specific moment,
[0154] Selecting the global information and the local information at the same time to be aligned for combination; and / or
[0155] The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.
[0156] EEE 12. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:
[0157] transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and
[0158] Information combining is performed based on the at least one of the transformed local information and the global information.
[0159] EEE 13. The electronic device according to EEE 1, wherein combining the local information and the global information comprises:
[0160] The local information and the global information at the specific part of the object are weightedly combined, wherein the weight includes a weight set based on at least one of the velocity-related information of the specific part from the local information and the velocity-related information of the specific part from the global information.
[0161] EEE 14. The electronic device according to EEE 13, wherein weighted combination of the local information at a specific part of the object and the global information comprises:
[0162] The spatial position information contained in or derived from the local characteristic information at the characteristic part of the object is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on the variance correlation value of the speed-related information at the specific part in the global information.
[0163] EEE 15. The electronic device according to EEE 13 or 14, wherein weighted combination of the local information at a specific part of the object and the global information comprises:
[0164] The speed-related information included in or derived from the local characteristic information at the characteristic part of the object and the speed-related information included in or derived from the global information are weightedly combined.
[0165] EEE 16. The apparatus according to any one of EEE 13-15, wherein:
[0166] The speed-related information at the specific part in the local information includes the speed of the specific part of the object at a specific moment in the local information and the average speed in the previous specific time period;
[0167] The speed-related information at the specific part in the global information includes the speed at a specific moment of the specific part of the object in the global information and the average speed in a previous specific period of time.
[0168] EEE 17. The apparatus according to EEE 13, wherein combining the local information and the global information comprises:
[0169] The speed information at a specific moment in a specific part, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment in the local information are weightedly combined with the speed information at a specific moment in a specific part, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment in the global information.
[0170] EEE 18. The apparatus according to EEE 1, wherein combining the local information and the global information further comprises:
[0171] The combined result of global information and local information obtained at a specific moment is used to correct the position information of the specific part of the object in the local information at a moment before the specific moment, and use it as the position information of the specific part of the object in the corrected local information.
[0172] EEE 19. The apparatus according to EEE 18, wherein the modification comprises
[0173] The combination result of the global information and the local information at a specific moment in a specific part is weightedly combined with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific moment in the specific part from the local information and the speed-related information at the specific moment in the specific part from the global information.
[0174] EEE 20. The device according to EEE 1, wherein the local information is acquired during acquisition of the global information, and / or a frequency of acquiring the local information is the same as or different from a frequency of acquiring the global information.
[0175] EEE 21. The device according to EEE 1, wherein the specific part comprises a representative part or a basic part that can characterize or derive the motion posture of the object.
[0176] EEE 22. The device according to EEE 1, wherein the object is a person whose motion is captured, and the specific parts include basic joints of the human body, including at least one of the head, waist, wrists, ankles, etc.
[0177] EEE 23. A method for motion capture, comprising:
[0178] Obtaining local information related to the posture of a specific part of the motion capture object;
[0179] Obtaining global information related to the posture of specific parts of the motion capture object;
[0180] The local information and the global information are combined to determine posture information of the motion capture object.
[0181] EEE 24. The method according to EEE 23, wherein combining the local information and the global information to determine the posture information of the motion capture object comprises:
[0182] combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and
[0183] Based on the posture related information of the specific part of the motion capture object obtained by combining, the motion information of the motion capture object is determined.
[0184] EEE 25. The method according to EEE 23, wherein combining the local information and the global information comprises:
[0185] The local information and the global information are time-aligned, and then
[0186] The time-aligned local information and the global information are combined.
[0187] EEE 26. The method according to EEE 25, wherein time-aligning the local information with the global information comprises: for the local information at a specific moment,
[0188] Selecting the global information and the local information at the same time to be aligned for combination; and / or
[0189] The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.
[0190] EEE 27. The method according to EEE 23, wherein combining the local information and the global information comprises:
[0191] transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and
[0192] Information combining is performed based on the at least one of the transformed local information and the global information.
[0193] EEE 28. The method according to EEE 23, wherein combining the local information and the global information comprises:
[0194] The local information and the global information at the specific part of the object are weightedly combined, wherein the weight includes a weight set based on at least one of the velocity-related information of the specific part from the local information and the velocity-related information of the specific part from the global information.
[0195] EEE 29. The method according to EEE 28, wherein weighted combination of the local information and the global information at a specific part of the object comprises:
[0196] The spatial position information contained in or derived from the local characteristic information at the characteristic part of the object is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on the variance correlation value of the speed-related information at the specific part in the global information.
[0197] EEE 30. The method according to EEE 23, wherein weighted combination of the local information and the global information at a specific part of the object comprises:
[0198] The speed-related information included in or derived from the local characteristic information at the characteristic part of the object and the speed-related information included in or derived from the global information are weightedly combined.
[0199] EEE 31. The method according to EEE 23, wherein combining the local information and the global information comprises:
[0200] The speed information at a specific moment in a specific part, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment in the local information are weightedly combined with the speed information at a specific moment in a specific part, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment in the global information.
[0201] EEE 32. The method according to EEE 23, wherein combining the local information and the global information further comprises:
[0202] The combined result of global information and local information obtained at a specific moment is used to correct the position information of the specific part of the object in the local information at a moment before the specific moment, and use it as the position information of the specific part of the object in the corrected local information.
[0203] EEE 33. A method according to EEE 32, wherein the correction includes weighted combination of the combination result of global information and local information at a specific moment at a specific part and the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific moment of the specific part from the local information and the speed-related information at the specific moment of the specific part from the global information.
[0204] EEE 34. A motion capture system comprising:
[0205] a wearable device, which is disposed on a specific part of a motion capture subject and is configured to obtain local information related to a posture of the specific part of the motion capture subject;
[0206] a camera device configured to obtain global information related to a posture of a specific part of a motion capture object;
[0207] A processor is configured to combine the local information and the global information to determine posture information of the motion capture object.
[0208] EEE 35. A device comprising at least one processor; and at least one storage device storing thereon instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of EEE 23-33.
[0209] EEE 36. A storage medium storing instructions, which, when executed by a processor, enable the processor to perform the method according to any one of EEE23-33.
[0210] EEE 37. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of EEE 23-33.
[0211] EEE 38. A computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of EEE 23-33.
[0212] Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and transformations can be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. Moreover, the terms "comprises," "comprising," or any other variations thereof in the embodiments of the present disclosure are intended to cover non-exclusive inclusions, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.
[0213] Although some specific embodiments of the present disclosure have been described in detail, it should be understood by those skilled in the art that the above embodiments are merely illustrative and do not limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be combined, modified, or replaced without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. An electronic device for motion capture, the device comprising a processing circuit configured to: Acquire local information related to the posture of a specific part of the motion capture object; Obtaining global information related to the posture of specific parts of the motion capture object; The local information and the global information are combined to determine the posture information of the motion capture object.
2. The electronic device according to claim 1, wherein: The local information includes first spatial information related to the posture of the specific part expressed in a local space or a local coordinate system, which may include at least one of the three-dimensional spatial position information and motion-related information of the specific part.
3. The electronic device according to claim 1, wherein: The motion-related information of the specific part includes at least one of the motion acceleration, angular velocity, speed, and displacement of the specific part.
4. The electronic device according to claim 1, wherein: The global information includes second spatial information related to the posture of the specific part expressed in a global space or a global coordinate system.
5. The electronic device according to claim 4, wherein: The second spatial information is obtained by analyzing or extracting from an overall image and / or video of the motion capture object, or by analyzing or extracting from an image and / or video of a specific part of the motion capture object.
6. The electronic device according to claim 4, wherein: The second spatial information includes: The two-dimensional spatial position information and motion-related information of the specific part obtained by the camera device, and / or Three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information.
7. The electronic device according to claim 6, wherein: The three-dimensional spatial position information and motion-related information obtained based on the conversion of the two-dimensional position information are in the same format as the first spatial information or in the same coordinate system.
8. The electronic device according to claim 1, wherein: Combining the local information with the global information to determine the posture information of the motion capture object includes: combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and Based on the combined posture-related information of the specific part of the motion capture object, the motion information of the motion capture object is determined.
9. The electronic device according to claim 8, wherein: Based on the correlation between various parts in the object and / or various parts in the object, information about the overall posture of the object is inferred based on the posture-related information of the specific part of the motion capture object obtained by combination, as the motion information.
10. The electronic device according to claim 1, wherein: Combining the local information with the global information comprises: The local information and the global information are time-aligned, and then The time-aligned local information and the global information are combined.
11. The electronic device according to claim 10, wherein: The time alignment of the local information and the global information includes: for the local information at a specific time, Selecting global information at the same time as the global information aligned with the local information for combination; and / or The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.
12. The electronic device according to claim 1, wherein: Combining the local information with the global information comprises: transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and Information combining is performed based on the at least one of the transformed local information and the global information.
13. The electronic device according to claim 1, wherein: Combining the local information with the global information comprises: The local information at the specific part and the global information are weightedly combined, wherein the weight includes a weight set based on at least one of the speed-related information of the specific part from the local information and the speed-related information of the specific part from the global information.
14. The electronic device according to claim 13, wherein: The weighted combination of the local information and the global information at a specific location includes: The spatial position information contained in or derived from the local characteristic information at the characteristic part is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on a variance correlation value of the speed-related information at a specific part in the global information.
15. The electronic device according to claim 13 or 14, wherein: The weighted combination of the local information and the global information at a specific location includes: The speed-related information contained in or obtained in the local information at a specific location and the speed-related information contained in or obtained in the global information are weightedly combined.
16. The electronic device according to any one of claims 13 to 15, wherein: The speed-related information at the specific part in the local information includes the speed at the specific part of the object in the local information at a specific moment and the average speed in the previous specific period; The speed-related information at the specific part in the global information includes the speed at the specific part of the object in the global information at a specific moment and the average speed in the previous specific period.
17. The electronic device according to claim 13, wherein: Combining the local information with the global information comprises: The speed information at a specific moment at a specific part in the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment are weightedly combined with the speed information at a specific moment at a specific part in the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.
18. The electronic device according to claim 1, wherein: Combining the local information with the global information further comprises: The combination of global information and local information obtained at a specific moment is used to correct the specific local information. The position information of the part at the previous moment of the specific moment is used as the position information of the specific part in the corrected local information.
19. The electronic device according to claim 18, wherein: Modifications include The combination result of the global information and the local information at a specific part at a specific moment is weightedly combined with the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific part at the specific moment from the local information and the speed-related information at the specific part at the specific moment from the global information.
20. The electronic device according to claim 1, wherein: The local information is acquired during the acquisition of the global information, and / or the frequency of acquiring the local information is the same as or different from the frequency of acquiring the global information.
21. The electronic device according to claim 1, wherein: The specific part includes a representative part or a basic part that can characterize or derive the motion posture of the object.
22. The electronic device according to claim 1, wherein: The object is a person whose motion is captured, and the specific parts include basic joints of the human body, including at least one of the head, waist, wrist, and ankle.
23. A method for motion capture, comprising: Acquire local information related to the posture of a specific part of the motion capture object; Obtaining global information related to the posture of specific parts of the motion capture object; The local information and the global information are combined to determine the posture information of the motion capture object.
24. The method according to claim 23, wherein: Combining the local information with the global information to determine the posture information of the motion capture object includes: combining the local information of the specific part of the motion capture object with the global information to obtain posture-related information of the specific part, and Based on the combined posture-related information of the specific part of the motion capture object, the motion information of the motion capture object is determined.
25. The method according to claim 23, wherein: Combining the local information with the global information comprises: The local information and the global information are time-aligned, and then The time-aligned local information and the global information are combined.
26. The method according to claim 25, wherein: The time alignment of the local information and the global information includes: for the local information at a specific time, Selecting global information at the same time as the global information aligned with the local information for combination; and / or The global information in a specific time period adjacent to the specific moment is combined, and then the combined result is used as the global information aligned with the local information at the specific moment for combination.
27. The method according to claim 23, wherein: Combining the local information with the global information comprises: transforming at least one of the local information and the global information so that both are represented in the same format in the same coordinate system, and Information combining is performed based on the at least one of the transformed local information and the global information.
28. The method of claim 23, wherein: Combining the local information with the global information comprises: The local information at the specific part and the global information are weightedly combined, wherein the weight includes a weight set based on at least one of the speed-related information of the specific part from the local information and the speed-related information of the specific part from the global information.
29. The method according to claim 28, wherein: The weighted combination of the local information and the global information at a specific location includes: The spatial position information contained in or derived from the local characteristic information at the characteristic part is weightedly combined with the spatial position information contained in or derived from the global information, wherein the weight is based on a variance correlation value of the speed-related information at a specific part in the global information.
30. The method of claim 23, wherein: The weighted combination of the local information and the global information at a specific location includes: The speed-related information contained in or obtained in the local characteristic information at the characteristic part and the speed-related information contained in or obtained in the global information are weightedly combined.
31. The method of claim 23, wherein: Combining the local information with the global information comprises: The speed information at a specific moment at a specific part in the local information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment are weightedly combined with the speed information at a specific moment at a specific part in the global information, the spatial position information, and the average speed information in a specific time period adjacent to the specific moment.
32. The method of claim 23, wherein: Combining the local information with the global information further comprises: The combination result of the global information and the local information obtained at a specific moment is used to correct the position information of the specific part in the local information at the previous moment of the specific moment, and use it as the position information of the specific part in the corrected local information.
33. The method of claim 32, wherein: The correction includes weighted combination of the combination result of global information and local information at a specific part at a specific moment and the position information at a previous moment, wherein the weight includes a weight set based on at least one of the speed-related information at the specific part at the specific moment from the local information and the speed-related information at the specific part at the specific moment from the global information.
34. A motion capture system comprising: A wearable device, which is disposed on a specific part of a motion capture object and is configured to obtain local information related to a posture of the specific part of the motion capture object; A camera device configured to obtain global information related to the posture of a specific part of the motion capture object; A processor is configured to combine the local information and the global information to determine posture information of the motion capture object.
35. An apparatus comprising at least one processor; and at least one storage device storing thereon instructions which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 23-33.
36. A storage medium storing instructions which, when executed by a processor, enable the processor to perform the method according to any one of claims 23 to 33.
37. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 23 to 33.
38. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 23 to 33.
Citation Information
Patent Citations
Human motion capturing method and system based on optical inertial fusion
CN116189297A
Mixed motion capture system and method
US20180089841A1
Pose determination method, robot using the same, and computer readable storage medium
US20220152835A1
Device for implementing motion on basis of convergence motion capture system, and method thereof
WO2020116837A1
