Motion capture method and device, electronic equipment and storage medium

Abstract

The invention provides a motion capture method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring human body inertial data collected by augmented reality equipment; inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information; wherein the motion capture model is obtained by training a neural network model according to a preset loss function; the motion capture model comprises one or more of a neural network used for predicting a human body posture, a neural network used for predicting global displacement information, a neural network used for predicting human body skeleton parameters and a classification network used for predicting a human body treading state. According to the method disclosed by the invention, end-to-end human body motion capture can be realized through the motion capture model.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent terminals, and in particular to a motion capture method, device, electronic device and storage medium. Background Art

[0002] Human motion capture technology refers to the use of certain sensors to capture the posture and motion data of human movement in the scene. Based on these motion posture data, the virtual image model can be driven or behavioral analysis can be performed. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] The present disclosure provides a motion capture method, device, electronic device and storage medium.

[0005] The present disclosure adopts the following technical solutions.

[0006] In some embodiments, the present disclosure provides a motion capture method, comprising:

[0007] Obtain human inertial data collected by the extended reality device;

[0008] Inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information;

[0009] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0010] In some embodiments, the present disclosure provides a motion capture device, comprising:

[0011] An acquisition module, used to acquire human inertial data collected by an extended reality device;

[0012] A processing module, used for inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information;

[0013] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0014] In some embodiments, the present disclosure provides an electronic device, comprising: at least one memory and at least one processor;

[0015] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to execute the above method.

[0016] In some embodiments, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the processor is prompted to perform the above method.

[0017] The motion capture method provided by the embodiment of the present disclosure obtains the human inertial data collected by the extended reality device; then the human inertial data is input into a pre-trained motion capture model to obtain human posture information; wherein the motion capture model is obtained by training a neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters, and a classification network for predicting the human body stepping state. It can be seen that the embodiment of the present disclosure can achieve accurate estimation of the user's whole body joint posture information based on the extended reality device and the motion capture model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0019] Figure 1 4 is a flow chart of a motion capture method according to an embodiment of the present disclosure.

[0020] Figure 2 It is a structural diagram of the motion capture model of the embodiment of the present disclosure.

[0021] Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in sequence and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] The term "including" and its variations used in this document are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. The term "responsive to" and related terms refer to a signal or event being affected to a certain extent by another signal or event, but not necessarily completely or directly affected. If event x occurs "responsive to" event y, x may be directly or indirectly responsive to y. For example, the occurrence of y may eventually lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. In addition, the term "responsive to" can also mean "at least partially responsive to".

[0025] The term "determine" broadly covers a variety of actions, which may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and similar actions, as well as parsing, selecting, choosing, establishing, and similar actions, etc. The relevant definitions of other terms will be given in the following description. The relevant definitions of other terms will be given in the following description.

[0026] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modification of “one” mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as “one or more” unless otherwise clearly indicated in the context.

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] It should be noted that the sensor used for human motion capture can be an inertial sensor IMU (Inertial Measurement Unit), or a three-color camera or a depth camera. Depending on the type of sensor used, the motion capture methods provided by related technologies can be divided into the following four categories: optical motion capture method, inertial motion capture method, visual motion capture method and hybrid motion capture method. Among them, except for the inertial motion capture method, the other three solutions require additional equipment to be deployed in the scene and are not suitable for application scenarios such as extended reality XR (XR-Extended Reality). Extended reality includes augmented reality AR (Augmented Reality), mixed reality MR (Miced Reality) and virtual reality VR (Virtual Reality).

[0030] However, the inertial motion capture method in the related technology requires the use of a large number of IMU sensors, and whether it is a wire-binding wearing solution or wearing a special jacket, it is time-consuming and prone to the problem of incorrect wearing order, which affects the user experience. It is also not suitable for application scenarios such as XR.

[0031] In response to the above problems, the related technology has proposed an inertial motion capture technology solution based on sparse IMU, which uses 6 IMU sensors at the hardware level, placed at the left wrist, right wrist, left knee, right knee, head and waist of the human body. Since this method only uses IMU sensors, it cannot provide accurate three-dimensional positioning information.

[0032] At the algorithm level, the relevant technology first collects inertial data measured by inertial sensors at the left wrist, right wrist, lower left knee, lower right knee, head and waist of the human body, and then obtains the bone orientation and acceleration of the bones in the human coordinate system based on the inertial data, and inputs the bone orientation and acceleration into the preset human kinematic model to predict the human body posture, joint movement speed, and human-ground contact information. Finally, the human body posture, joint movement speed, and human-ground contact information are input into the preset human dynamics model to predict the human body posture, movement, joint force and ground force.

[0033] It can be seen that the motion capture algorithm provided by the relevant technology is not an end-to-end learnable method. It requires an iterative post-processing process, so there are problems such as long calculation time and limited model performance.

[0034] In summary, the related technologies based on sparse IMU inertial motion capture technology all use 6 IMU sensors. In addition, at the algorithm level, the related technologies need to use human body dynamics post-processing modules to calculate the six degrees of freedom information 6DoF, which makes it difficult to achieve end-to-end learning and reasoning.

[0035] To solve the above problems, the solution provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, Figure 1 Detailed description of the invention The invention relates to a method for capturing motion according to an embodiment of the invention, which comprises the following steps.

[0037] Step S01: Acquire human body inertial data collected by an extended reality device;

[0038] In some embodiments, the present disclosure adopts a hardware system solution combining an extended reality head-mounted device and an IMU sensor, including a head-mounted device, two handheld controllers, a waist sensor, and two leg sensors. Among them, the head-mounted device has an integrated IMU sensor for outputting the inertial data of the human head. In addition, the head-mounted device also integrates the positioning and mapping SLAM (Simultaneous Localization And Mapping) technology, which can output the three-dimensional coordinate data of the human head. Among them, the two handheld controllers have integrated IMU sensors for outputting the inertial data of the two hands. In addition, through the sensor measurement between the head-mounted device and the handheld controller, the three-dimensional coordinate data of the handle can also be output.

[0039] In some embodiments, the human body inertial data X is obtained by the head mounted device, handheld controller, waist sensor and leg sensor. t . In the acquired set of time series data X={X1,X2,…,X t Before inputting the motion capture model, the input data needs to be preprocessed to convert the input data X of each frame into t Represented as a feature vector f t , in order to facilitate the motion capture model processing. Among them, each frame of human inertial data X t It includes the orientation data and acceleration data of 6 IMU sensors and the coordinate xyz axis data of 3 IMU sensors (head-mounted device and handheld controller).

[0040] It can be seen that the embodiment of the present disclosure uses an extended reality head-mounted device and a handheld controller to replace the IMU sensors set on the left and right hands and the head in the related technology. In addition, since the extended reality device integrates SLAM technology, it can also output the three-dimensional position data of the human head, thereby significantly improving the accuracy of the human body's 6DoF positioning.

[0041] Step S02: inputting the human body inertia data into a pre-trained motion capture model to obtain human body posture information;

[0042] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0043] In some embodiments, Figure 2 As shown, the motion capture model mainly includes two parts: a backbone network (i.e., backbone, such as a feature fusion network) and a head network (i.e., head, including a posture estimation network, a root node global displacement estimation network, a bone length estimation network, and a ground-stepping state binary classification network). The backbone network is composed of N basic blocks (basic_block) of the same structure, and each basic block contains a fully connected layer (Full Connection, FC), a normalization layer (LayerNorm) and a recurrent neural network (Recurrent Neural Network, RNN) structure. The main function of the backbone network is to fuse temporal feature information. The motion capture model disclosed in the present invention includes 4 head networks, which are used to estimate the whole body posture, the global displacement information of the root node, the bone parameters and the ground-stepping state information, and each head network is composed of a fitting network (e.g., Multi-Layer Perception MLP).

[0044] In some embodiments, after the preprocessed IMU sensor data is input into the motion capture model, human posture parameters are output, including six-degree-of-freedom information, global displacement information, bone parameters and ground contact status.

[0045] In some embodiments, the motion capture model is obtained by training the neural network model according to a preset loss function. The neural network model training includes the following 7 loss functions Loss, and the final loss function is obtained by weighted summation of the following 7 loss functions:

[0046] 1. Human posture prediction loss function: obtained by calculating the mean square error between the human posture parameters predicted by the model and the true posture parameters.

[0047] 2. Root node position estimation loss function: obtained by calculating the mean square error between the three-dimensional coordinate parameters of the root node predicted by the calculation model and the three-dimensional coordinate parameters of the true root node.

[0048] 3. Bone length estimation loss function: obtained by calculating the mean square error between the bone parameters predicted by the model and the true bone parameters.

[0049] 4. Joint point position estimation loss function: obtained by calculating the mean square error between the joint point position predicted by the model and the true value of the joint point position parameters; the model-predicted joint point position is calculated by the model-predicted posture parameters, the model-predicted root node three-dimensional coordinate parameters and the model-predicted bone parameters.

[0050] 5. Smoothness loss function: In order to ensure the smoothness of the output posture results in time series as much as possible, the embodiment of the present disclosure introduces a smoothness loss function.

[0051] 6. Foot-on-ground binary classification loss function: obtained by calculating the binary cross entropy between the foot-on-ground parameters predicted by the model and the true value of the foot-on-ground parameters. It includes two aspects of judgment and comparison: one is to judge whether the current foot is stationary, and the other is to judge whether the height of the current foot from the ground plane is less than the preset threshold.

[0052] 7. Sliding loss function: The three-dimensional coordinates of the human body joints need to be calculated based on the posture data, root node position and bone length predicted by the model. In terms of timing, due to the errors in the posture parameters, root node position and bone length estimation of the model, the calculated joints of the left and right feet of the human body have timing fluctuation problems. The phenomenon manifested is that when a person is standing all the time, the legs are always moving relative to the ground, which is the sliding problem. To address this problem, the embodiment of the present disclosure introduces a sliding loss function. For the foot that is on the ground, its temporal motion speed is calculated as the sliding loss. The greater the speed, the more serious the sliding phenomenon, and the greater the loss value. By minimizing the sliding loss of the model, the model can avoid the sliding problem while ensuring the accuracy of 6DoF estimation.

[0053] It can be seen that the embodiment of the present disclosure proposes a RNN-based model structure, which can output 6DoF information of all joints of the human body end-to-end, and can avoid problems such as jitter and sliding by introducing a smoothness loss function and a sliding loss function, without introducing additional post-processing logic.

[0054] The motion capture method provided by the embodiment of the present disclosure obtains the human inertial data collected by the extended reality device; then the human inertial data is input into a pre-trained motion capture model to obtain human posture information; wherein the motion capture model is obtained by training a neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters, and a classification network for predicting the human body stepping state. It can be seen that the embodiment of the present disclosure can achieve accurate estimation of the user's whole body joint posture information based on the extended reality device and the motion capture model.

[0055] In some embodiments, the extended reality device includes a head mounted device, a handheld controller, a waist sensor, and a leg sensor;

[0056] Wherein, the head mounted device, handheld controller, waist sensor and leg sensor include inertial sensors for collecting inertial data of the human body.

[0057] In some embodiments, after acquiring the human body inertial data collected by the extended reality device, the method further includes:

[0058] The human body inertial data is preprocessed to obtain a feature vector for inputting into the motion capture model.

[0059] In some embodiments, the pre-processing process is as follows:

[0060] 1. Convert the orientation matrices of multiple IMU sensors into 6D vectors;

[0061] 2. Calculate the angular velocity information of each IMU sensor (by multiplying the inverse of the direction matrix of the t-1 frame by the direction matrix of the t frame) and convert it into a 6D vector;

[0062] 3. Calculate the velocity information of the three IMU sensors of the head mounted device and the two handheld controllers (using the xyz coordinates of the t-1 frame minus the xyz coordinates of the t-1 frame);

[0063] 4. Integrate all the information from 1 to 3 and all the data from IMU sensors (such as acceleration data) to obtain a one-dimensional feature vector f t .

[0064] In some embodiments, the training of the neural network model according to a preset loss function includes:

[0065] The human inertial data for model training is pre-processed and then input into the neural network model, and the time series feature information of the human inertial data is fused according to the feature fusion network of the neural network model; the feature fusion network is composed of a fully connected layer, a normalization layer and a recurrent neural network;

[0066] Outputting human body posture parameters, global displacement information, human body skeleton parameters and human body ground stepping status through the neural network and the classification network;

[0067] According to the human body posture parameters, global displacement information, human body skeleton parameters, human body ground contact status and a preset loss function, the corresponding loss value is calculated, and the parameters of the neural network model are adjusted according to the loss value until the loss value is less than the preset value. The training of the neural network model is completed to obtain the motion capture model.

[0068] In some embodiments, the present disclosure can achieve real-time full-body motion capture effects based on an extended reality device, and has broad application prospects in the field of extended reality.

[0069] In some embodiments, the preset loss function is obtained by weighted summing of a human posture prediction loss function, a root node position estimation loss function, a bone length estimation loss function, a joint point position estimation loss function, a smoothness loss function, a ground contact binary classification loss function, and a sliding loss function;

[0070] Wherein, the human body posture prediction loss function is obtained by calculating the mean square error between the human body posture parameters predicted by the model and the true value posture parameters;

[0071] The root node position estimation loss function is obtained by calculating the mean square error between the three-dimensional coordinate parameters of the root node predicted by the calculation model and the three-dimensional coordinate parameters of the true value root node;

[0072] The bone length estimation loss function is obtained by calculating the mean square error between the bone parameters predicted by the model and the true value bone parameters;

[0073] The position estimation loss function of the joint point is obtained by calculating the mean square error between the joint point position predicted by the calculation model and the true value joint point position parameters; the joint point position predicted by the model is calculated by the posture parameters predicted by the model, the three-dimensional coordinate parameters of the root node predicted by the model and the skeleton parameters predicted by the model;

[0074] The smoothness loss function is used to output a human body posture result that is smooth in time series;

[0075] The ground contact binary classification loss function is obtained by calculating the binary cross entropy between the ground contact parameters predicted by the model and the ground contact parameters of the true value;

[0076] The sliding loss function is used to calculate the corresponding sliding loss value according to the temporal motion speed of the left and right foot joints, and optimize the neural network model according to the sliding loss value.

[0077] In some embodiments, the human body posture parameters include six-degree-of-freedom information of human body joints.

[0078] In some embodiments, it also includes:

[0079] According to the human body posture parameters, corresponding human body movements are generated and displayed in an extended reality environment.

[0080] The present disclosure also provides a motion capture device, including:

[0081] An acquisition module, used to acquire human inertial data collected by an extended reality device;

[0082] A processing module, used for inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information;

[0083] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0084] In some embodiments, the extended reality device includes a head mounted device, a handheld controller, a waist sensor, and a leg sensor;

[0085] Wherein, the head mounted device, handheld controller, waist sensor and leg sensor include inertial sensors for collecting inertial data of the human body.

[0086] In some embodiments, the acquisition module is further specifically used for:

[0087] The human body inertial data is preprocessed to obtain a feature vector for inputting into the motion capture model.

[0088] In some embodiments, the processing module is specifically used to:

[0089] The human inertial data for model training is pre-processed and then input into the neural network model, and the time series feature information of the human inertial data is fused according to the feature fusion network of the neural network model; the feature fusion network is composed of a fully connected layer, a normalization layer and a recurrent neural network;

[0090] Outputting human body posture parameters, global displacement information, human body skeleton parameters and human body ground stepping status through the neural network and the classification network;

[0091] According to the human body posture parameters, global displacement information, human body skeleton parameters, human body ground contact status and a preset loss function, the corresponding loss value is calculated, and the parameters of the neural network model are adjusted according to the loss value until the loss value is less than the preset value. The training of the neural network model is completed to obtain the motion capture model.

[0092] In some embodiments, the preset loss function is obtained by weighted summing of a human posture prediction loss function, a root node position estimation loss function, a bone length estimation loss function, a joint point position estimation loss function, a smoothness loss function, a ground contact binary classification loss function, and a sliding loss function;

[0093] Wherein, the human body posture prediction loss function is obtained by calculating the mean square error between the human body posture parameters predicted by the model and the true value posture parameters;

[0094] The root node position estimation loss function is obtained by calculating the mean square error between the three-dimensional coordinate parameters of the root node predicted by the calculation model and the three-dimensional coordinate parameters of the true value root node;

[0095] The bone length estimation loss function is obtained by calculating the mean square error between the bone parameters predicted by the model and the true value bone parameters;

[0096] The position estimation loss function of the joint point is obtained by calculating the mean square error between the joint point position predicted by the calculation model and the true value joint point position parameters; the joint point position predicted by the model is calculated by the posture parameters predicted by the model, the three-dimensional coordinate parameters of the root node predicted by the model and the skeleton parameters predicted by the model;

[0097] The smoothness loss function is used to output a human body posture result that is smooth in time series;

[0098] The ground contact binary classification loss function is obtained by calculating the binary cross entropy between the ground contact parameters predicted by the model and the ground contact parameters of the true value;

[0099] The sliding loss function is used to calculate the corresponding sliding loss value according to the temporal motion speed of the left and right foot joints, and optimize the neural network model according to the sliding loss value.

[0100] In some embodiments, the human body posture parameters include six-degree-of-freedom information of human body joints.

[0101] In some embodiments, the apparatus further comprises:

[0102] The display module is used to generate and display corresponding human body movements in an extended reality environment according to the human body posture parameters.

[0103] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separation modules may or may not be separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present embodiment. Those of ordinary skill in the art can understand and implement without paying creative work.

[0104] The method and device of the present disclosure are described above based on the embodiments and application examples. In addition, the present disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0105] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in the figure is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.

[0106] The electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0107] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0108] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0109] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0110] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0111] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0112] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0113] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0116] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] According to one or more embodiments of the present disclosure, a motion capture method is provided, comprising:

[0119] Obtain human inertial data collected by the extended reality device;

[0120] Inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information;

[0121] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0122] According to one or more embodiments of the present disclosure, the extended reality device includes a head-mounted device, a handheld controller, a waist sensor, and a leg sensor;

[0123] Wherein, the head mounted device, handheld controller, waist sensor and leg sensor include inertial sensors for collecting inertial data of the human body.

[0124] According to one or more embodiments of the present disclosure, after acquiring the human body inertial data collected by the extended reality device, the method further includes:

[0125] The human body inertial data is preprocessed to obtain a feature vector for inputting into the motion capture model.

[0126] According to one or more embodiments of the present disclosure, the training of the neural network model according to a preset loss function includes:

[0127] The human inertial data for model training is pre-processed and then input into the neural network model, and the time series feature information of the human inertial data is fused according to the feature fusion network of the neural network model; the feature fusion network is composed of a fully connected layer, a normalization layer and a recurrent neural network;

[0128] Outputting human body posture parameters, global displacement information, human body skeleton parameters and human body ground stepping status through the neural network and the classification network;

[0129] According to the human body posture parameters, global displacement information, human body skeleton parameters, human body ground contact status and a preset loss function, the corresponding loss value is calculated, and the parameters of the neural network model are adjusted according to the loss value until the loss value is less than the preset value. The training of the neural network model is completed to obtain the motion capture model.

[0130] According to one or more embodiments of the present disclosure, the preset loss function is obtained by weighted summation of a human posture prediction loss function, a root node position estimation loss function, a bone length estimation loss function, a joint point position estimation loss function, a smoothness loss function, a ground contact binary classification loss function, and a sliding loss function;

[0131] Wherein, the human body posture prediction loss function is obtained by calculating the mean square error between the human body posture parameters predicted by the model and the true value posture parameters;

[0132] The root node position estimation loss function is obtained by calculating the mean square error between the three-dimensional coordinate parameters of the root node predicted by the calculation model and the three-dimensional coordinate parameters of the true value root node;

[0133] The bone length estimation loss function is obtained by calculating the mean square error between the bone parameters predicted by the model and the true value bone parameters;

[0134] The position estimation loss function of the joint point is obtained by calculating the mean square error between the joint point position predicted by the calculation model and the true value joint point position parameters; the joint point position predicted by the model is calculated by the posture parameters predicted by the model, the three-dimensional coordinate parameters of the root node predicted by the model and the skeleton parameters predicted by the model;

[0135] The smoothness loss function is used to output a human body posture result that is smooth in time series;

[0136] The ground contact binary classification loss function is obtained by calculating the binary cross entropy between the ground contact parameters predicted by the model and the ground contact parameters of the true value;

[0137] The sliding loss function is used to calculate the corresponding sliding loss value according to the temporal motion speed of the left and right foot joints, and optimize the neural network model according to the sliding loss value.

[0138] According to one or more embodiments of the present disclosure, the human body posture parameters include six-degree-of-freedom information of human body joints.

[0139] The method provided according to one or more embodiments of the present disclosure further includes:

[0140] According to the human body posture parameters, corresponding human body movements are generated and displayed in an extended reality environment.

[0141] According to one or more embodiments of the present disclosure, a motion capture device is provided, comprising:

[0142] An acquisition module, used to acquire human inertial data collected by an extended reality device;

[0143] A processing module, used for inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information;

[0144] Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

[0145] According to one or more embodiments of the present disclosure, there is provided an electronic device, including: at least one memory and at least one processor;

[0146] The at least one memory is used to store program codes, and the at least one processor is used to call the program codes stored in the at least one memory to execute any one of the above methods.

[0147] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the processor is prompted to execute the above method.

[0148] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0149] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0150] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A motion capture method, characterized in that: include: Obtain human inertial data collected by the extended reality device; Inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information; Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

2. The method according to claim 1, characterized in that: The extended reality device includes a head-mounted device, a handheld controller, a waist sensor and a leg sensor; Wherein, the head mounted device, handheld controller, waist sensor and leg sensor include inertial sensors for collecting inertial data of the human body.

3. The method according to claim 1, characterized in that After obtaining the human body inertial data collected by the extended reality device, the method further includes: The human body inertial data is preprocessed to obtain a feature vector for inputting into the motion capture model.

4. The method according to claim 1, characterized in that The training of the neural network model according to a preset loss function includes: The human inertial data for model training is pre-processed and then input into the neural network model, and the time series feature information of the human inertial data is fused according to the feature fusion network of the neural network model; the feature fusion network is composed of a fully connected layer, a normalization layer and a recurrent neural network; Outputting human body posture parameters, global displacement information, human body skeleton parameters and human body ground stepping status through the neural network and the classification network; According to the human body posture parameters, the global displacement information, the human body skeleton parameters, the human body ground contact state and the preset loss function, the corresponding loss value is calculated, and the parameters of the neural network model are adjusted according to the loss value until the loss value is less than the preset value. The training of the neural network model is completed to obtain the motion capture model.

5. The method according to claim 4, characterized in that The preset loss function is obtained by weighted summing of a human posture prediction loss function, a root node position estimation loss function, a bone length estimation loss function, a joint point position estimation loss function, a smoothness loss function, a ground-stepping binary classification loss function, and a sliding loss function; Wherein, the human body posture prediction loss function is obtained by calculating the mean square error between the human body posture parameters predicted by the model and the true value posture parameters; The root node position estimation loss function is obtained by calculating the mean square error between the three-dimensional coordinate parameters of the root node predicted by the calculation model and the three-dimensional coordinate parameters of the true value root node; The bone length estimation loss function is obtained by calculating the mean square error between the bone parameters predicted by the model and the true value bone parameters; The position estimation loss function of the joint point is obtained by calculating the mean square error between the joint point position predicted by the calculation model and the true value joint point position parameters; the joint point position predicted by the model is calculated by the posture parameters predicted by the model, the three-dimensional coordinate parameters of the root node predicted by the model and the skeleton parameters predicted by the model; The smoothness loss function is used to output a human body posture result that is smooth in time series; The ground contact binary classification loss function is obtained by calculating the binary cross entropy between the ground contact parameters predicted by the model and the ground contact parameters of the true value; The sliding loss function is used to calculate the corresponding sliding loss value according to the temporal motion speed of the left and right foot joints, and optimize the neural network model according to the sliding loss value.

6. The method according to claim 4, characterized in that The human body posture parameters include six-degree-of-freedom information of human body joints.

7. The method according to claim 6, characterized in that Also includes: According to the human body posture parameters, corresponding human body movements are generated and displayed in an extended reality environment.

8. A motion capture device, characterized in that: include: An acquisition module, used to acquire human inertial data collected by an extended reality device; A processing module, used for inputting the human body inertial data into a pre-trained motion capture model to obtain human body posture information; Among them, the motion capture model is obtained after training the neural network model according to a preset loss function; the motion capture model includes one or more of a neural network for predicting human posture, a neural network for predicting global displacement information, a neural network for predicting human skeletal parameters and a classification network for predicting human body ground stepping status.

9. An electronic device, comprising: at least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, wherein when the program codes are executed by a computer device, the computer device is prompted to execute the method according to claim 1 .

Images