Motion capture method and device based on AR glasses and electronic equipment

AR glasses combine multiple sensor data to build a three-dimensional point cloud model, solving the problem of limited accuracy of motion capture under a single camera, and achieving high-precision motion capture and virtual interactive experience.

CN119068151APending Publication Date: 2024-12-03GUDONG TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411200414.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing motion capture methods rely on a single external camera, which is affected by ambient light, reducing the accuracy of motion capture.

Method used

Depth images are obtained through AR glasses, combined with multi-source data from inertial measurement units, optical sensors and pressure sensors, a three-dimensional point cloud model is built, spatially matched and fusion is performed, and a realistic user action model is generated, which is mapped to the virtual environment in real time.

Benefits of technology

Improves motion capture accuracy, reduces errors in a single data source, provides an immersive interactive experience, and enhances the authenticity and detailed performance of virtual interactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068151A_ABST
    Figure CN119068151A_ABST
Patent Text Reader

Abstract

The invention discloses a motion capture method and device based on AR glasses and electronic equipment, and relates to the field of AR. The method comprises the following steps: acquiring a depth image of a user; constructing a three-dimensional point cloud model of the user according to the depth image; sensor data of sensors are obtained, the sensors comprise an inertial measurement unit, an optical sensor and a pressure sensor, and the sensor data comprise inertial data, image data and pressure data; performing spatial matching on the inertial data, the image data and the pressure data to obtain user action information; fusing the user action information to the three-dimensional point cloud model to obtain a user action model; and sending the user action model to a display interface of the VR glasses for display. By implementing the technical scheme provided by the invention, the dynamic behavior of the user is more comprehensively captured by combining various sensor data, the error of a single data source is reduced, and the overall motion capture precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a motion capture method, device and electronic equipment based on AR glasses. Background Art

[0002] With the advancement of science and technology, augmented reality (AR) technology has been widely used in many fields, especially in fitness and sports training. AR glasses can render the user and the environment in real time, providing an immersive interactive experience, which is especially important for fitness training. With AR glasses, users can instantly see their own performance and improvement suggestions at any location, greatly improving the efficiency and motivation of training.

[0003] Currently, traditional motion capture methods mainly rely on a single external camera to track the user's movements. These methods are often limited by ambient light, which reduces the accuracy of motion capture.

[0004] Therefore, there is an urgent need for a motion capture method, device and electronic equipment based on AR glasses. Summary of the invention

[0005] The present application provides a motion capture method, device and electronic device based on AR glasses, which can more comprehensively capture the user's dynamic behavior by combining multiple sensor data, reduce the error of a single data source, and improve the overall motion capture accuracy.

[0006] In a first aspect of the present application, a motion capture method based on AR glasses is provided, the method comprising: obtaining a depth image of a user; constructing a three-dimensional point cloud model of the user according to the depth image; obtaining sensor data of a sensor, the sensor comprising an inertial measurement unit, an optical sensor and a pressure sensor, the sensor data comprising inertial data, image data and pressure data, the inertial data corresponding to the inertial measurement unit, the image data corresponding to the optical sensor, and the pressure data corresponding to the pressure sensor; spatially matching the inertial data, the image data and the pressure data to obtain user motion information; fusing the user motion information into the three-dimensional point cloud model to obtain a user motion model; and sending the user motion model to a display interface of the VR glasses for display.

[0007] By adopting the above technical solution, the user's depth image is obtained to construct a three-dimensional point cloud model of the user, accurately capturing and modeling the user's body structure. Comprehensive user motion information is obtained by combining multi-source sensor data from inertial measurement units, optical sensors, and pressure sensors. Inertial data, image data, and pressure data are spatially matched to achieve accurate perception and characterization of user movements. The user motion information is then fused into the three-dimensional point cloud model to generate a realistic user motion model that truly reproduces the user's motion posture. Finally, the user motion model is sent to the VR glasses for display, realizing real-time mapping of user movements in the real world into the virtual environment, providing an immersive interactive experience, and more comprehensively capturing the user's dynamic behavior by combining multiple sensor data, reducing the error of a single data source, and improving the overall motion capture accuracy.

[0008] Optionally, constructing the three-dimensional point cloud model of the user based on the depth image specifically includes: converting the depth image into three-dimensional point cloud data to obtain an initial three-dimensional point cloud; removing the background of the initial three-dimensional point cloud, and extracting a foreground point cloud containing the user's body area; clustering the foreground point cloud using a point cloud segmentation algorithm to obtain a plurality of point cloud clusters, wherein the point cloud clusters represent various body parts of the user; matching each of the point cloud clusters with a preset human body model to obtain a refined point cloud model corresponding to each of the point cloud clusters, wherein the refined point cloud model is a point cloud model of various body parts of the user; and spatially stitching the refined point cloud models to obtain the three-dimensional point cloud model.

[0009] By adopting the above technical solution, the user's body point cloud representation containing rich spatial information is obtained by converting the depth image into three-dimensional point cloud data. Extracting the foreground point cloud can eliminate background interference and highlight the user's body area. Clustering the foreground point cloud can identify various parts of the user's body and achieve a fine division of the human body structure. Each point cloud cluster is matched with a preset human body model to obtain a fine point cloud model of each part of the corresponding body, further improving the accuracy and detail expression of human body modeling. Finally, a complete three-dimensional point cloud model is generated through spatial splicing, laying a high-quality human body model foundation for subsequent motion capture.

[0010] Optionally, the user motion information includes posture data and motion data of the user, and the spatial matching of the inertial data, the image data and the pressure data to obtain the user motion information specifically includes: time synchronizing the inertial data of the inertial measurement unit with the image data of the optical sensor to obtain target inertial data and target image data, the target inertial data being the data after time synchronization with the inertial data, and the target image data being the data after time synchronization with the image data; spatially aligning the pressure data of the pressure sensor, the target inertial data and the target optics to obtain the pressure distribution of various body parts of the user; and determining the posture data and motion data of the user according to the pressure distribution to obtain the user motion information.

[0011] By adopting the above technical solution, the inertial data and image data are synchronized in time, eliminating the data deviation caused by the inconsistency of sensor sampling frequency and timestamp, and aligning the multi-source data in the time dimension. The pressure data, synchronized inertial data and image data are spatially registered, unified into the same coordinate system, and the spatial mapping relationship between sensor data and user body parts is established. Based on the pressure distribution information, the user's posture data and motion data are accurately determined to obtain comprehensive and fine-grained user motion information.

[0012] Optionally, fusing the user motion information into the three-dimensional point cloud model to obtain the user motion model specifically includes: determining motion parameters of various parts of the user's body based on the user motion information; and applying the motion parameters to corresponding body parts in the three-dimensional point cloud model to obtain the user motion model.

[0013] By adopting the above technical solution, the motion parameters are applied to the corresponding body parts of the 3D point cloud model to generate a realistic user motion model. This method deeply integrates the user motion information with the 3D point cloud model, and realizes the fine control of the human body model by the motion data. By mapping the high-level semantic motion information to the low-level geometric deformation, the 3D point cloud model of the static human body is brought to life, showing a coherent and natural dynamic effect. At the same time, the fused user motion model inherits the authenticity and accuracy of the 3D point cloud model, and the visual effect is more realistic, which provides a guarantee for immersive virtual interaction.

[0014] Optionally, determining the motion parameters of each part of the user's body based on the user motion information specifically includes: determining the rotation matrix and translation vector of each part of the user's body based on the posture data; determining the joint angle change and muscle deformation amount of each part of the user's body based on the motion data; and using the rotation matrix, the translation vector, the joint angle change and the muscle deformation amount as the motion parameters.

[0015] By adopting the above technical solution, by combining the rotation matrix and the translation vector, the spatial position and direction of each part of the user's body can be accurately represented; by calculating the change in joint angle and muscle deformation, more detailed and realistic body dynamics can be simulated.

[0016] Optionally, applying the motion parameters to corresponding body parts in the three-dimensional point cloud model to obtain the user motion model specifically includes: performing rotational transformation and translational transformation on each body part in the three-dimensional point cloud model according to the rotation matrix and the translation vector to obtain a transformed three-dimensional point cloud model; adjusting the joint parts in the transformed three-dimensional point cloud model according to the joint angle change, and deforming the muscle area in the transformed three-dimensional point cloud model according to the muscle deformation amount to obtain the user motion model.

[0017] By adopting the above technical scheme, the three-dimensional point cloud model is geometrically transformed according to the rotation matrix and translation vector, the rigid body motion of each part of the human body is accurately controlled, and the overall posture adjustment of the human body model is realized. The joint angle change is used to adjust the transformed three-dimensional point cloud model, simulating the rotational motion of the human joints to make the movement more natural. The soft tissue area of ​​the transformed three-dimensional point cloud model is deformed according to the muscle deformation, simulating the deformation of the skin and clothing caused by muscle contraction and relaxation, and enhancing the realism of the character image. By applying the motion parameters in layers to the three-dimensional point cloud model, the motion control and deformation simulation of the three-dimensional point cloud model are realized, and a highly realistic user action model is generated. This method makes full use of the fine geometric representation of the three-dimensional point cloud model, and combines the simulation technology of kinematics and soft body dynamics, so that the action model reaches a high level in terms of motion coherence, authenticity and detail expression, providing high-quality visual presentation for immersive virtual interaction.

[0018] Optionally, after sending the user action model to the display interface of the VR glasses for display, the method further includes: acquiring voice data collected by a voice sensor; recognizing the voice data to obtain the user's voice command; and controlling the VR glasses to perform corresponding operations according to the voice command.

[0019] By adopting the above technical solution, by obtaining the user's voice data and converting the voice into text instructions using voice recognition technology, the function of controlling VR glasses by voice is realized, providing a natural and convenient human-computer interaction method. Users can issue commands by voice, such as scene switching, object selection, menu operation, etc., without manual operation, which enhances the sense of immersion and convenience of operation. This method combines voice interaction with visual presentation to enrich the means of interaction between users and virtual environments. Users feel the dual stimulation of vision and hearing in the virtual environment, and can actively control and explore the environment through voice to obtain a more real and natural interactive experience. The addition of voice interaction improves the ease of use and intelligence level of the system, allowing users to focus more on the virtual content itself without being disturbed by complex operation methods, thereby bringing a smoother and more immersive virtual reality experience.

[0020] In a second aspect of the present application, a motion capture device based on AR glasses is provided, the device comprising: an acquisition module and a processing module, the acquisition module being used to acquire a depth image of a user; the processing module being used to construct a three-dimensional point cloud model of the user based on the depth image; the acquisition module being also used to acquire sensor data of a sensor, the sensor comprising an inertial measurement unit, an optical sensor and a pressure sensor, the sensor data comprising inertial data, image data and pressure data, the inertial data corresponding to the inertial measurement unit, the image data corresponding to the optical sensor, and the pressure data corresponding to the pressure sensor; the processing module being used to spatially match the inertial data, the image data and the pressure data to obtain user motion information; the processing module being also used to fuse the user motion information into the three-dimensional point cloud model to obtain a user motion model; the processing module being also used to send the user motion model to a display interface of the VR glasses for display.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By acquiring the user's depth image, a 3D point cloud model of the user is constructed to accurately capture and model the user's body structure. Combine multi-source sensor data from inertial measurement units, optical sensors, and pressure sensors to obtain comprehensive user motion information. Spatial matching of inertial data, image data, and pressure data is performed to achieve accurate perception and characterization of user motion. The user motion information is then integrated into the 3D point cloud model to generate a realistic user motion model that truly reproduces the user's motion posture. Finally, the user motion model is sent to the VR glasses for display, which enables real-time mapping of user motion in the real world into the virtual environment, providing an immersive interactive experience. By combining multiple sensor data, the user's dynamic behavior can be more comprehensively captured, reducing the error of a single data source and improving the overall motion capture accuracy.

[0024] 2. By converting the depth image into three-dimensional point cloud data, a point cloud representation of the user's body containing rich spatial information is obtained. Extracting the foreground point cloud can eliminate background interference and highlight the user's body area. Clustering the foreground point cloud can identify various parts of the user's body and achieve a fine division of the human body structure. Match each point cloud cluster with the preset human body model to obtain a fine point cloud model of each part of the corresponding body, further improving the accuracy and detail expression of human body modeling. Finally, a complete three-dimensional point cloud model is generated through spatial splicing, laying a high-quality human body model foundation for subsequent motion capture.

[0025] 3. Time-synchronize inertial data and image data to eliminate data deviations caused by inconsistent sensor sampling frequency and timestamps, and align multi-source data in the time dimension. Spatial registration is performed on pressure data, synchronized inertial data, and image data, unifying them to the same coordinate system to establish a spatial mapping relationship between sensor data and user body parts. Based on the pressure distribution information, the user's posture data and motion data are accurately determined to obtain comprehensive and fine-grained user motion information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a motion capture method based on AR glasses disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a motion capture device based on AR glasses disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0027] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0028] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0029] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0031] This application provides a motion capture method based on AR glasses, referring to Figure 1 , Figure 1 1 is a flow chart of a motion capture method based on AR glasses provided in an embodiment of the present application. The method is applied to a server, including steps S101 to S106, which are as follows: Step S101: Acquire a depth image of the user.

[0032] In step S101, the server obtains multiple frames of depth images from the depth sensor equipped on the AR glasses. The depth sensor can be a camera, a structured light camera, or a stereo camera. The AR glasses establish a communication connection with the server via a wireless network. Before wearing the AR glasses, the user needs to use the depth sensor of the AR glasses to take pictures. The depth sensor on the AR glasses continuously captures the user's depth image at a certain frame rate to obtain a continuous depth image sequence. The AR glasses transmit the collected depth image to the server in real time. The server receives the depth image from the AR glasses.

[0033] Step S102: constructing a three-dimensional point cloud model of the user according to the depth image.

[0034] In step S102, the depth image is converted into three-dimensional point cloud data to obtain an initial three-dimensional point cloud; the background of the initial three-dimensional point cloud is removed to extract a foreground point cloud containing the user's body area; the foreground point cloud is clustered using a point cloud segmentation algorithm to obtain a plurality of point cloud clusters, wherein the point cloud clusters represent various body parts of the user; each of the point cloud clusters is matched with a preset human body model to obtain a refined point cloud model corresponding to each of the point cloud clusters, wherein the refined point cloud model is a point cloud model of various body parts of the user; and the refined point cloud models are spatially spliced ​​to obtain the three-dimensional point cloud model.

[0035] Specifically, the server converts each frame of the depth image into three-dimensional point cloud data. The depth image is essentially a two-dimensional matrix, and each element of the two-dimensional matrix represents the depth value of the corresponding pixel point. The server converts the pixel points into points in the three-dimensional space through back projection transformation according to the internal parameters of the depth camera (such as focal length, principal point coordinates) and the depth value. The set of these three-dimensional points is the initial three-dimensional point cloud, which contains the geometric information of the entire shooting scene. Since this scheme focuses on the user's body parts, the background objects in the shooting scene will interfere with subsequent processing, so they need to be removed. The background removal method used in this scheme is the distance threshold method. In the distance threshold method, a preset distance threshold is pre-set inside the server, and the points that are more than the preset distance threshold from the depth sensor are regarded as background points, and they are removed from the initial three-dimensional point cloud. After removing the background points, the point cloud obtained mainly contains the user's body area, which is called the foreground point cloud. The server uses a point cloud segmentation algorithm to cluster the foreground point cloud and divide the foreground point cloud into multiple point cloud clusters. Each point cloud cluster corresponds to a part of the user's body, such as the head, torso, left arm, and right arm. The point cloud segmentation algorithm adopted in this application is the K-means clustering algorithm, which clusters the foreground point cloud into K point cloud clusters. The value of K is set according to the number of human body parts, and this application does not limit this.

[0036] Next, the server matches the segmented point cloud clusters with the preset human body model. The preset human body model is a general 3D human body model that contains the geometric shape and topological structure of each body part. The server registers each point cloud cluster with the corresponding part in the human body model to obtain a fine point cloud model of the part. The registration process uses the iterative closest point (ICP) algorithm to estimate the optimal transformation matrix by minimizing the distance between two point clouds. The fine point cloud model retains the detailed features of human body parts, such as joints and muscles. Finally, the server spatially splices the fine point cloud models of each part to obtain a complete 3D point cloud model of the user. The splicing process needs to determine the relative position and posture of each part according to the topological structure of the human body model. Each fine point cloud model is aligned to the correct position through rigid body transformation, and then the point cloud is fused and smoothed to finally obtain a continuous and complete 3D point cloud model.

[0037] For example, the server receives a frame of depth image from AR glasses with a resolution of 640×480. The server first converts the depth image into an initial three-dimensional point cloud containing 300,000 points. The background points are removed by the distance threshold method to obtain a foreground point cloud containing 150,000 points. The server then uses the K-means clustering algorithm to divide the foreground point cloud into 8 point cloud clusters, corresponding to the head, torso, left arm, right arm, left leg, and right leg. Then, each point cloud cluster is ICP-aligned with the corresponding part in the preset human body model to obtain 8 fine point cloud models. Finally, the server splices the 8 fine point cloud models into a complete three-dimensional point cloud model through rigid body transformation and point cloud fusion.

[0038] Step S103: Acquire sensor data of the sensor, wherein the sensor includes an inertial measurement unit, an optical sensor and a pressure sensor, and the sensor data includes inertial data, image data and pressure data, the inertial data corresponds to the inertial measurement unit, the image data corresponds to the optical sensor, and the pressure data corresponds to the pressure sensor.

[0039] In step S103 , the server acquires sensor data, including inertial data from an inertial measurement unit (IMU), image data from an optical sensor, and pressure data from a pressure sensor.

[0040] Specifically, the server obtains inertial data from an inertial measurement unit (IMU). IMU consists of an accelerometer, a gyroscope, a magnetometer, etc., and can measure the acceleration, angular velocity, and direction of an object. In this application, the IMU is worn in the form of a strap on key parts of the user's body, such as the head, torso, and limbs. When the user moves, the IMU moves and rotates accordingly, generating corresponding acceleration and angular velocity. The server receives data packets sent by the IMU at a certain frequency (such as 100Hz). Each data packet contains a timestamp, a three-axis acceleration, and a three-axis angular velocity field. The server obtains image data from an optical sensor. Optical sensors such as ordinary cameras, depth cameras, and infrared cameras are built into AR glasses to capture local movements of the user's hands, face, etc. The server receives image streams sent by the optical sensor. For example, a user wears a pair of AR glasses, and an infrared structured light depth camera is embedded in the front end of the glasses, which can collect a depth map (image data) of the user's hand. The server obtains pressure data from a pressure sensor. The pressure sensor can measure the magnitude and distribution of pressure generated when the human body comes into contact with the external environment. The pressure sensor is also worn in the form of a strap on the key parts of the user's body, such as the head, torso, and limbs. The server collects the output of the pressure sensor through the wireless network interface, and each output includes the sensor ID and pressure value fields.

[0041] Step S104: spatially matching the inertial data, the image data and the pressure data to obtain user motion information.

[0042] In step S104, the user motion information includes the posture data and motion data of the user, and the spatial matching of each of the sensor data to obtain the user motion information specifically includes: time-synchronizing the inertial data of the inertial measurement unit with the image data of the optical sensor to obtain target inertial data and target image data, wherein the target inertial data is the data after time synchronization of the inertial data, and the target image data is the data after time synchronization of the image data; spatially aligning the pressure data of the pressure sensor, the target inertial data and the target optics to obtain the pressure distribution of each body part of the user; and determining the posture data and motion data of the user according to the pressure distribution to obtain the user motion information.

[0043] Specifically, since different sensor data are collected and transmitted independently, there may be deviations in the timestamps of different sensor data, and direct fusion will lead to time mismatch problems. Therefore, the server first needs to synchronize the inertial data of the IMU and the image data of the optical sensor. The server uses the Kalman filter, takes the IMU data as the state equation, and the image data as the observation equation, and aligns their timestamps through the optimal estimation algorithm. The synchronized inertial data and image data are aligned on the time axis, recorded as target inertial data and target image data. After time synchronization, the target inertial data and target image data correspond one-to-one in time, describing the movement state of the user's body at the same moment.

[0044] The server spatially registers the pressure data of the pressure sensor, the target inertial data, and the target image data to obtain the pressure distribution of each part of the user's body. Spatial registration is the process of aligning different sensor data to the same spatial coordinate system. The server first calibrates the relative position and direction of each sensor and establishes a coordinate transformation relationship. Then, the server maps the pressure data to the body surface, the target inertial data to the body joints, and the target image data to the body contour. By projecting the three types of data onto the three-dimensional point cloud model of the user's body, the server realizes the spatial registration of different sensor data. After registration, the server can calculate the size and distribution of pressure on each part of the body. Finally, the server outputs the user's posture data and motion data based on the pressure distribution. The posture data is the position and orientation of each part of the body in three-dimensional space, and the motion data describes the overall movement form and content of the body, such as walking, running, jumping, squatting, waving, and grabbing.

[0045] For example, take a virtual fitness application. The user is equipped with 5 IMUs (head, torso, left arm, right arm, left leg, right leg), 2 depth cameras (front and rear), and 4 pressure sensor arrays (left foot, right foot, left hand, right hand). The user performs aerobic exercise on the treadmill for 30 minutes, and the server captures and analyzes the user's running movements in real time. First, the server aligns the timestamps of the IMU (100Hz) and the depth camera (60Hz) to obtain a unified motion sequence. Then, the server estimates the relative position of the IMU and the camera in the user's body coordinate system, and aligns the inertial data and image data into a unified space. Next, the server maps the pressure distribution map of the soles of the feet and palms onto the three-dimensional point cloud model to obtain the overall mechanical state.

[0046] Step S105: Fusing the user action information into the three-dimensional point cloud model to obtain a user action model.

[0047] In step S105, the motion parameters of each body part of the user are determined according to the user motion information; the motion parameters are applied to the corresponding body parts in the three-dimensional point cloud model to obtain the user motion model.

[0048] Specifically, the server determines the motion parameters of each part of the body based on the user's motion information. The user's motion information includes posture data and motion data, which respectively describe the spatial configuration of the body when static and the motion form when dynamic. The server applies the motion parameters to the 3D point cloud model and generates a user motion model through the skeletal binding technology.

[0049] In a possible implementation, motion parameters of various parts of the user's body are determined based on the user motion information, specifically including: determining the rotation matrix and translation vector of various parts of the user's body based on the posture data; determining the joint angle change and muscle deformation amount of various parts of the user's body based on the motion data; and using the rotation matrix, the translation vector, the joint angle change and the muscle deformation amount as the motion parameters.

[0050] Specifically, the server determines the rotation matrix and translation vector of each part of the user's body based on the posture data. The posture data describes the spatial position and direction of each part of the body, reflecting the configuration of the body when it is static. The server converts these spatial positions and directions into motion parameters in algebraic form. For each joint, the server calculates its rotation matrix and translation vector relative to the parent joint. The rotation matrix is ​​a 3x3 matrix that describes the direction transformation of the joint. The translation vector is a 3x1 vector that describes the position transformation of the joint. Through layer-by-layer recursion, the server calculates the local transformation of each joint from bottom to top, and finally obtains the rotation matrix and translation vector of each part of the whole body. The server determines the joint angle change and muscle deformation of each part of the user's body based on the motion data. The motion data describes the morphological changes of the body during the movement, reflecting the rotation of the joints and the contraction of the muscles. The server converts these physiological motion information into physical parameters. For each joint, the server calculates the joint angle change relative to the initial state. The joint angle change is a 3x1 vector, which represents the rotation angle of the joint around the three local coordinate axes. For each muscle, the server calculates its muscle shape relative to its resting state. Muscle shape is a scalar that indicates the degree of muscle contraction. When a muscle contracts, its length decreases and its volume increases, exerting a pulling force on the bones and driving joint movement. By integrating the various muscle shape variables, the server obtains a dynamic description of body movement.

[0051] The server combines the posture parameters and motion parameters to generate a complete set of motion parameters. The rotation matrix, translation vector, joint angle change, and muscle deformation together constitute the low-level representation of human motion. The rotation matrix and translation vector describe the position and direction of the body in the global coordinate system and are parameters at the kinematic level. The joint angle change and muscle deformation describe the relative movement of the internal structure of the body and are parameters at the dynamic level.

[0052] In a possible implementation, the applying the motion parameters to the corresponding body parts in the three-dimensional point cloud model to obtain the user motion model specifically includes: performing rotational transformation and translational transformation on each body part in the three-dimensional point cloud model according to the rotation matrix and the translation vector to obtain a transformed three-dimensional point cloud model; adjusting the joint parts in the transformed three-dimensional point cloud model according to the joint angle change, and deforming the muscle area in the transformed three-dimensional point cloud model according to the muscle deformation amount to obtain the user motion model.

[0053] Specifically, the server performs a rigid body transformation on the 3D point cloud model according to the rotation matrix and the translation vector to obtain a transformed 3D point cloud model. The 3D point cloud model consists of a large number of discrete points, each of which represents a sampling position on the human body surface. The server applies the rotation matrix and the translation vector to each point and performs a linear transformation on its coordinates, namely, a rotation transformation and a translation transformation. The rotation matrix describes the rotation of a point around the origin, and the translation vector describes the displacement of a point along the coordinate axis. By multiplying the point coordinates in the local coordinate system by the rotation matrix and adding the translation vector, the server obtains the new coordinates of the point in the global coordinate system. The rigid body transformation maintains the relative position relationship between the points without changing the shape and size of the object. The server applies the rigid body transformation to the point clouds of various parts of the body in turn to obtain the overall transformed 3D point cloud model. The transformed 3D point cloud model reflects the kinematic state of the body, but the joints have not yet rotated and the muscles have not yet deformed.

[0054] The server adjusts the joint parts in the transformed 3D point cloud model according to the change in joint angle. Human joints are hubs that connect adjacent body parts and have a certain degree of freedom. The change in joint angle describes the rotation of the joint within its degree of freedom. The server decomposes the change in joint angle into three rotational components in orthogonal directions, rotating around the horizontal axis, vertical axis, and anterior-posterior axis of the joint respectively. For each joint, the server determines the corresponding point cloud area and divides the area into two parts, which belong to two adjacent body parts of the joint. The server fixes one part and rotates the other part so that it rotates a given angle relative to the fixed part. In this way, the server simulates the rotation effect of the joint, resulting in relative movement between body parts. After the joint adjustment, the transformed 3D point cloud model presents a more natural posture, but the shape of the muscles is still relatively stiff.

[0055] The server deforms the muscle area in the transformed 3D point cloud model according to the muscle deformation. The muscle deformation describes the degree of change in length and thickness when the muscle contracts. The server uses a physics-based deformation algorithm and uses the muscle deformation as a boundary condition to simulate the stress distribution inside the muscle. For each muscle, the server determines the corresponding point cloud area and interpolates a volume mesh inside the area. Then, the server applies external force to the nodes of the volume mesh to cause deformation displacement under the action of stress. The magnitude and direction of the external force are controlled by the muscle deformation. By solving the simplified elastic mechanics equation, the server can obtain the deformation displacement of the muscle surface point. The deformation displacement is applied to the original point coordinates, and the server obtains the deformed muscle shape.

[0056] By integrating rigid body transformation, joint adjustment, and muscle deformation, the server finally obtains a realistic user motion model. The user motion model incorporates motion parameters based on the 3D point cloud, realizing the transition from static shape to dynamic motion.

[0057] Step S106: Send the user motion model to the display interface of the VR glasses for display.

[0058] In step S101, the server converts the user action model from the internal representation into a universal 3D model format, and compresses the 3D model format to generate a model file with a smaller data volume. The server sends the model file in blocks to the VR glasses through a reliable network transmission protocol. The VR glasses receive and check the integrity of the data, and request the server to retransmit the lost data blocks. After receiving the complete model file, the VR glasses load it into the video memory and use the graphics engine for real-time rendering to present a realistic user action model on the display interface.

[0059] After step S106, the method further includes: acquiring voice data collected by a voice sensor; recognizing the voice data to obtain a user's voice command; and controlling the VR glasses to perform corresponding operations according to the voice command.

[0060] Specifically, the server can collect voice data from the user through a voice sensor (such as a microphone) connected to the VR glasses. The voice sensor converts the collected voice signal into a digital signal and sends the voice data to the server. After receiving the voice data, the server can call the voice recognition engine to analyze and recognize the voice data and convert the voice into corresponding text instructions. Based on the recognized user voice instructions, the server can further parse the meaning of the instructions, determine the user's intention, and control the VR glasses to perform corresponding operations according to the content of the instructions.

[0061] For example, users can control VR glasses through voice commands, such as "zoom in", "zoom out", "switch to the next scene", etc. After the server recognizes these commands, it can send corresponding control commands to the VR glasses, such as sending zoom multiple parameters, switching scene ID, etc., to control the VR glasses to adjust the display of the virtual scene.

[0062] Reference Figure 2 The present application also provides a motion capture device based on AR glasses, which is a server. The server includes an acquisition module 201 and a processing module 202. The acquisition module 201 is used to acquire a depth image of a user; the processing module 202 is used to construct a three-dimensional point cloud model of the user according to the depth image; the acquisition module 201 is also used to acquire sensor data of a sensor, the sensor includes an inertial measurement unit, an optical sensor and a pressure sensor, the sensor data includes inertial data, image data and pressure data, the inertial data corresponds to the inertial measurement unit, the image data corresponds to the optical sensor, and the pressure data corresponds to the pressure sensor; the processing module 202 is used to spatially match the inertial data, the image data and the pressure data to obtain user motion information; the processing module 202 is also used to fuse the user motion information into the three-dimensional point cloud model to obtain a user motion model; the processing module 202 is also used to send the user motion model to the display interface of the VR glasses for display.

[0063] In a possible implementation, the processing module 202 constructs a three-dimensional point cloud model of the user based on the depth image, specifically including: the processing module 202 converts the depth image into three-dimensional point cloud data to obtain an initial three-dimensional point cloud; the processing module 202 removes the background of the initial three-dimensional point cloud and extracts a foreground point cloud containing the user's body area; the processing module 202 clusters the foreground point cloud using a point cloud segmentation algorithm to obtain a plurality of point cloud clusters, and the point cloud clusters represent various body parts of the user; the processing module 202 matches each of the point cloud clusters with a preset human body model to obtain a refined point cloud model corresponding to each of the point cloud clusters, and the refined point cloud model is a point cloud model of each body part of the user; the processing module 202 spatially splices the refined point cloud models to obtain the three-dimensional point cloud model.

[0064] In a possible implementation, the user motion information includes posture data and motion data of the user, and the processing module 202 spatially matches the inertial data, the image data, and the pressure data to obtain the user motion information, specifically including: the processing module 202 time-synchronizes the inertial data of the inertial measurement unit with the image data of the optical sensor to obtain target inertial data and target image data, the target inertial data is the data after time synchronization of the inertial data, and the target image data is the data after time synchronization of the image data; the processing module 202 spatially aligns the pressure data of the pressure sensor, the target inertial data, and the target optics to obtain the pressure distribution of each body part of the user; the processing module 202 determines the posture data and motion data of the user according to the pressure distribution to obtain the user motion information.

[0065] In a possible implementation, the processing module 202 fuses the user motion information into the three-dimensional point cloud model to obtain a user motion model, which specifically includes: the processing module 202 determines the motion parameters of each body part of the user according to the user motion information; the processing module 202 applies the motion parameters to the corresponding body parts in the three-dimensional point cloud model to obtain the user motion model.

[0066] In a possible implementation, the processing module 202 determines the motion parameters of each body part of the user based on the user motion information, specifically including: the processing module 202 determines the rotation matrix and translation vector of each body part of the user based on the posture data; the processing module 202 determines the joint angle change and muscle deformation amount of each body part of the user based on the motion data; the processing module 202 uses the rotation matrix, the translation vector, the joint angle change and the muscle deformation amount as the motion parameters.

[0067] In a possible implementation, the processing module 202 applies the motion parameters to the corresponding body parts in the three-dimensional point cloud model to obtain the user motion model, which specifically includes: the processing module 202 performs rotational transformation and translational transformation on each body part in the three-dimensional point cloud model according to the rotation matrix and the translation vector to obtain a transformed three-dimensional point cloud model; the processing module 202 adjusts the joint parts in the transformed three-dimensional point cloud model according to the joint angle change, and deforms the muscle area in the transformed three-dimensional point cloud model according to the muscle deformation amount to obtain the user motion model.

[0068] In a possible implementation, after the processing module 202 sends the user action model to the display interface of the VR glasses for display, the method further includes: the acquisition module 201 acquires voice data collected by the voice sensor; the processing module 202 recognizes the voice data to obtain the user's voice command; the processing module 202 controls the VR glasses to perform corresponding operations according to the voice command.

[0069] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0070] The present application also provides an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0071] The communication bus 302 is used to realize the connection and communication between these components.

[0072] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0073] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0074] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0075] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a motion capture method based on AR glasses.

[0076] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the memory 305 to store an application program of a motion capture method based on AR glasses. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0077] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0078] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0083] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0084] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A motion capture method based on AR glasses, characterized in that: The method comprises: Get the user's depth image; Constructing a three-dimensional point cloud model of the user according to the depth image; Acquire sensor data of a sensor, the sensor comprising an inertial measurement unit, an optical sensor, and a pressure sensor, the sensor data comprising inertial data, image data, and pressure data, the inertial data corresponding to the inertial measurement unit, the image data corresponding to the optical sensor, and the pressure data corresponding to the pressure sensor; spatially matching the inertial data, the image data, and the pressure data to obtain user motion information; Fusion the user action information to the three-dimensional point cloud model to obtain a user action model; The user motion model is sent to the display interface of the VR glasses for display.

2. The method according to claim 1, characterized in that The constructing the three-dimensional point cloud model of the user according to the depth image specifically includes: Converting the depth image into three-dimensional point cloud data to obtain an initial three-dimensional point cloud; Performing background removal on the initial three-dimensional point cloud to extract a foreground point cloud containing a user's body area; Clustering the foreground point cloud using a point cloud segmentation algorithm to obtain a plurality of point cloud clusters, wherein the point cloud clusters represent various body parts of the user; Matching each of the point cloud clusters with a preset human body model to obtain a fine point cloud model corresponding to each of the point cloud clusters, wherein the fine point cloud model is a point cloud model of each body part of the user; The fine point cloud model is spatially stitched to obtain the three-dimensional point cloud model.

3. The method according to claim 1, characterized in that The user motion information includes the posture data and motion data of the user, and the spatial matching of the inertial data, the image data and the pressure data to obtain the user motion information specifically includes: Time-synchronizing the inertial data of the inertial measurement unit with the image data of the optical sensor to obtain target inertial data and target image data, wherein the target inertial data is the data after time synchronization with the inertial data, and the target image data is the data after time synchronization with the image data; spatially registering the pressure data of the pressure sensor, the target inertial data and the target optics to obtain pressure distribution of each body part of the user; The posture data and motion data of the user are determined according to the pressure distribution to obtain the user motion information.

4. The method according to claim 2, characterized in that: The fusing the user action information into the three-dimensional point cloud model to obtain a user action model specifically includes: Determining motion parameters of various parts of the user's body according to the user motion information; The motion parameters are applied to corresponding body parts in the three-dimensional point cloud model to obtain the user motion model.

5. The method according to claim 4, characterized in that Determining the motion parameters of each part of the user's body according to the user motion information specifically includes: Determine the rotation matrix and translation vector of each part of the user's body according to the posture data; Determining the joint angle changes and muscle deformation amounts of various parts of the user's body according to the motion data; The rotation matrix, the translation vector, the joint angle change and the muscle deformation are used as the motion parameters.

6. The method according to claim 5, characterized in that The applying the motion parameters to the corresponding body parts in the three-dimensional point cloud model to obtain the user motion model specifically includes: According to the rotation matrix and the translation vector, performing rotation transformation and translation transformation on each body part in the three-dimensional point cloud model to obtain a transformed three-dimensional point cloud model; The joint parts in the transformed three-dimensional point cloud model are adjusted according to the joint angle change, and the muscle area in the transformed three-dimensional point cloud model is deformed according to the muscle deformation amount to obtain the user action model.

7. The method according to claim 1, characterized in that After sending the user action model to the display interface of the VR glasses for display, the method further includes: Acquire voice data collected by the voice sensor; Recognizing the voice data to obtain a user's voice command; According to the voice command, the VR glasses are controlled to perform corresponding operations.

8. A motion capture device based on AR glasses, characterized in that: The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to acquire a depth image of the user; The processing module (202) is used to construct a three-dimensional point cloud model of the user based on the depth image; The acquisition module (201) is further used to acquire sensor data of a sensor, wherein the sensor comprises an inertial measurement unit, an optical sensor and a pressure sensor, and the sensor data comprises inertial data, image data and pressure data, wherein the inertial data corresponds to the inertial measurement unit, the image data corresponds to the optical sensor, and the pressure data corresponds to the pressure sensor; The processing module (202) is used to spatially match the inertial data, the image data and the pressure data to obtain user action information; The processing module (202) is further used to fuse the user action information into the three-dimensional point cloud model to obtain a user action model; The processing module (202) is also used to send the user action model to the display interface of the VR glasses for display.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Citation Information

Cited By

  • Man-machine co-fusion system

    CN119987557A

  • Motion evaluation method, device and equipment based on multi-modal perception and medium

    CN120448963A