Calibration method and apparatus for motion capture device, device, and medium

Abstract

The present application provides a calibration method and apparatus for a motion capture device, a device, and a medium. The motion capture device is worn on a human limb, the method is applied to a head-mounted display device, and the head-mounted display device comprises a camera. The method comprises: capturing an image of the motion capture device by means of the camera; on the basis of the image of the motion capture device, determining target attitude information of a device coordinate system relative to a world coordinate system; and calibrating the motion capture device on the basis of the target attitude information. According to the present application, by using an image of a motion capture device to calibrate the motion capture device, the difficulty in calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, the calibration error can be reduced, and the calibration accuracy of the motion capture device can be improved.

Description

Calibration methods, devices, equipment and media for motion capture equipment

[0001] This application claims priority to Chinese Patent Application No. 202311110278.X, filed on August 30, 2023, entitled "Calibration Method, Apparatus, Device and Medium for Motion Capture Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of motion capture technology, and in particular to a calibration method, apparatus, device, and medium for motion capture equipment. Background Technology

[0003] Human motion capture technology has enormous application prospects in fields such as Virtual Reality (VR) and Human-Computer Interaction. Currently, human motion capture solutions primarily use motion capture devices (such as inertial sensors or tracking devices with inertial sensors) to capture human movement data in the real world.

[0004] Before collecting human motion data using motion capture devices, the devices need to be calibrated to ensure the accuracy and precision of the collected data. In related technologies, calibration of motion capture devices requires the use of reference objects and a fixed wearing position. This method undoubtedly increases the difficulty of operation for users, causing inconvenience, and may even introduce calibration errors due to inaccurate device placement, resulting in low calibration accuracy and poor performance.

[0005] Summary of the Invention

[0006] This application provides a calibration method, apparatus, device, and medium for motion capture equipment. By utilizing images from the motion capture equipment to calibrate it, the calibration difficulty of the motion capture equipment can be effectively reduced, the calibration time can be shortened, calibration errors can be reduced, and the calibration accuracy of the motion capture equipment can be improved.

[0007] In a first aspect, embodiments of this application provide a calibration method for a motion capture device, wherein the motion capture device is worn on a human limb, and the method is applied to a head-mounted display device, the head-mounted display device including a camera, the method comprising:

[0008] The camera captures images from the motion capture device.

[0009] Based on the motion capture device image, the target pose information of the device coordinate system relative to the world coordinate system is determined, wherein the device coordinate system is the motion capture device coordinate system;

[0010] The motion capture device is calibrated based on the target posture information.

[0011] Secondly, embodiments of this application provide a calibration device for a motion capture device, wherein the motion capture device is worn on a human limb, and the device is configured in a head-mounted display device, the head-mounted display device including a camera, comprising:

[0012] The image acquisition module is used to acquire images from the motion capture device via the camera;

[0013] The attitude determination module is used to determine the target attitude information of the device coordinate system relative to the world coordinate system based on the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system;

[0014] The device calibration module is used to calibrate the motion capture device based on the target posture information.

[0015] Thirdly, embodiments of this application provide an electronic device, including:

[0016] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the calibration method of the motion capture device described in the first aspect embodiment or its various implementations.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform a calibration method for a motion capture device as described in the first aspect embodiment or its various implementations.

[0018] Fifthly, embodiments of this application provide a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the calibration method for a motion capture device as described in the first aspect embodiment or its various implementations.

[0019] The technical solutions disclosed in the embodiments of this application have at least the following beneficial effects:

[0020] This application utilizes images from motion capture devices worn on the human body to capture motion capture equipment. Based on these images, the target posture information of the device's coordinate system relative to the world coordinate system is determined. Then, the motion capture device is calibrated using this target posture information. By using motion capture device images for calibration, this application effectively reduces the difficulty and time required for calibration, minimizes calibration errors, and improves the calibration accuracy. Therefore, the calibrated motion capture device can enhance limb motion capture performance, thus providing a basis for improving the accuracy of human posture reconstruction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0023] Figure 2 is a schematic diagram of wearing a head-mounted display device, motion capture device and peripheral devices on different parts of the human body according to an embodiment of this application;

[0024] Figure 3 is a flowchart illustrating a calibration method for a motion capture device provided in an embodiment of this application;

[0025] Figure 4 is a flowchart illustrating another calibration method for a motion capture device provided in an embodiment of this application;

[0026] Figure 5 is a schematic diagram of the relationship between the world coordinate system and the device coordinate system provided in the embodiments of this application;

[0027] Figure 6 is a flowchart illustrating another calibration method for a motion capture device provided in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of the relationship between the world coordinate system, the reference coordinate system, and the device coordinate system provided in the embodiments of this application;

[0029] Figure 8 is a flowchart illustrating another calibration method for a motion capture device provided in an embodiment of this application;

[0030] Figure 9 is a schematic diagram of the relationship between the device coordinate system, the world coordinate system, and the limb coordinate system provided in the embodiments of this application;

[0031] Figure 10 is a schematic block diagram of a calibration device for a motion capture device provided in an embodiment of this application;

[0032] Figure 11 is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0035] This application pertains to human motion capture scenarios. Currently, human motion capture solutions primarily utilize motion capture devices to collect real-world human motion data. These devices are generally inertial sensors or tracking devices with inertial sensors. To ensure the accuracy and precision of the motion data collected by these devices, calibration is necessary. Considering that calibration in related technologies requires reference objects and a fixed wearing position, this increases the difficulty of operation for users, causing inconvenience and potentially introducing calibration errors due to inaccurate device placement, resulting in low calibration accuracy and poor performance. Therefore, this application designs a calibration method for motion capture devices. This method effectively reduces the difficulty of calibration, shortens calibration time, reduces calibration errors, and improves the calibration accuracy of the motion capture devices.

[0036] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0037] 1) Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It defines and generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of interactive three-dimensional dynamic visual scenes and simulation of physical behavior in the virtual environment, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0038] 2) Virtual reality devices (VR devices) are terminals that realize virtual reality effects. They can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0039] Optionally, the virtual reality devices described in the embodiments of this application may include, but are not limited to, the following types:

[0040] 2.1) PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices use the data output from the PC to achieve virtual reality effects.

[0041] 2.2) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0042] 2.3) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0043] 3) Virtual field of view: The area in the virtual environment that a user can perceive through the lens in a virtual reality device. The perceived area is represented by the field of view (FOV).

[0044] 4) Augmented Reality (AR): A technology that calculates the camera's pose parameters in the real world (or 3D world, real world) in real time during image acquisition, and adds virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0045] 5) Mixed Reality (MR): A simulated scene that integrates computer-created sensory input (e.g., virtual objects) with sensory input or its representation from a physical setting. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0046] 6) Extended Reality (XR) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices, including virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0047] 7) A virtual scene is a virtual scene displayed (or provided) by an application while running on an electronic device. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. Users can control virtual objects to move within this virtual scene.

[0048] 8) Virtual objects are objects that interact in a virtual scene. They are controlled by the user or a robot program (e.g., an AI-based robot program) and can remain still, move, and perform various behaviors in the virtual scene, such as various characters in a game.

[0049] To clearly illustrate the technical solution of this application, the application scenarios of this application are described below. It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to them:

[0050] For example, Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application. As shown in Figure 1, the application scenario 1000 may include a head-mounted display device 100 and a motion capture device 200. Furthermore, the head-mounted display device 100 and the motion capture device 200 can communicate with each other.

[0051] In some implementations, the head-mounted display device 100 may be an HMD, such as a head-mounted display in a VR all-in-one device, etc. This embodiment does not impose any restrictions on this.

[0052] Furthermore, the head-mounted display device 100 is equipped with cameras to collect data about the surrounding environment. Based on the collected environmental data, it uses the Simultaneous Localization and Mapping (SLAM) algorithm from computer vision to perform tracking and localization. The number of cameras can be at least one; Figure 1 illustrates this with an example of four cameras. Moreover, the type of camera can be a fisheye camera, a regular camera, or other types of cameras; this application does not impose any restrictions on this.

[0053] In some implementations, the motion capture device 200 can be an inertial sensor, or any device with an inertial sensor, etc.

[0054] In some implementations, the motion capture device 200 can be worn on human limbs, such as the limbs and torso. The limbs include the upper and lower limbs. The torso includes the waist, etc. This application does not impose any limitations on the human limbs. For example, as shown in Figure 2, the motion capture device 200 is worn on the lower limbs of a human body. The lower limbs include the thigh and calf. That is, by wearing four motion capture devices 200 on the thighs and calves of a human body respectively, the motion capture devices 200 can collect thigh and calf movement data (collectively referred to as lower limb movement data), and send the lower limb movement data to the head-mounted display device 100 to achieve the purpose of tracking the movement of the human lower limbs.

[0055] It should be noted that the motion capture device 200 worn on the human limbs can collect limb data in the form of 3 degrees of freedom (DoF) data or 6 DoF data.

[0056] Considering that the human upper limbs can also perform different movements, this application allows for the following options: in addition to wearing the motion capture device 200 on the human upper limbs, the peripheral device 300 can also be worn on the human upper limbs. For example, it can be worn on the hand and / or arm. Then, the upper limb motion data collected by the peripheral device 300 is sent to the head-mounted display device 100 to achieve the tracking of the human upper limb movements. For details on how the peripheral device 300 is worn, please refer to Figure 2.

[0057] In some implementations, the peripheral device 300 may be, but is not limited to, a handle, glove, wristband, ring, and other wearable devices.

[0058] Furthermore, the peripheral device 300 is equipped with an inertial sensor, which can provide 6 degrees of freedom data including the position and posture of the human upper limbs.

[0059] It should be noted that the inertial sensor involved in this embodiment specifically refers to an inertial measurement unit (IMU). Furthermore, the IMU can be a six-axis IMU or a nine-axis IMU; no specific limitation is made here.

[0060] It is understood that the embodiments of this application can achieve full-body tracking of a human body by wearing motion capture devices 200 on the limbs of the human body, or wearing peripheral devices 300 on the upper limbs of the human body, or wearing motion capture devices on the lower limbs of the human body.

[0061] It should be understood that the head-mounted display device 100, motion capture device 200, and peripheral device 300 shown in Figures 1 and 2 are merely illustrative and are not intended to limit the scope of this application.

[0062] After introducing the application scenarios of the embodiments of this application, the calibration method, apparatus, device and medium of a motion capture device provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] Figure 3 is a schematic flowchart illustrating a calibration method for a motion capture device according to an embodiment of this application. This embodiment is applicable to calibration scenarios for motion capture devices in human motion capture, and the calibration method can be performed by a calibration device for the motion capture device. This calibration device can consist of hardware and / or software and can be integrated into a head-mounted display device.

[0064] In this embodiment, the head-mounted display device can be any electronic device capable of reproducing human movements. Optionally, the electronic device can be an extended reality (XR) device. The XR device can be a VR device, an augmented reality (AR) device, or a mixed reality (MR) device, etc. This application does not impose specific limitations on the type of electronic device.

[0065] As shown in Figure 3, the method may include the following steps:

[0066] S101 captures images from motion capture devices via a camera.

[0067] Considering that there may be multiple cameras installed on a head-mounted display device, as shown in Figure 1, this application prioritizes using the downward-facing camera located on the head-mounted display device to capture images of the motion capture device when there are multiple cameras on the head-mounted display device. This makes it easier to capture images of the motion capture device worn on the human body using a camera closer to the side of the human body.

[0068] Optionally, before executing S101, the user first wears the head-mounted display device and motion capture device on different parts of the body. For example, the head-mounted display device can be worn on the head, the motion capture device on the lower limbs, and the external device on the upper limbs, as shown in Figure 2 above. Alternatively, the head-mounted display device can be worn on the head, and the motion capture device on the upper and lower limbs. Yet another example is wearing the head-mounted display device on the head, and the motion capture device on the upper limbs, waist, and lower limbs.

[0069] After the head-mounted display, motion capture device, and external devices are put on, the user can activate the head-mounted display, motion capture device, and external devices to establish communication connections between the head-mounted display and each motion capture device, as well as between the head-mounted display and the external devices, laying the foundation for subsequent data communication.

[0070] The specific implementation methods for establishing communication connections between the head-mounted display device and each motion capture device, as well as between the head-mounted display device and external devices, can be found in existing methods, and will not be elaborated upon here.

[0071] Furthermore, the head-mounted display device can capture images of motion capture devices worn on the human body via a camera. Considering that the captured motion capture device images include not only the motion capture devices themselves but also the human body wearing the motion capture devices, a single frame of motion capture device image captured by the camera on the head-mounted display device in this application can include at least one motion capture device. Therefore, the head-mounted display device can distinguish each motion capture device in the motion capture device image based on the motion capture devices in the image and the human body to which each motion capture device belongs. Consequently, the motion capture devices worn on the human body can be calibrated based on each motion capture device in the motion capture device image.

[0072] In some optional embodiments, when this application acquires images of the motion capture device using a camera on a head-mounted display device, it may optionally acquire only one motion capture device image within a time period and calibrate the motion capture device based on that image. This can reduce the image processing complexity of the head-mounted display device and improve the calibration efficiency for the motion capture device.

[0073] For example, in the first time period, an image of the motion capture device on the right calf is captured. In the second time period, an image of the motion capture device on the right thigh is captured. In the third time period, an image of the motion capture device on the left calf is captured. In the fourth time period, an image of the motion capture device on the left thigh is captured. The order of the first, second, third, and fourth time periods can be first time period → second time period → third time period → fourth time period, and can be flexibly adjusted according to actual needs. This application does not impose any restrictions on this.

[0074] In some optional embodiments, considering that the head-mounted display device can support Video See-Through (VST) functionality, users can see a virtualized real space and real objects in proportion to the real world through the head-mounted display device. These virtualized real objects include, but are not limited to, tables and chairs. Therefore, before capturing images of the motion capture device using a camera, the optional head-mounted display device can display a video see-through image in proportion to the real world via the VST function. Simultaneously, guidance information is displayed in any area of ​​the video see-through image, guiding the user to place any motion capture device within it. Thus, when the user places any motion capture device in the video see-through image according to the guidance information, the camera on the head-mounted display device can capture the image of the motion capture device. The advantage of this setup is that by using the VST function to output guidance information to the user, the user can be guided to perform corresponding actions based on the guidance information. The user can also see their own actions through the image perspective function, making the user's actions more visual and the interaction with the head-mounted display device more intuitive, thus improving the ease of use for calibrating the motion capture device.

[0075] It should be noted that, considering that the calibration principles and calibration processes of various motion capture devices worn on the human body are the same, in order to simplify the description, this application will use the calibration of one of the motion capture devices as an example to describe the technical solution of this application in the following embodiments.

[0076] S102, Based on the image from the motion capture device, determine the target pose information of the device coordinate system relative to the world coordinate system. The device coordinate system is the motion capture device coordinate system.

[0077] The device coordinate system can be represented by Li, where L stands for Local and i is the identification information of the motion capture device. This identification information can be any information that can uniquely identify the motion capture device, such as number or name, etc. There are no specific restrictions on it here.

[0078] The world coordinate system refers to the global world coordinate system, which can be represented by G, and G stands for Global.

[0079] In some feasible implementations, the relationship between the world coordinate system and the device coordinate system can be shown in Figure 4. Where O... G For the world coordinate system, O L1 The coordinate system of a motion capture device worn on one thigh of the human lower limb, and O L2 This indicates the coordinate system of another motion capture device worn on the lower leg associated with the aforementioned thigh.

[0080] Optionally, after acquiring the motion capture image from the camera, the head-mounted display device can extract image feature points from the motion capture image. Then, based on the extracted image feature points, the target pose information of the device coordinate system relative to the world coordinate system is determined.

[0081] Considering that the motion capture image captured by the camera may include at least one motion capture device, the head-mounted display device can extract image feature points from the motion capture image using different methods depending on the number of motion capture devices in the image, as follows:

[0082] In the first method, when there is only one motion capture device in the motion capture device image, the head-mounted display device extracts image feature points from the motion capture device image.

[0083] In the second method, when the motion capture device image includes at least two motion capture devices, the head-mounted display device extracts the image feature points of each motion capture device from the motion capture device image.

[0084] In this application, the aforementioned image feature points can be relatively prominent or representative points in the motion capture device image, such as corner points, contour points, dark points in brighter areas, and bright points in darker areas of the motion capture device image. This application does not impose any restrictions on image feature points.

[0085] Furthermore, the image feature points extracted from the motion capture device images are 2D points.

[0086] As an optional implementation, this application may employ a feature point extraction algorithm when extracting image feature points from images captured by a motion capture device. For example, feature point extraction algorithms may include, but are not limited to: SIFT (Scale Invariant Feature Transform) extraction algorithm, SURF (Speeded-Up Robust Features) extraction algorithm, and ORB (Oriented Fast and Rotated BRIEF) extraction algorithm, etc.

[0087] After extracting image feature points from the motion capture device's image, the head-mounted display device can determine the target pose information of the device coordinate system relative to the world coordinate system based on the image feature points extracted from the motion capture device. That is, it determines the pose transformation relationship between the device coordinate system and the world coordinate system.

[0088] Considering that the model, specifications, and shape of motion capture devices worn on human limbs are known, it can be determined that the 3D model of the motion capture device is also known. Therefore, this application, based on extracting image feature points of any motion capture device from its image and its 3D model, can determine the 3D points corresponding to the 2D image feature points of the motion capture device. Furthermore, using preset calculation rules, the attitude transformation relationship between the device coordinate system and the world coordinate system is calculated based on the 2D and 3D points of the motion capture device.

[0089] The aforementioned preset calculation rules can be any algorithm or strategy that can calculate the attitude transformation relationship between the device coordinate system and the world coordinate system based on 2D points and 3D points. This application does not impose any restrictions on this.

[0090] In this application, the aforementioned 2D point can be understood as the projection of the 3D point onto the imaging plane, and the 3D point can be understood as a 3D coordinate point in the world coordinate system.

[0091] Since the pose information of the head-mounted coordinate system relative to the world coordinate system is known, when the aforementioned preset calculation rule is the Perspective-n-Point (PnP) algorithm in computer vision, the head-mounted display device can utilize the PNP algorithm to calculate the target pose information of the device coordinate system relative to the world coordinate system based on n points (2D and 3D points of the motion capture device in the motion capture device image). Then, the target pose information of the device coordinate system relative to the world coordinate system is extracted from the target pose information.

[0092] Typically, PnP algorithms include P3P, P5P, and others. The P3P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on three points. The P5P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on five points. Therefore, in this application, at least three motion capture device feature points are extracted from the motion capture device image, and correspondingly, at least three 3D points need to be determined based on these at least three 2D points. Thus, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device image captured by the camera, different PnP algorithms can be selected based on the number of image feature points of the motion capture device in the motion capture device image.

[0093] S103, calibrate the motion capture device based on the target posture information.

[0094] After determining the target posture information of the device coordinate system relative to the world coordinate system, this application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the limb movements captured by the subsequent calibrated motion capture device are more accurate.

[0095] In some alternative embodiments, the attitude of the motion capture device relative to the world is calibrated. Optionally, the initial attitude of the motion capture device relative to the world is calibrated, or the real-time attitude of the motion capture device relative to the world is calibrated. This application does not impose any limitations on this.

[0096] The calibration method for motion capture devices provided in this application involves capturing images of the motion capture device worn on a human limb using a camera, determining the target posture information of the device's coordinate system relative to the world coordinate system based on the motion capture device images, and then calibrating the motion capture device based on the target posture information. This application utilizes motion capture device images for calibration, effectively reducing the difficulty and time required for calibration, minimizing calibration errors, and improving the calibration accuracy of the motion capture device. Therefore, the calibrated motion capture device can improve the limb motion capture effect, providing conditions for improving the accuracy of human posture reconstruction.

[0097] In another alternative implementation scenario, the motion capture device may include a light-emitting unit. That is, the motion capture device can be an inertial sensor with a light-emitting unit, or any device with both an inertial sensor and a light-emitting unit, such as an optical tracker. Therefore, this application may optionally acquire a light spot image of the light-emitting unit on the motion capture device image via a camera, in order to calibrate the motion capture device based on the light spot image.

[0098] It should be noted that the light-emitting units on the aforementioned motion capture device can be LEDs or other light-emitting devices, and the emitted light can be visible or invisible light, as long as it can be captured by the camera. Furthermore, the aforementioned light-emitting units can be distributed inside the housing of the motion capture device body or in accessories rigidly connected to the motion capture device body, such as the base, according to a preset deployment method. In this application, the aforementioned preset deployment method can be flexibly set according to the shape of the motion capture device body or the accessories rigidly connected to the motion capture device body, and no restrictions are placed on it here.

[0099] Furthermore, when the motion capture device worn on the human limbs includes a light-emitting unit, the limb data collected by the motion capture device includes two types:

[0100] The first type is when the light-emitting unit on the motion capture device is lit up and observed by the camera on the head-mounted display device. At this time, the limb movement data collected by the motion capture device is 6 degrees of freedom (DOF) data, including the position and posture of the human limbs.

[0101] The second category is when the light-emitting unit on the motion capture device is lit up but cannot be observed by the camera on the head-mounted display device. In this case, the limb motion data collected by the motion capture device is 3-degree-of-freedom (DOF) data that includes the posture of the human limbs.

[0102] As shown in the example in Figure 2 above, the lower limb motion data collected by the motion capture device 200 includes two types: First, when the light-emitting unit on the motion capture device 200 is lit and observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 6 degrees of freedom (DOF) data, including the position and posture of the human lower limbs. Second, when the light-emitting unit on the motion capture device 200 is lit but cannot be observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 3 degrees of freedom (DOF) data, including the posture of the human lower limbs.

[0103] The process of calibrating the motion capture device based on spot images, as described above, will be explained in detail below with reference to Figure 5. As shown in Figure 5, the method includes the following steps:

[0104] S201 captures images of light spots from the light-emitting unit on the motion capture device using a camera.

[0105] Considering that each motion capture device worn on the human body has a light-emitting unit, if the light-emitting units of each motion capture device are lit up simultaneously, and the camera on the head-mounted display device captures the light spot images of the light-emitting units of all motion capture devices at the same time, the head-mounted display device may not be able to accurately distinguish which motion capture device each light spot corresponds to based on the light spot images captured by the camera. This could cause confusion in the calibration of the motion capture devices, or even calibration errors.

[0106] For the reasons mentioned above, when calibrating all motion capture devices, this application adopts a time-division lighting method to illuminate the light-emitting units on the motion capture devices. That is, each motion capture device illuminates in a certain order in each cycle, so that the camera on the head-mounted display device can only capture the light spot image of the light-emitting unit on one motion capture device in a time period, and use the light spot image to calibrate the motion capture device, thereby ensuring that the calibration of each motion capture device is more accurate.

[0107] For example, in time period 1, the light-emitting unit of the motion capture device worn on the left thigh is illuminated, while the light-emitting units of the motion capture devices worn on the right thigh, right calf, and left calf are kept off. In time period 2, the light-emitting unit of the motion capture device worn on the left calf is illuminated, while the light-emitting units of the motion capture devices worn on the right thigh, right calf, and left thigh are kept off. In time period 3, the light-emitting unit of the motion capture device worn on the right thigh is illuminated, while the light-emitting units of the motion capture devices worn on the right calf, left thigh, and left calf are kept off. In time period 4, the light-emitting unit of the motion capture device worn on the right calf is illuminated, while the light-emitting units of the motion capture devices worn on the left thigh, left calf, and right thigh are kept off. The order of time periods 1, 2, 3, and 4 can be selected as time period 1 → time period 2 → time period 3 → time period 4, or any other order is possible; no specific restriction is placed on this.

[0108] The light-emitting unit on the motion capture device can be lit manually by the user, or the head-mounted display device can light up the light-emitting unit on the motion capture device by sending a lighting command to the motion capture device. This application does not impose any restrictions on this.

[0109] In some alternative implementations, when capturing the light spot image of the light-emitting unit on the motion capture device in a lit state using a camera, if the user's clothing is loose or other conditions exist, the camera on the head-mounted display device may be unable to capture the light spot image of the light-emitting unit on the motion capture device in a lit state.

[0110] Therefore, after illuminating the light-emitting unit on any motion capture device, this application guides the user to perform a head-down movement, a leg-kicking movement, and / or an arm-raising movement, so that the camera on the head-mounted display device can capture the light spot image of the light-emitting unit on the motion capture device. The leg performing the kicking movement or the arm-raising movement should be the leg or upper limb wearing the illuminated motion capture device. For example, assuming the light-emitting unit on the motion capture device on the left calf is illuminated, the user is guided to kick their left calf. Similarly, assuming the light-emitting unit on the motion capture device on the right forearm is illuminated, the user is guided to raise their right forearm.

[0111] It is understood that the aforementioned head-down movement, leg-kicking movement, and / or arm-raising movement can be a head-down movement; or, it can be a leg-kicking movement; or, it can be an arm-raising movement; or, it can be a head-down movement and an arm-raising movement; or, it can be a head-down movement and a leg-kicking movement, etc., and no restrictions are placed on them here.

[0112] In addition, in this application, when guiding users to make head-down, leg-kicking, and / or arm-raising movements, the guiding voice, guiding text, and / or guiding animation may be output to the user, and no restrictions are placed on them here.

[0113] In some optional embodiments, considering that the head-mounted display device can support Video See-Through (VST) functionality, based on this VST function, users can see a virtualized real space and real objects in proportion to the real world through the head-mounted display device. These virtualized real objects include, but are not limited to, tables and chairs. Therefore, before capturing the light spot image of the light-emitting unit on the motion capture device via the camera, the optional head-mounted display device can display a video see-through image in proportion to the real world through the VST function. Simultaneously, guidance information is displayed in any area of ​​the video see-through image, guiding the user to place any motion capture device with its light-emitting unit lit within the video see-through image. This allows the user to place the motion capture device with its light-emitting unit lit within the video see-through image based on the guidance information. Thus, when the user places any motion capture device with its light-emitting unit lit within the video see-through image according to the guidance information, the camera on the head-mounted display device can capture the light spot image of the light-emitting unit on that motion capture device. The advantage of this setup is that by using the VST function to output guidance information to the user, the user can be guided to perform corresponding actions based on the guidance information. The user can also see their own actions through the image perspective function, making the user's head-down, leg-kicking and / or arm-raising actions visible. The interaction with the head-mounted display device is more intuitive, and the ease of calibration of the motion capture device is improved.

[0114] S202, Based on the spot image, determine the target attitude information of the device coordinate system relative to the world coordinate system.

[0115] Optionally, the head-mounted display device of this application may use preset calculation rules to calculate the target attitude information of the device coordinate system relative to the world coordinate system based on the light spot image.

[0116] Considering that the pose information of the head-mounted coordinate system relative to the world coordinate system is known, this application can utilize the Perspective-n-Point (PnP) algorithm in computer vision to calculate the pose information of the device coordinate system relative to the world coordinate system based on n points (n points corresponding to the light spot in the light spot image). Therefore, this application can calculate the target pose information of the device coordinate system relative to the world coordinate system using the pose estimation algorithm based on the light spot image captured by the camera, given the deployment position of the light-emitting unit on the motion capture device. Then, the pose information is extracted from the target pose information, and the extracted pose information is determined as the target pose information of the device coordinate system relative to the world coordinate system.

[0117] Typically, PnP algorithms include P3P, P5P, and other algorithms. The P3P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on three points. The P5P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on five points. Therefore, in this application, the motion capture device has at least three light spots. Thus, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the light spot images captured by the camera, different PnP algorithms can be selected based on the number of light spots in the light spot image.

[0118] The number of light spots on the motion capture device can be the same as or different from the number of light-emitting units on the motion capture device. If the number of light spots and the number of light-emitting units are the same, then the number of light-emitting units on the motion capture device is at least three. If the number of light spots and the number of light-emitting units on the motion capture device are not equal, then the number of light-emitting units on the motion capture device is at least one.

[0119] In some alternative embodiments, considering that the motion capture device may include a motion capture device body or a motion capture device body and an accessory rigidly connected to the motion capture device body, such as a base, then when the number of light-emitting units on the motion capture device is at least three, as an optional implementation, the at least three light-emitting units can be distributed within the housing of the motion capture device body according to a preset deployment method. Alternatively, one light-emitting unit can be installed within the housing of the motion capture device body according to a preset deployment method, and the remaining light-emitting units can be installed within the base according to a preset deployment method. Or, one light-emitting unit can be installed within the base according to a preset deployment method, and the remaining light-emitting units can be installed within the housing of the motion capture device body according to a preset deployment method. The above preset deployment methods can be flexibly set according to the shape of the motion capture device body and the base, and no limitations are imposed here.

[0120] S203, calibrate the motion capture device based on the target posture information.

[0121] After determining the target posture information of the device coordinate system relative to the world coordinate system, this application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the limb movements captured by the subsequent calibrated motion capture device are more accurate.

[0122] In some alternative embodiments, the attitude of the motion capture device relative to the world is calibrated. Optionally, the initial attitude of the motion capture device relative to the world is calibrated, or the real-time attitude of the motion capture device relative to the world is calibrated. This application does not impose any limitations on this.

[0123] The calibration method for motion capture devices provided in this application involves capturing light spot images of the light-emitting units on the motion capture device worn on the human limb using a camera. Based on these light spot images, the target posture information of the device's coordinate system relative to the world coordinate system is determined. Then, the motion capture device is calibrated based on this target posture information. This application utilizes the light spot images of the light-emitting units on the motion capture device for calibration, effectively reducing the difficulty and time required for calibration. It also reduces calibration errors and improves the calibration accuracy of the motion capture device. Therefore, the calibrated motion capture device can improve the limb motion capture effect, providing conditions for improving the accuracy of human posture reconstruction.

[0124] The calibration method for the motion capture device provided in this embodiment of the application will be further explained below with reference to Figure 6. As shown in Figure 6, the method includes the following steps:

[0125] S301, Based on the inertial measurement unit data sent by the motion capture device, determine the initial attitude information of the device coordinate system relative to the reference coordinate system.

[0126] The reference coordinate system can be a geographic coordinate system or any user-defined coordinate system that differs from the world coordinate system. In this application, the roll angle (also known as the pitch angle) and pitch angle in the reference coordinate system are the same as those in the world coordinate system.

[0127] The aforementioned reference coordinate system can be represented as ref. Furthermore, when the reference coordinate system is a geographic coordinate system, it can be selected as the NE-GY geographic coordinate system.

[0128] In this embodiment, the relationship between the world coordinate system, the reference coordinate system, and the device coordinate system can be illustrated in Figure 7. Wherein, O G For the world coordinate system, O ref As the reference coordinate system, O L1The coordinate system of a motion capture device worn on one thigh of the human lower limb, and O L2 This indicates the coordinate system of another motion capture device worn on the lower leg associated with the aforementioned thigh.

[0129] In some optional embodiments, considering that when the motion capture device is in operation, the inertial sensors (i.e., inertial measurement units) on the motion capture device will send inertial measurement unit data to the head-mounted display device in real time, so that the head-mounted display device can calculate the initial attitude information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data. That is, the attitude transformation relationship between the device coordinate system and the reference coordinate system.

[0130] Since the inertial measurement unit data sent by the motion capture device may include accelerometer data, gyroscope data, and magnetometer data, this application can utilize a preset attitude calculation algorithm, such as a complementary filtering algorithm, to determine the initial attitude information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, gyroscope data, and magnetometer data sent by the motion capture device.

[0131] The inertial measurement unit (IMU) on the aforementioned motion capture device sends IMU data to the head-mounted display device. This can be done when the camera on the head-mounted display device captures a light spot image, at which point the motion capture device begins acquiring IMU data and sending the IMU data; or it can begin at any moment before the camera on the head-mounted display device captures the light spot image, after which the motion capture device continuously acquires and sends IMU data. This application does not impose any limitations on this.

[0132] In this application, based on the inertial measurement unit (IMU) data sent by the motion capture device, the number of initial attitude information values ​​of the device coordinate system relative to the reference coordinate system can be one or more. Specifically, if the inertial sensor on the motion capture device sends IMU data only once, then the calculated initial attitude information value of the device coordinate system relative to the reference coordinate system is one. If the inertial sensor on the motion capture device sends IMU data multiple times, then the calculated initial attitude information value of the device coordinate system relative to the reference coordinate system is multiple. Multiple initial attitude information values ​​can be understood as the ability to calculate an initial attitude information value of the device coordinate system relative to the reference coordinate system for each received IMU data value.

[0133] Furthermore, the aforementioned initial posture information can be the same or different, depending on the user's motion state.

[0134] In this application, "multiple" can be understood as two or more, that is, at least two.

[0135] Optionally, the initial attitude information of the i-th device coordinate system relative to the reference coordinate system in this application can be represented as: Where ref represents the reference coordinate system, t1 represents time t1, and L i Let R represent the coordinate system of the i-th motion capture device, i.e., the coordinate system of the i-th device, and let R represent the posture information.

[0136] S302 captures images of light spots from the light-emitting unit on the motion capture device using a camera.

[0137] S303. Based on the spot image, determine the target pose information of the device coordinate system relative to the world coordinate system. The device coordinate system is the motion capture device coordinate system.

[0138] S304, based on the target attitude information and the initial attitude information, corrects the attitude transformation relationship between the world coordinate system and the reference coordinate system.

[0139] In some optional embodiments, considering that the motion capture device can simultaneously send inertial measurement unit data to the head-mounted display device during the process of the camera on the head-mounted display device acquiring the spot image, the present application can optionally correct the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information determined by the spot image at the same moment and the initial attitude information determined by the inertial measurement unit data. Optionally, this can be achieved by the following formula (1):

[0140] in, G R ref This indicates the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system. This represents the target attitude information of the i-th device coordinate system relative to the world coordinate system at time t1. This represents the initial attitude information of the i-th device coordinate system relative to the reference coordinate system at time t1, where ref represents the reference coordinate system, and L... i Let represent the coordinate system of the i-th motion capture device, i.e. the i-th device coordinate system, G represent the world coordinate system, R represent the attitude information, and -1 represent the inverse.

[0141] Considering the attitude transformation relationship between the world coordinate system and the reference coordinate system, it can be represented by Euler angles. Furthermore, Euler angles can include: yaw angle, pitch angle, and roll angle. Therefore, this application corrects the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information, which can optionally be the correction of the yaw angle, pitch angle, and / or roll angle between the world coordinate system and the reference coordinate system.

[0142] It is understood that the aforementioned heading angle, pitch angle, and / or roll angle may be a heading angle; or a pitch angle; or a roll angle; or a heading angle and a pitch angle; or a heading angle and a roll angle; or a pitch angle and a roll angle; or a heading angle, a pitch angle, and a roll angle. This application does not impose any specific restrictions on these.

[0143] In this application, the aforementioned heading angle can be understood as the angle at which the motion capture device rotates around the X-axis, and the X-axis is perpendicular to the ground. The pitch angle can be understood as the angle at which the motion capture device rotates around the Y-axis, and the roll angle can be understood as the angle at which the motion capture device rotates around the Z-axis.

[0144] In some optional embodiments, considering that when the attitude transformation relationship between the world coordinate system and the reference coordinate system is represented by Euler angles, the pitch and roll angles in the world coordinate system and the reference coordinate system are exactly the same, with only the heading angle parameter differing. Therefore, this application corrects the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information. Specifically, it corrects the heading angle parameter in the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information.

[0145] In this application, the heading angle in the attitude transformation relationship between the world coordinate system and the reference coordinate system is corrected, specifically by: correcting the heading angle transformation relationship in the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information.

[0146] S305 calibrates the motion capture device based on the corrected attitude transformation relationship between the world coordinate system and the reference coordinate system.

[0147] Optionally, after correcting the attitude transformation relationship between the world coordinate system and the reference coordinate system, this application can determine a new target attitude information with high accuracy relative to the world coordinate system based on the corrected attitude transformation relationship and the initial attitude information of the device coordinate system relative to the reference coordinate system. Furthermore, based on this new target attitude information, the motion capture device can be calibrated for its initial or real-time attitude, thereby further improving the calibration accuracy of the motion capture device.

[0148] In some optional embodiments, based on the corrected attitude transformation relationship and the initial attitude information of the device coordinate system relative to the reference coordinate system, the new target attitude information of the device relative to the world coordinate system is determined, which can be achieved by the following formula (2):

[0149] in, This represents the new target attitude information of the i-th device coordinate system relative to the world coordinate system at any time after time t1. G R ref This indicates the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system. This represents the initial attitude information of the i-th device coordinate system relative to the reference coordinate system at any time after time t1, where ref represents the reference coordinate system, and L... i Let G represent the coordinate system of the i-th motion capture device, i.e., the i-th device coordinate system, and let R represent the attitude information.

[0150] The calibration method for motion capture devices provided in this application involves capturing light spot images of the light-emitting units on the motion capture device worn on the human limb using a camera. The target posture information of the device coordinate system relative to the world coordinate system is determined based on the light spot images. Then, the motion capture device is calibrated based on the target posture information. This application utilizes the light spot images of the light-emitting units on the motion capture device for calibration, effectively reducing the difficulty and time required for calibration. It also reduces calibration errors and improves the calibration accuracy of the motion capture device. Therefore, the calibrated motion capture device can improve the limb motion capture effect, providing conditions for improving the accuracy of human posture reconstruction. Furthermore, by determining the initial posture information of the device coordinate system relative to the reference coordinate system and correcting the posture transformation relationship between the world coordinate system and the reference coordinate system based on the target posture information and the initial posture information, the calibration of the motion capture device's posture relative to the world based on the corrected posture transformation relationship can further reduce calibration errors and improve the calibration accuracy of the motion capture device.

[0151] In one optional implementation scenario, after calibrating the motion capture device, this application can further determine the posture transformation relationship between the limb coordinate system and the world coordinate system, and then determine the posture of the limb corresponding to the motion capture device in the world coordinate system based on this posture transformation relationship. The process of determining the posture transformation relationship between the limb coordinate system and the world coordinate system, and determining the posture of the limb in the world coordinate system based on this posture transformation relationship, will be specifically explained below with reference to Figure 8.

[0152] As shown in Figure 8, the method includes the following steps:

[0153] S401, determine the attitude information of the device coordinate system relative to the limb coordinate system.

[0154] In this embodiment of the application, the limb coordinate system specifically refers to the limb coordinate system of the human body, which can be represented as B. iWhere B stands for Body, and i is the identification information of the limb corresponding to the limb coordinate system. This identification information can be any information that can uniquely identify the limb, such as the number or name of an important human bone, etc. There are no specific restrictions on it here.

[0155] As an optional implementation, the relationship between the device coordinate system, world coordinate system, and limb coordinate system can be illustrated as shown in Figure 9. Wherein, O G O represents the world coordinate system. B1 The coordinate system representing a leg, O B2 O represents the limb coordinate system of the lower leg associated with the thigh. L1 This represents the device coordinate system of a motion capture device worn on the thigh, and O. L2 The device coordinate system represents another motion capture device worn on the aforementioned lower leg.

[0156] Optionally, before determining the posture information of the device coordinate system relative to the limb coordinate system, this application first guides the user to perform a preset calibration action (or preset calibration posture) and maintain it for a certain period of time. This preset calibration action ensures that the user's face and the limb wearing the motion capture device have a defined relative posture. The preset calibration action can be, but is not limited to, a T-pose, an N-pose, or other actions that ensure the user's face, chest, and lower limbs (i.e., legs) face the same direction, or other actions that ensure a defined relative posture between the user's face and the limb wearing the motion capture device. Then, the posture information of the device coordinate system relative to the limb coordinate system is determined based on this preset action. The certain period of time in this application can be flexibly set according to actual calculation needs, such as 1 second, 2 seconds, or 3 seconds, etc., and this application does not impose any restrictions on this.

[0157] It should be noted that the T-pose mentioned above can be understood as a posture with both arms extended horizontally to the sides in a T-shape. The N-pose mentioned above can be understood as similar to the attention posture.

[0158] In some alternative implementations, determining the orientation information of the device coordinate system relative to the limb coordinate system may include the following steps:

[0159] Step 1: Under the preset calibration posture, determine the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system, as well as the posture information of the head-mounted coordinate system relative to the world coordinate system, where the head-mounted coordinate system is the coordinate system of the head-mounted display device.

[0160] Among them, the preset calibration posture can be understood as the aforementioned preset calibration action, such as T-pose, N-pose, other actions that can ensure that the user's face, chest and lower limbs (i.e. legs) are facing the same direction, or other actions that ensure that the user's face and the limbs wearing the motion capture device have a certain relative posture.

[0161] In some optional embodiments, the positions of the human limbs and head remain fixed after the user assumes a preset calibration posture. Correspondingly, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is also fixed. That is, at the moment the user performs the preset action, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is a known quantity, and this known quantity is typically a constant, which is the identity matrix.

[0162] In some implementations, since the camera on the head-mounted display device can collect surrounding environmental data in real time, and the head-mounted display device can determine its pose information in real time based on the surrounding environmental data collected by the camera using a computer vision SLAM algorithm, the pose information of the head-mounted display device relative to the world coordinate system can be determined by using a computer vision SLAM algorithm based on the surrounding environmental data collected by the camera. The specific determination process is a conventional technique in this field and will not be elaborated on here.

[0163] Step 2: Based on the attitude transformation relationship between the limb coordinate system and the head-mounted coordinate system, the attitude information of the head-mounted coordinate system relative to the world coordinate system, and the attitude information of the device coordinate system relative to the world coordinate system, determine the attitude information of the device coordinate system relative to the limb coordinate system.

[0164] Optionally, the attitude information of the device coordinate system relative to the limb coordinate system can be determined by the following formula (3):

[0165] in, This represents the attitude information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2. This represents the pose transformation relationship between the coordinate system of the i-th limb and the head-mounted coordinate system at time t2. G R HMD This represents the attitude information of the head-mounted coordinate system relative to the world coordinate system at time t2. B represents the attitude information of the i-th device coordinate system relative to the world coordinate system at time t2. i Let L represent the coordinate system of the i-th limb. i Let represent the coordinate system of the i-th motion capture device, i.e. the coordinate system of the i-th device, G represent the world coordinate system, HMD represent the head-mounted coordinate system, R represent the posture information, and -1 represent the inverse.

[0166] The above in, This represents the attitude information of the i-th device coordinate system relative to the world coordinate system at time t2. G R ref This indicates the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system. This represents the initial attitude information of the device coordinate system relative to the reference coordinate system at time t2, where ref represents the reference coordinate system, and L... i Let G represent the coordinate system of the i-th motion capture device, i.e. the i-th device coordinate system, G represent the world coordinate system, and R represent the posture information.

[0167] It should be understood that t1 refers to the time for acquiring the motion capture device image or light spot image, or the time for acquiring the motion capture device image or light spot image and completing the calibration of the motion capture device; t2 refers to the time for determining the posture of the device coordinate system relative to the limb coordinate system under the preset calibration posture, and t2 can be later than t1, i.e., t1 <t2。

[0168] Considering that the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system has been determined in S304 of the aforementioned embodiment, this attitude transformation relationship is fixed during calibration or after the last calibration and before the next calibration. Furthermore, the inertial measurement unit (IMU) in the motion capture device sends IMU data in real time; therefore, the initial attitude information of the device coordinate system relative to the reference coordinate system at time t2 can be calculated based on the received IMU data. Consequently, based on the initial attitude information of the device coordinate system relative to the reference coordinate system at time t2, and the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, this application can determine the attitude information of the i-th device coordinate system relative to the world coordinate system at time t2 corresponding to the user performing the preset calibration posture.

[0169] The attitude information of the device coordinate system relative to the world coordinate system is determined based on the initial attitude information of the device coordinate system relative to the reference coordinate system at time t2, and the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system. For details, please refer to the above embodiment S305 section, which will not be elaborated on here.

[0170] S402, Based on the attitude information of the device coordinate system relative to the limb coordinate system and the attitude information of the device coordinate system relative to the world coordinate system, determine the attitude transformation relationship between the limb coordinate system and the world coordinate system.

[0171] In some alternative implementations, the pose transformation relationship between the limb coordinate system and the world coordinate system can be determined by the following formula (4):

[0172] in, This represents the pose transformation relationship between the limb coordinate system and the world coordinate system at time t2. This represents the attitude information of the i-th device relative to the world coordinate system at time t2. B represents the pose information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2. i Let L represent the coordinate system of the i-th limb. i Let represent the coordinate system of the i-th motion capture device, i.e. the i-th device coordinate system, G represent the world coordinate system, R represent the attitude information, and -1 represent the inverse.

[0173] Furthermore, the limb coordinate system can be calibrated based on the posture transformation relationship between the limb coordinate system and the world coordinate system.

[0174] It should be understood that the above-mentioned posture transformation relationship between the limb coordinate system and the world coordinate system can be understood as the posture information of the limb coordinate system relative to the world coordinate system, and this posture information can represent the posture of the limb in the world coordinate system.

[0175] In some optional implementations, the determination of the device coordinate system's pose information relative to the limb coordinate system and the execution order of calibrating the motion capture device can be performed by first calibrating the motion capture device and then determining the pose information of the device coordinate system relative to the limb coordinate system, thereby obtaining the pose information of the limb coordinate system relative to the world coordinate system and realizing the corresponding limb motion capture function; or, the determination of the device coordinate system's pose information relative to the limb coordinate system can be performed first and then the calibration operation of the motion capture device can be performed, thereby obtaining the pose information of the limb coordinate system relative to the world coordinate system and realizing the corresponding limb motion capture function. This application does not impose specific limitations on this.

[0176] In this application, when determining the posture information of the device coordinate system relative to the limb coordinate system first, and then performing a calibration operation on the motion capture device, the posture information of the device coordinate system relative to the limb coordinate system at time t2 is cached first. Then, when performing the calibration operation, the motion capture device is calibrated based on the cached posture information of the device coordinate system relative to the limb coordinate system at time t2.

[0177] The above-mentioned first step of determining the attitude information of the device coordinate system relative to the limb coordinate system can be achieved through the following formula (5):

[0178] in, This represents the attitude information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2. This represents the pose transformation relationship between the i-th limb coordinate system and the head-mounted coordinate system at time t2. This pose transformation relationship is a fixed value when the user's limbs are in a preset calibration posture, and this fixed value is usually an identity matrix. G R HMD This represents the attitude information of the head-mounted coordinate system relative to the world coordinate system at time t2. G R ref This represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system at time t2. B represents the initial attitude information of the i-th device coordinate system relative to the reference coordinate system at time t2. i Let L represent the coordinate system of the i-th limb. i Let represent the coordinate system of the i-th motion capture device, i.e. the coordinate system of the i-th device, G represent the world coordinate system, HMD represent the head-mounted coordinate system, ref represent the reference coordinate system, R represent the pose information, and -1 represent the inverse.

[0179] It should be noted that in this application, the limbs can be regarded as fixed in the limb coordinate system, and the pose of the limb coordinate system relative to the world can be equivalent to the pose of the limbs relative to the world coordinate system.

[0180] Furthermore, the motion capture device can be fixed to the motion capture device coordinate system, and the relationship between the motion capture device coordinate system and the reference coordinate system or the world coordinate system is equivalent to the relationship between the motion capture device and the reference coordinate system or the world coordinate system.

[0181] The calibration method for motion capture devices provided in this application involves capturing light spot images of the light-emitting units on the motion capture device worn on a human limb using a camera. Based on these light spot images, the target posture information of the device's coordinate system relative to the world coordinate system is determined. Then, the motion capture device is calibrated based on this target posture information. This application utilizes light spot images of the light-emitting units on the motion capture device for calibration, effectively reducing the difficulty and time required for calibration. It also reduces calibration errors and improves the calibration accuracy of the motion capture device. Therefore, the calibrated motion capture device can improve the limb motion capture effect, providing a basis for improving the accuracy of human posture reconstruction. Furthermore, this application can also calibrate the limb coordinate system based on the posture transformation relationship between the limb coordinate system and the world coordinate system. Based on the calibrated limb coordinate system, the posture of the limb in the world coordinate system is determined, thereby improving the accuracy and correctness of determining the posture of the human limb in the world coordinate system and enhancing the limb motion capture effect.

[0182] The calibration device for a motion capture device according to an embodiment of this application will now be described with reference to Figure 10. Figure 10 is a schematic block diagram of a calibration device for a motion capture device provided in an embodiment of this application.

[0183] The motion capture device is worn on the body's limbs and is mounted on a head-mounted display device, which includes a camera. As shown in Figure 10, the calibration device 500 of the motion capture device includes an image acquisition module 510, a posture determination module 520, and a device calibration module 530.

[0184] The image acquisition module 510 is used to acquire images from the motion capture device through the camera.

[0185] The attitude determination module 520 is used to determine the target attitude information of the device coordinate system relative to the world coordinate system based on the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system;

[0186] The device calibration module 530 is used to calibrate the motion capture device based on the target posture information.

[0187] In one optional implementation of this application embodiment, the attitude determination module 520 is specifically used for:

[0188] Extract image feature points from the motion capture device image;

[0189] Based on the image feature points, the target pose information of the device coordinate system relative to the world coordinate system is determined.

[0190] In one optional implementation of this application embodiment, the motion capture device includes a light-emitting unit and an image acquisition module 510, specifically used for:

[0191] The camera captures the light spot image of the light-emitting unit on the motion capture device.

[0192] Accordingly, the attitude determination module 520 is specifically used for:

[0193] Based on the light spot image, the target attitude information of the device coordinate system relative to the world coordinate system is determined.

[0194] In one optional implementation of this application embodiment, the attitude determination module 520 is further used for:

[0195] Using a pose estimation algorithm, the target pose information of the device coordinate system relative to the world coordinate system is determined based on the light spot image;

[0196] The attitude information is extracted from the target pose information, and the attitude information is determined as the target attitude information of the device coordinate system relative to the world coordinate system.

[0197] In one optional implementation of this application embodiment, the apparatus 500 further includes:

[0198] The first determining module is used to determine the initial attitude information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data sent by the motion capture device.

[0199] Accordingly, the device calibration module 530 is specifically used for:

[0200] Based on the target attitude information and the initial attitude information, the attitude transformation relationship between the world coordinate system and the reference coordinate system is corrected;

[0201] The motion capture device is calibrated based on the corrected attitude transformation relationship between the world coordinate system and the reference coordinate system.

[0202] In an optional implementation of this application embodiment, the device calibration module 530 is further configured to:

[0203] Correct the heading angle in the attitude transformation relationship between the world coordinate system and the reference coordinate system.

[0204] In one optional implementation of this application embodiment, the inertial measurement unit data includes: accelerometer data, gyroscope data, and magnetometer data, and the first determining module is specifically used for:

[0205] Using a complementary filtering algorithm, the initial attitude information of the device coordinate system relative to the reference coordinate system is determined based on the accelerometer data, gyroscope data, and magnetometer data sent by the motion capture device.

[0206] In one optional implementation of this application embodiment, the apparatus 500 further includes:

[0207] The second determining module is used to determine the posture information of the device coordinate system relative to the limb coordinate system;

[0208] The third determining module is used to determine the attitude transformation relationship between the limb coordinate system and the world coordinate system based on the attitude information of the device coordinate system relative to the limb coordinate system and the attitude information of the device coordinate system relative to the world coordinate system.

[0209] In one optional implementation of this application embodiment, the second determining module is specifically used for:

[0210] Under a preset calibration posture, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is determined, as well as the posture information of the head-mounted coordinate system relative to the world coordinate system. The head-mounted coordinate system is the coordinate system of the head-mounted display device.

[0211] Based on the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system, the posture information of the device coordinate system relative to the limb coordinate system is determined.

[0212] In an optional implementation of this application embodiment, the second determining module is further configured to:

[0213] Based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, the attitude information of the device coordinate system relative to the world coordinate system is determined.

[0214] In one optional implementation of this application, the number of light spots on the motion capture device is at least three.

[0215] An optional implementation of this application's embodiments further includes:

[0216] The display module is used to display a perspective view of the image and to display guidance information in the perspective view of the image to guide the user to place the motion capture device in the perspective view of the image.

[0217] The calibration device for motion capture equipment provided in this application acquires images of the motion capture equipment worn on a human limb using a camera. Based on these images, it determines the target posture information of the equipment's coordinate system relative to the world coordinate system, and then calibrates the motion capture equipment according to this target posture information. By utilizing motion capture equipment images for calibration, this application effectively reduces the difficulty and time required for calibration, minimizes calibration errors, and improves the calibration accuracy. Consequently, the calibrated motion capture equipment enhances limb motion capture performance, providing a basis for improving the accuracy of human posture reconstruction.

[0218] It should be understood that the device embodiments and the foregoing method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device 500 shown in FIG10 can execute the method embodiment corresponding to FIG3, and the foregoing and other operations and / or functions of each module in the device 500 are respectively to implement the corresponding processes in each method in FIG3. For the sake of brevity, they will not be repeated here.

[0219] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the first aspect method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the first aspect method embodiment described above.

[0220] Figure 11 is a schematic block diagram of an electronic device provided in an embodiment of this application. As shown in Figure 11, the electronic device 600 may include:

[0221] The system includes a memory 610 and a processor 620. The memory 610 stores computer programs and transfers the program code to the processor 620. In other words, the processor 620 can retrieve and run the computer program from the memory 610 to implement the calibration method of the motion capture device in this embodiment.

[0222] For example, the processor 620 can be used to execute the above-described calibration method embodiment for the motion capture device according to instructions in the computer program.

[0223] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0224] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0225] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0226] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0227] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the calibration method for the motion capture device provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0228] As shown in Figure 11, the electronic device 600 may further include:

[0229] Transceiver 630, which can be connected to processor 620 or memory 610.

[0230] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0231] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0232] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above method embodiments.

[0233] This application also provides a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the above method embodiments.

[0234] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0235] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0237] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0238] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for calibrating a motion capture device, comprising: capturing a motion capture device image by a camera included in a head-mounted display device, the motion capture device being worn on a limb of a human body, and the method being applied to the head-mounted display device; determining target pose information of a device coordinate system relative to a world coordinate system according to the motion capture device image, the device coordinate system being a motion capture device coordinate system; calibrating the motion capture device according to the target pose information.

2. The method of claim 1, wherein the determining target pose information of a device coordinate system relative to a world coordinate system according to the motion capture device image comprises: extracting image feature points from the motion capture device image; determining target pose information of a device coordinate system relative to a world coordinate system according to the image feature points.

3. The method of claim 1, wherein the motion capture device includes a light-emitting unit, and the capturing a motion capture device image by a camera comprises: capturing a light spot image of the light-emitting unit on the motion capture device by the camera; and wherein the determining target pose information of a device coordinate system relative to a world coordinate system according to the motion capture device image comprises: determining target pose information of a device coordinate system relative to a world coordinate system according to the light spot image.

4. The method of claim 3, wherein the determining target pose information of a device coordinate system relative to a world coordinate system according to the light spot image comprises: determining target pose information of a device coordinate system relative to a world coordinate system according to the light spot image using a pose estimation algorithm; extracting pose information from the target pose information, and determining the pose information as the target pose information of the device coordinate system relative to the world coordinate system.

5. The method of claim 3, further comprising: determining initial pose information of the device coordinate system relative to a reference coordinate system according to inertial measurement unit data sent by the motion capture device; and wherein the calibrating the motion capture device comprises: correcting a pose conversion relationship between the world coordinate system and the reference coordinate system according to the target pose information and the initial pose information; and calibrating the motion capture device according to the corrected pose conversion relationship between the world coordinate system and the reference coordinate system.

6. The method of claim 5, wherein the correcting a pose conversion relationship between the world coordinate system and the reference coordinate system comprises: correcting a heading angle in the pose conversion relationship between the world coordinate system and the reference coordinate system. accelerometer data, gyroscope data, and magnetometer data, and wherein the determining initial pose information of the device coordinate system relative to a reference coordinate system according to inertial measurement unit data sent by the motion capture device comprises: determining initial pose information of the device coordinate system relative to the reference coordinate system according to the accelerometer data, the gyroscope data, and the magnetometer data sent by the motion capture device using a complementary filtering algorithm.

7. The method of claim 5, wherein the inertial measurement unit data comprises: ​ ​ ​ 8. The method of any one of claims 1-7, further comprising: determining pose information of the device coordinate system relative to the limb coordinate system; determining a pose transformation relationship between the limb coordinate system and the world coordinate system according to the pose information of the device coordinate system relative to the limb coordinate system and the pose information of the device coordinate system relative to the world coordinate system.

9. The method of claim 8, wherein the determining the pose information of the device coordinate system relative to the limb coordinate system comprises: determining a pose transformation relationship between the limb coordinate system and a head-mounted coordinate system and pose information of the head-mounted coordinate system relative to the world coordinate system at a preset calibration pose, the head-mounted coordinate system being a head-mounted display device coordinate system; determining the pose information of the device coordinate system relative to the limb coordinate system according to the pose transformation relationship between the limb coordinate system and the head-mounted coordinate system, the pose information of the head-mounted coordinate system relative to the world coordinate system, and the pose information of the device coordinate system relative to the world coordinate system.

10. The method of claim 9, wherein the determining the pose information of the device coordinate system relative to the world coordinate system comprises: determining the pose information of the device coordinate system relative to the world coordinate system according to a pose transformation relationship between a corrected world coordinate system and a reference coordinate system.

11. The method of claim 3, wherein the number of light spots on the motion capture device is at least three.

12. The method of claim 1, further comprising: displaying a video see-through image and displaying guide information in the video see-through image for guiding a user to place the motion capture device in the video see-through image.

13. A calibration apparatus for a motion capture device, comprising: an image acquisition module configured to acquire an image of the motion capture device by a camera included in a head-mounted display device, the motion capture device being worn on a limb of a human body, and the apparatus being applied to the head-mounted display device; a pose determination module configured to determine target pose information of a device coordinate system relative to a world coordinate system according to the image of the motion capture device, the device coordinate system being a motion capture device coordinate system; a device calibration module configured to calibrate the motion capture device according to the target pose information.

14. An electronic device, comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to perform the calibration method of the motion capture device according to any one of claims 1-12.

15. A computer readable storage medium configured to store a computer program, the computer program causing a computer to perform the calibration method of the motion capture device according to any one of claims 1-12.

16. A computer program product comprising program instructions which, when run on an electronic device, cause the electronic device to perform the calibration method of the motion capture device according to any one of claims 1-12.

Images