Calibration method, apparatus, device, and medium for motion capture devices

Abstract

This application provides a calibration method, apparatus, device, and medium for motion capture devices. [Solution] A motion capture device is attached to the limbs of a human body, and the method is applied to a head-mounted display device, which includes a camera. The calibration method for a motion capture device according to the present invention includes: collecting motion capture device images with a camera; determining target orientation information of the device coordinate system relative to the world coordinate system based on the motion capture device images; and calibrating the motion capture device based on the target orientation information. By calibrating the motion capture device using motion capture device images, the present invention can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve the calibration accuracy of the motion capture device.

Description

[Technical Field]

[0001] The embodiments of this application relate to the technical field of motion capture, and more particularly to methods, apparatus, devices, and media for calibrating motion capture devices.

[0002] This application claims priority to a Chinese patent application filed on August 30, 2023, with the title of the invention "Calibration method, apparatus, device, and medium for motion capture devices," application number 202311110278.X, and all the contents of said application are incorporated into this application by reference. [Background technology]

[0003] Human motion capture technology has great potential for applications in fields such as virtual reality (VR) and human-computer interaction. Current human motion capture methods primarily involve collecting real-world human movement data by performing motion capture using motion capture devices (e.g., inertial sensors or tracking devices with inertial sensors).

[0004] Before collecting human motion data using a motion capture device, it is necessary to calibrate the motion capture device to ensure the accuracy and precision of the collected motion data. In related technologies, when calibrating a motion capture device, it is necessary to fix the mounting position of the motion capture device using a reference object. This method increases the difficulty of operation for the user, causing inconvenience to the user, and may also induce calibration errors due to inaccurate mounting of the motion capture device, leading to a decrease in the calibration accuracy and functional degradation of the motion capture device. [Overview of the project] [Problems that the invention aims to solve]

[0005] This application provides a method, apparatus, device, and medium for calibrating a motion capture device. By calibrating a motion capture device using motion capture device images, the difficulty of calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, and calibration errors can be reduced, thereby improving the calibration accuracy of the motion capture device. [Means for solving the problem]

[0006] In a first embodiment, an embodiment of the present application provides a method for calibrating a motion capture device. The motion capture device is attached to the limbs of a human body, and the method is applied to a head-mounted display device, the head-mounted display device includes a camera. The method is The steps include: collecting images of the motion capture device using a camera; The step of determining target attitude information of the device coordinate system relative to the world coordinate system based on the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system. The procedure includes the step of calibrating the motion capture device based on the target posture information.

[0007] In a second embodiment, an embodiment of the present application provides a calibration apparatus for a motion capture device. The motion capture device is attached to the limbs of a human body, the apparatus is arranged on a head-mounted display device, the head-mounted display device includes a camera. The apparatus is An image acquisition module used to collect motion capture device images using the aforementioned camera, The attitude determination module is used to determine the target attitude information of the device coordinate system relative to the world coordinate system, and the device coordinate system is the motion capture device coordinate system. The system includes a device calibration module used to calibrate the motion capture device based on the target posture information.

[0008] In a third embodiment, an embodiment of the present application provides an electronic device including a processor and memory. The memory is used to store computer programs, and the processor is used to cause the electronic device to perform the calibration method for a motion capture device described in the embodiment of the first embodiment or each implementation method by calling and executing the computer programs stored in the memory.

[0009] In a fourth embodiment, an embodiment of the present application provides a computer-readable storage medium used to store a computer program which causes the computer to perform a calibration method for a motion capture device as described in the embodiment of the first embodiment or each implementation.

[0010] In a fifth embodiment, an embodiment of the present application provides a computer program product including program instructions. When the program instructions are executed by an electronic device, the electronic device is instructed to perform the calibration method for a motion capture device described in the embodiment of the first embodiment or each implementation method. [Effects of the Invention]

[0011] The technical modes disclosed in the embodiments of this application have at least the following beneficial effects.

[0012] By collecting the motion capture device images worn on the limbs of the human body by the camera, based on the motion capture device images, the target pose information of the device coordinate system with respect to the world coordinate system is determined, and based on the target pose information, the motion capture device is calibrated. This application can effectively reduce the calibration difficulty of the motion capture device, shorten the calibration time, reduce the calibration error, and improve the calibration accuracy of the motion capture device by performing a calibration operation on the motion capture device using the motion capture device images. Thereby, the calibrated motion capture device improves the motion capture effect of the limbs and improves the accuracy of the reproduction of the human body pose.

Brief Description of the Drawings

[0013] To more clearly explain the technical forms in the embodiments of this application, the attached drawings used in the description of the embodiments will be briefly introduced below. However, the attached drawings in the following description are only a part of the embodiments of this application, and it is obvious that those skilled in the art can obtain other drawings based on these drawings without creative efforts. <O000083> [Figure 1] It is a schematic diagram of an application example provided by the embodiment of this application. [Figure 2] It is a schematic diagram of a head-mounted display device, a motion capture device, and peripheral devices worn on different parts of the human body provided by the embodiment of this application. [Figure 3] It is a flowchart of a calibration method for a motion capture device provided by the embodiment of this application. [Figure 4] It is a schematic diagram of the relationship between the world coordinate system and the device coordinate system provided by the embodiment of this application. [Figure 5] It is a flowchart of a calibration method for another motion capture device provided by the embodiment of this application. [Figure 6] It is a flowchart of a calibration method for a further motion capture device provided by the embodiment of this application. [Figure 7] A schematic diagram of the relationship among the world coordinate system, the reference coordinate system, and the device coordinate system provided by an embodiment of the present application. [Figure 8] A flowchart of a calibration method for another motion capture device provided by an embodiment of the present application. [Figure 9] A schematic diagram of the relationship among the device coordinate system, the world coordinate system, and the limb coordinate system provided by an embodiment of the present application. [Figure 10] A schematic block diagram of a calibration device for a motion capture device provided by an embodiment of the present application. [Figure 11] A schematic block diagram of an electronic device provided by an embodiment of the present application.

Mode for Carrying Out the Invention

[0014] Hereinafter, while referring to the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. It is obvious that the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. According to the embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without creative efforts also belong to the protection scope of the present application.

[0015] Furthermore, terms such as “first,” “second,” etc., in the specification, claims, and accompanying drawings of this application are used merely to distinguish similar subjects and are not required to describe a specific order or sequence. The data used in this manner may be interchangeable, and it should be understood that the embodiments of the application described herein can be realized in an order other than that illustrated or described herein. In addition, the terms “includes,” “has,” and their variations are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server including a series of steps or units does not have to be limited to those steps or units explicitly listed, but rather may include other steps or units that are not explicitly listed or are specific to those processes, methods, products, or apparatus.

[0016] This invention is suitable for applications in human motion capture. Currently, human motion capture methods mainly involve collecting human movement data in the real world by performing motion capture using a motion capture device. Motion capture devices are generally inertial sensors or tracking devices with inertial sensors. In order to ensure the accuracy and precision of the motion data collected by the motion capture device, it is necessary to calibrate the motion capture device. In related technologies, when calibrating a motion capture device, it is necessary to fix the mounting position of the motion capture device using a reference object. This increases the difficulty of operation for the user, causing inconvenience to the user, and consequently, calibration errors due to inaccurate mounting position of the motion capture device are induced, leading to a decrease in the calibration accuracy and functional degradation of the motion capture device. Taking these problems into consideration, this invention provides a calibration method for motion capture devices. By calibrating the motion capture device using this method, the difficulty of calibration of the motion capture device can be effectively reduced, the calibration time can be shortened, and calibration errors can be reduced, thereby improving the calibration accuracy of the motion capture device.

[0017] To facilitate understanding of the embodiments of this application, before describing the various embodiments of this application, we will first appropriately interpret and explain some of the concepts related to all embodiments of this application.

[0018] 1) Virtual Reality (VR) Virtual reality is a technology for constructing and experiencing virtual worlds, using multi-source information (virtual reality as referred to herein includes at least visual perception, and may also include auditory, tactile, and kinetic perception, and may even include gustatory and olfactory perception, etc.) to generate a specific virtual environment, thereby realizing a fused and interactive three-dimensional dynamic view and simulation of physical actions within the virtual environment. This allows users to immerse themselves in the simulated virtual reality environment, achieving diverse applications in virtual environments such as maps, games, videos, education, healthcare, simulations, collaborative training, sales, manufacturing support, maintenance, and repair.

[0019] 2) Virtual reality devices (VR devices) A virtual reality device is a terminal that realizes virtual reality effects, and can typically be provided in the form of glasses for visual and other sensory perception, a helmet-mounted display (HMD), contact lenses, etc. Of course, the form in which virtual reality devices are realized is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0020] Selectively, the virtual reality devices described in the embodiments of this application include, but are not limited to, the following types:

[0021] 2.1) PC-based virtual reality (PCVR) devices A PC-based virtual reality device performs calculations and data output related to virtual reality functions on a PC terminal. An externally connected PC-based virtual reality device uses the data output from the PC terminal to realize virtual reality effects.

[0022] 2.2) Mobile virtual reality devices Mobile virtual reality devices support setting up mobile devices (e.g., smartphones) in various ways (e.g., head-mounted displays with dedicated card slots) and connect to the mobile device via wired or wireless connections. The mobile device then performs calculations related to virtual reality functions and outputs data to the virtual reality device. For example, virtual reality images can be viewed through an app on the mobile device.

[0023] 2.3) Integrated virtual reality device Because the integrated virtual reality device has a processor for performing calculations related to virtual functions, it has independent virtual reality input and output functions, does not require connection to a PC terminal or mobile terminal, and offers a high degree of flexibility in use.

[0024] 3) Virtual field of view The virtual field of view refers to the area within the virtual environment that a user can perceive through the lenses of a virtual reality device, and represents the perceived area, which is the field of view (FOV) of the virtual field of view.

[0025] 4) Augmented Reality (AR) Augmented reality (AR) refers to a technology that calculates the pose parameters of a camera in the real world (or the 3D world, or the real world) in real time and adds virtual elements to the collected images based on these parameters. These virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen and enable interaction that links the two.

[0026] 5) Mixed Reality (MR) Mixed reality refers to a simulated environment that integrates computer-generated sensory input (e.g., virtual objects) with sensory input from the physical environment, or its representation. In some MR environments, computer-generated sensory input can adapt to changes in sensory input from the physical environment. Furthermore, some electronic systems used to present the MR environment can monitor the orientation and / or position relative to the physical environment so that virtual objects can interact with real objects (physical elements or their representations from the physical environment). For example, the system may monitor movement to ensure that a virtual plant appears stationary relative to a physical building.

[0027] 6) Augmented Reality (XR) Augmented reality refers to any combined real and virtual environment and human-computer interaction generated by computer technology and wearable devices, and includes multiple forms such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0028] 7) Virtual Scene A virtual scene is a virtual scene that is displayed (or provided) when an application is run on an electronic device. This virtual scene may be an environment that simulates the real world, a semi-simulated, semi-fictional virtual environment, or a completely fictional virtual environment. The virtual scene may be a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene, and the embodiments of this application do not limit the dimensions of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts, cities, etc., and the user can control the movement of virtual objects within this virtual scene.

[0029] 8) Virtual Objects A virtual object is an object that interacts within a virtual scene, controlled by a user or a bot program (e.g., an AI-based bot program), and capable of remaining still, moving, and performing various actions within the virtual scene. Examples include individual characters in a game.

[0030] To clearly explain the technical mode of this application, the following describes examples of its application. The technical mode of this application is applicable to, but is not limited to, the following scenarios.

[0031] As an example, Figure 1 is a schematic diagram of an application example provided by an embodiment of the present invention. As shown in Figure 1, the application example 1000 includes a head-mounted display device 100 and a motion capture device 200. The head-mounted display device 100 and the motion capture device 200 are also capable of communication.

[0032] In some implementations, the head-mounted display device 100 is an HMD such as a head-mounted display for a VR all-in-one device, and this embodiment is not limited to this.

[0033] Furthermore, the head-mounted display device 100 is equipped with a camera, which collects data on the surrounding environment. Based on the collected data on the surrounding environment, position tracking and positioning are performed using a computer vision algorithm called simultaneous localization and mapping (SLAM). The number of cameras is at least one, and the example given is four cameras as shown in Figure 1. Moreover, the type of camera may be a fisheye camera, a normal camera, or another type of camera, and this invention is not limited to these.

[0034] In some implementations, the motion capture device 200 may be an inertial sensor, or any device having an inertial sensor, etc.

[0035] In several implementations, the motion capture device 200 can be attached to parts of the human body, such as the limbs and trunk. The limbs include the upper and lower limbs, and the trunk includes the waist, but this invention does not limit the parts of the human body in any way. Exemplarily, as shown in Figure 2, the motion capture device 200 is attached to the lower limbs of a human body. The lower limbs of a human body include the thighs and shins. That is, by attaching four motion capture devices 200 to the thighs and shins of a human body, the motion capture devices 200 collect thigh movement data and lower leg movement data (collectively referred to as lower limb movement data) of the human body, and transmit this lower limb movement data to the head-mounted display device 100, thereby achieving the objective of tracking the movement of the human lower limbs.

[0036] The limb data collected by the motion capture device 200, which is attached to the limbs of the human body, is either 3-degree-of-freedom (Dof) data or 6-degree-of-freedom data.

[0037] Considering that the upper limbs of the human body may perform different movements, in this invention, as an option, in addition to attaching the motion capture device 200 to the upper limbs of the human body, peripheral devices 300 may also be attached to the upper limbs of the human body, for example, the hands and / or arms. Then, upper limb movement data of the human body is collected by the peripheral devices 300 and transmitted to the head-mounted display device 100, thereby enabling tracking of the upper limb movements of the human body. For specific methods of attaching the peripheral devices 300, please refer to Figure 2.

[0038] In some implementations, the peripheral device 300 includes, but is not limited to, a handle, a glove, a wristband, an armband, a ring-type device, or other wearable devices. Furthermore, the peripheral device 300 is equipped with an inertial sensor, which can provide 6-degree-of-freedom data including the position and orientation of the human upper limbs.

[0039] In this embodiment, the inertial sensor specifically refers to an inertial measurement unit (IMU). This IMU may be a 6-axis IMU28 or a 9-axis IMU, and is not specifically limited in this application.

[0040] In the embodiments of the present invention, it should be understood that tracking of the entire human body's motion is possible by attaching the motion capture device 200 to the limbs of the human body, or by attaching the peripheral device 300 to the upper limbs and the motion capture device 200 to the lower limbs, etc.

[0041] It should be understood that the head-mounted display device 100, motion capture device 200, and peripheral devices 300 shown in Figures 1 and 2 are merely illustrative examples and do not constitute a specific limitation of this invention.

[0042] After describing an example of the application of the embodiment of this application, the calibration method, apparatus, equipment, and medium of the motion capture device provided by the embodiment of this application will be described in detail below with reference to the attached drawings.

[0043] Figure 3 is a flowchart of a calibration method for a motion capture device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the calibration of a motion capture device in human body motion capture, and the calibration method for the motion capture device is performed by a calibration device for the motion capture device. The calibration device for the motion capture device consists of hardware and / or software and can be incorporated into a head-mounted display device.

[0044] In the embodiments of the present invention, the head-mounted display device may be any electronic device capable of reproducing human body movements. Optionally, the electronic device may be an Extended Reality (XR) device. The XR device may be a VR device, an Augmented Reality (AR) device, or a Mix Reality (MR) device, and the present invention does not impose any specific limitations on the type of electronic device.

[0045] As shown in Figure 3, the method includes the following steps.

[0046] In S101, motion capture device images are collected by a camera.

[0047] For example, as shown in Figure 1, consider the possibility of multiple cameras being attached to a head-mounted display device. Therefore, when there are multiple cameras on the head-mounted display device, the present invention prefers to collect motion capture device images using a downward-view camera located on the head-mounted display device, and enables easier collection of motion capture device images attached to the limbs of the human body by using a camera located closer to the limbs of the human body.

[0048] Selectively, before executing S101, the user first attaches devices such as a head-mounted display device and a motion capture device to different parts of the human body. For example, the head-mounted display device is attached to the head, the motion capture device to the lower limbs, and the external device to the upper limbs. Specifically, refer to Figure 2 above. Alternatively, for example, the head-mounted display device is attached to the head, and the motion capture device is attached to the upper and lower limbs. Furthermore, for example, the head-mounted display device is attached to the head, and the motion capture device is attached to parts such as the upper limbs, waist, and lower limbs.

[0049] After putting on the head-mounted display device, motion capture device, and external device, the user can activate the head-mounted display device, motion capture device, and external device to establish communication connections between the head-mounted display device and each motion capture device, and between the head-mounted display device and the external device, thereby establishing a foundation for subsequent data communication.

[0050] The specific methods for establishing communication connections between the head-mounted display device and each motion capture device, and between the head-mounted display device and external devices, can be found in conventional methods and will not be described again in this application.

[0051] Furthermore, the head-mounted display device can collect images of motion capture devices attached to the limbs of a human body using its camera. The collected motion capture device images include not only the motion capture devices themselves, but also the limbs of the human body to which the motion capture devices are attached. Therefore, in this application, a single frame of motion capture device image collected by the camera of the head-mounted display device includes at least one motion capture device. This allows the head-mounted display device to distinguish each motion capture device in the motion capture device image based on the motion capture device in the image and the limb of the human body to which each motion capture device belongs. Furthermore, the motion capture devices attached to the limbs of a human body can be calibrated based on each motion capture device in the motion capture device image.

[0052] In some selective embodiments, the present invention may, when collecting motion capture device images using a camera of a head-mounted display device, collect only one motion capture device image during a predetermined time period and calibrate the motion capture device based on that motion capture device image. This reduces the complexity of image processing in the head-mounted display device and improves the calibration efficiency of the motion capture device.

[0053] For example, during the first time period, motion capture device images of the right shin are collected. During the second time period, motion capture device images of the right thigh are collected. During the third time period, motion capture device images of the left shin are collected. During the fourth time period, motion capture device images of the left thigh are collected. The order of the first, second, third, and fourth time periods described above may be first time period → second time period → third time period → fourth time period, and can be flexibly adjusted according to actual needs, and this application is not limited thereto.

[0054] In some selective embodiments, considering that the head-mounted display device can support video see-through (VST) functionality, the user can view a virtualized real space and real objects at a 1:1 ratio with respect to the real world through the head-mounted display device. The virtualized real objects include, but are not limited to, tables, chairs, etc. Therefore, in this application, before the camera collects motion capture device images, the head-mounted display device can display a video image perspective view at a 1:1 ratio with respect to the real world via the VST functionality. Simultaneously, guide information is displayed in any area of ​​the video image perspective view to guide the user to position a specific motion capture device in the video, so that the user can position the motion capture device in the video image perspective view based on this information. As a result, when the user positions any motion capture device in the video image perspective view based on the guide information, the camera of the head-mounted display device can collect images of the motion capture device. By installing the system in this way, the VST function can be used to output guide information to the user, guiding the user to perform corresponding actions based on that information. Furthermore, the video see-through function allows the user to see their own movements, thereby visualizing the user's actions, enabling intuitive interaction with the head-mounted device, and improving the ease of calibration by the motion capture device.

[0055] Furthermore, considering that the calibration principle and calibration process for each motion capture device attached to the limbs of the human body are the same, in order to simplify the explanation, this application will describe the technical embodiments of this application in detail using one motion capture device as an example in each of the following embodiments.

[0056] In S102, the target attitude information of the device coordinate system relative to the world coordinate system is determined based on the motion capture device image, and the device coordinate system is the motion capture device coordinate system.

[0057] The device coordinate system is,

number

[0058] The world coordinate system refers to the global coordinate system, which can be denoted by G, where G is an abbreviation for Global.

[0059] In some implementation methods, the relationship between the world coordinate system and the device coordinate system is shown in Figure 4.

number

number

number

[0060] After selectively acquiring motion capture device images collected by the camera, the head-mounted display device can extract image feature points from the motion capture device images. Based on the extracted image feature points, the target attitude information of the device coordinate system relative to the world coordinate system is determined.

[0061] Considering that images collected by a camera include at least one motion capture device, a head-mounted display device can extract image feature points from motion capture images in different ways depending on the number of motion capture devices in the motion capture image. Specifically, If only one motion capture device is present in the motion capture device image, the head-mounted display device extracts image feature points from the motion capture device image, in a first method. A second method is conceivable in which, if the motion capture device image includes at least two motion capture devices, the head-mounted display device extracts the image feature points of each motion capture device from the motion capture device image.

[0062] In this application, the image feature points are prominent or representative points within the motion capture device image. For example, these may be corner points, contour points, dark spots within bright areas, bright spots within dark areas, etc., and this application does not impose any restrictions on image feature points.

[0063] Furthermore, the image feature points extracted from motion capture device images are two-dimensional points (i.e., 2D points).

[0064] As a selective implementation method, the present invention may employ a feature point extraction algorithm when extracting image feature points from motion capture device images. Exemplary feature point extraction algorithms include, but are not limited to, SIFT (Scale Invariant Feature Transform) extraction algorithms, SURF (Speeded-Up Robust Features) extraction algorithms, and ORB (Oriented FAST and Rotated BRIEF) extraction algorithms.

[0065] After extracting image feature points from the motion capture device image, the head-mounted display device determines the target attitude information of the device coordinate system corresponding to the motion capture device relative to the world coordinate system, based on the image feature points extracted from the motion capture device within the motion capture device image. In other words, it identifies the attitude transformation relationship between the device coordinate system and the world coordinate system.

[0066] Considering that the type, specifications, shape, etc., of motion capture devices attached to the limbs of the human body are known, it can be determined that the three-dimensional model (3D model) of the motion capture device is also known. Therefore, this application can identify 3D points corresponding to the 2D image feature points of an arbitrary motion capture device, based on the 2D image feature points of the motion capture device extracted from the motion capture device image and the 3D model of the motion capture device. Furthermore, by applying a pre-set calculation rule, the attitude transformation relationship between the device coordinate system and the world coordinate system is calculated based on the 2D points and 3D points of the motion capture device.

[0067] The pre-set calculation rules described above can be selected as any algorithm or strategy capable of calculating the attitude transformation relationship between the device coordinate system and the world coordinate system based on 2D points and 3D points, and this application imposes no limitations on this.

[0068] In this application, the 2D point can be understood as the 3D point projected onto the imaging plane, and the 3D point can be understood as a three-dimensional coordinate point under the world coordinate system.

[0069] Since the position and orientation information of the head-mounted coordinate system relative to the world coordinate system is known, if the pre-set calculation rule is a computer vision position and orientation estimation (Perspective-n-Point, PnP) algorithm, the head-mounted display device calculates target position and orientation information of the device coordinate system relative to the world coordinate system based on n points (2D and 3D points of the motion capture device in the motion capture device image) using the above PnP algorithm. Then, from the target position and orientation information, the target orientation information of the device coordinate system relative to the world coordinate system is extracted.

[0070] Typically, PnP algorithms include P3P algorithms, P5P algorithms, and other algorithms. The P3P algorithm calculates position and orientation information of the device coordinate system relative to the world coordinate system based on three points. The P5P algorithm calculates position and orientation information of the vice coordinate system relative to the world coordinate system based on five points. Therefore, in this application, it is necessary to extract at least three feature points of the motion capture device from the motion capture device image, and to determine at least three 3D points corresponding to the at least three 2D points. This makes it possible to select a different PnP algorithm depending on the number of image feature points of the motion capture device in the motion capture device image when calculating target position and orientation information of the device coordinate system relative to the world coordinate system based on the motion capture device image acquired by the camera.

[0071] S103. Calibrate the motion capture device based on the target posture information.

[0072] After determining the target orientation information for the device coordinate system relative to the world coordinate system, this invention calibrates the orientation of the motion capture device relative to the world based on the target orientation information, thereby making the limb movements captured by the calibrated motion capture device more accurate thereafter.

[0073] In some selective embodiments, when calibrating the orientation of a motion capture device relative to the world, the initial orientation of the motion capture device relative to the world may be calibrated, or the real-time orientation of the motion capture device relative to the world may be calibrated. The present invention imposes no limitations on this.

[0074] In the calibration method for a motion capture device provided in the embodiment of the present invention, images of the motion capture device attached to the limbs of a human body are collected by a camera, and after determining the target orientation information of the device coordinate system relative to the world coordinate system based on the motion capture images, the motion capture device is calibrated based on the target orientation information. By performing the calibration operation using motion capture device images, the present invention can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve calibration accuracy. As a result, the motion capture effect of the limbs can be improved by the calibrated motion capture device, and the accuracy of reproducing the posture of the human body can be improved.

[0075] In another optional implementation scenario, consider that the motion capture device includes a light-emitting unit. That is, the motion capture device is an inertial sensor having a light-emitting unit, or any device comprising an inertial sensor such as an optical tracker and a light-emitting unit. Therefore, as an option, the present invention makes it possible to collect a light spot image of the light-emitting unit on the motion capture device image using a camera and calibrate the motion capture device based on the light spot image.

[0076] The light-emitting unit on the motion capture device can be an LED or other light-emitting element, and the emitted light may be visible or invisible, as long as it is detectable by the camera. The light-emitting unit can be placed inside the housing of the motion capture device body or on an accessory rigidly connected to the motion capture device body, such as a base, according to a preset arrangement method. In this application, the preset arrangement method can be flexibly set according to the shape of the motion capture device body or the rigidly connected accessory, and no limitations are imposed here.

[0077] Furthermore, if a motion capture device attached to the limbs of a human body includes a light-emitting unit, the limb data collected by the motion capture device includes the following two types:

[0078] As the first type, when the illumination status of a light-emitting unit on a motion capture device is detected by a camera on a head-mounted display device, the limb movement data collected by the motion capture device is 6-degree-of-freedom (DoF) data, including the position and orientation of the human limbs.

[0079] As a second type, if the illumination status of the light-emitting unit on the motion capture device cannot be detected by the camera on the head-mounted display device, the limb movement data collected by the motion capture device is 3-degrees-of-freedom (Dof) data, including the posture of the human limbs.

[0080] In the example shown in Figure 2, the lower limb motion data collected by the motion capture device 200 includes the following two types. As a first type, when the illumination state of the light-emitting unit on the motion capture device 200 is detected by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 6-degrees-of-freedom (Dof) data including the position and posture of the human lower limbs. As a second type, when the illumination state of the light-emitting unit on the motion capture device 200 cannot be detected by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 3-degrees-of-freedom (Dof) data including the posture of the human lower limbs.

[0081] The calibration process for the motion capture device based on the light spot image described above will be specifically explained below with reference to Figure 5. As shown in Figure 5, the method includes the following steps.

[0082] In the S201, a camera collects a light spot image of the light-emitting unit on the motion capture device.

[0083] Considering that each motion capture device attached to the limbs of the human body is equipped with a light-emitting unit, if the light-emitting units on each motion capture device are simultaneously illuminated and the camera on the head-mounted display device collects light spot images of all motion capture devices at the same time, the head-mounted display device cannot accurately identify each light spot from the corresponding motion capture device based on the light spot images collected by the camera. This can lead to confusion in the calibration of the motion capture devices and, consequently, calibration errors.

[0084] Due to the reasons described above, in this invention, when calibrating all motion capture devices, the light-emitting units on the motion capture devices are illuminated at different timings. That is, each motion capture device is illuminated sequentially in a specific order at each cycle. As a result, the camera on the head-mounted display device collects only the light spot image of the light-emitting unit on one motion capture device at a time, and calibrates the motion capture device using this light spot image, thereby ensuring further improvement in the calibration accuracy of each motion capture device.

[0085] For example, in time zone 1, the light-emitting unit of the motion capture device attached to the left thigh is turned on, and the light-emitting units of the motion capture devices attached to the right thigh, right shin, and left shin are controlled to be turned off. In time zone 2, the light-emitting unit of the motion capture device attached to the left shin is turned on, and the light-emitting units of the motion capture devices attached to the right thigh, right shin, and left thigh are controlled to be turned off. In time zone 3, the light-emitting unit of the motion capture device attached to the right thigh is turned on, and the light-emitting units of the motion capture devices attached to the right shin, left thigh, and left shin are controlled to be turned off. In time zone 4, the light-emitting unit of the motion capture device attached to the right shin is turned on, and the light-emitting units of the motion capture devices attached to the left thigh, left shin, and right thigh are controlled to be turned off. The order of time zones 1, 2, 3, and 4 may be time zone 1 → time zone 2 → time zone 3 → time zone 4, or any other order; no specific restrictions are imposed here.

[0086] Regarding the illumination of the light-emitting unit on the motion capture device as described above, it may be activated manually by the user, or the head-mounted display device may transmit an illumination command to the motion capture device to illuminate the light-emitting unit on the motion capture device. This application does not impose any restrictions on this.

[0087] In some selective implementation methods, when a camera collects a spotlight image of a light-emitting unit on a motion capture device that is lit, the camera on the head-mounted display device may not be able to collect a spotlight image of the light-emitting unit on the motion capture device that is lit, due to the user's clothing being loose or other circumstances.

[0088] In this application, after illuminating a light-emitting unit on any of the motion capture devices, the user is instructed to perform a head-lowering motion, a kicking motion, or an arm-raising motion, allowing the camera on the head-mounted display device to collect a light spot image of the light-emitting unit on the motion capture device. The lower limb performing the kicking motion or the upper limb performing the arm-raising motion is the lower limb or upper limb to which the illuminated motion capture device is attached. For example, assuming that the light-emitting unit of the motion capture device on the left shin is illuminated, the user is guided to perform a kicking motion on their left shin. Similarly, assuming that the light-emitting unit of the motion capture device on the right forearm is illuminated, the user is guided to raise their right forearm.

[0089] The above-mentioned head-lowering motion, kicking motion, and / or arm-raising motion may be a head-lowering motion, a kicking motion, an arm-raising motion, a head-lowering motion and an arm-raising motion, or a head-lowering motion and a kicking motion, etc. It should be understood that no restrictions are imposed here.

[0090] Furthermore, in this application, when guiding the user to perform head-lowering, kicking, and / or arm-raising movements, it is conceivable to output guide audio, guide text, and / or guide video to the user, and no restrictions are imposed here.

[0091] In some selective embodiments, considering that the head-mounted display device supports Video See-Through (VST) functionality, the user can view a virtualized real space and real objects in proportion to the real world through the head-mounted display device. The virtualized real objects include, but are not limited to, tables and chairs. Accordingly, in this application, before the camera collects a light spot image of the light-emitting unit on the motion capture device, the head-mounted display device may optionally display a perspective view in proportion to the real world through the VST functionality. Furthermore, guide information may be displayed in an arbitrary area of ​​the perspective view to guide the user to position one of the motion capture devices that illuminates the light-emitting unit in the perspective view, and the user positions the motion capture device with the illuminated light-emitting unit in the perspective view based on the guide information. Thus, when the user positions one of the motion capture devices that illuminates the light-emitting unit in the perspective view based on the guide information, the camera on the head-mounted display device can collect a light spot image of the light-emitting unit on the motion capture device. The advantages of the above-mentioned setup include the ability to output guide information to the user via the VST function, the user performing corresponding actions based on the guide information, and the ability to visualize one's own actions via the see-through function. By visualizing the user's head-lowering, kicking, and / or arm-raising movements, interaction with the head-mounted display device becomes more intuitive, and calibration of the motion capture device becomes easier.

[0092] In S202, the target attitude information of the device coordinate system relative to the world coordinate system is determined based on the light spot image.

[0093] Selectively, the head-mounted display device of the present invention can employ pre-set calculation rules to calculate target attitude information of the device coordinate system relative to the world coordinate system from a light spot image.

[0094] In this application, the position and orientation information of the head-mounted coordinate system relative to the world coordinate system is utilized as a base, and the position and orientation information of the device coordinate system relative to the world coordinate system is calculated based on n points (n points corresponding to light spots in the light spot image) using a computer vision position and orientation estimation (abbreviated as Perspective-n-Point, PnP) algorithm. Therefore, in a situation where the placement position of the light-emitting unit on the motion capture device is known, the position and orientation estimation algorithm calculates target position and orientation information of the device coordinate system relative to the world coordinate system based on the light spot image collected by the camera. Then, orientation information is extracted from the target position and orientation information, and the extracted orientation information is determined as the target orientation information of the device coordinate system relative to the world coordinate system.

[0095] Typically, PnP algorithms include P3P algorithms, P5P algorithms, and other algorithms. The P3P algorithm calculates position and orientation information of the device coordinate system relative to the world coordinate system based on three points. The P5P algorithm calculates position and orientation information of the device coordinate system relative to the world coordinate system based on five points. Therefore, if the number of light spots on the motion capture device of this application is at least three, and the target position and orientation information of the device coordinate system relative to the world coordinate system is calculated based on the light spot image collected by the camera, a different PnP algorithm can be selected depending on the number of light spots in the light spot image.

[0096] Selectively, the number of light spots on the motion capture device may be the same as, or different from, the number of light-emitting units on the motion capture device. If the number of light spots on the motion capture device is the same as the number of light-emitting units, the number of light-emitting units on the motion capture device is at least three. If the number of light spots on the motion capture device is different from the number of light-emitting units, the number of light-emitting units on the motion capture device is at least one.

[0097] In some selective embodiments, the motion capture device may include a motion capture device body, or it may include a motion capture device body and an assembly, such as a base, rigidly connected to the motion capture device body. If there are at least three light-emitting units on the motion capture device, as a selective implementation, the at least three light-emitting units may be arranged within the housing of the motion capture device body in a predetermined arrangement. Alternatively, one light-emitting unit may be mounted within the housing of the motion capture device body in a predetermined arrangement, and the remaining light-emitting units may be mounted within the base in a predetermined arrangement. Alternatively, one light-emitting unit may be mounted within the base in a predetermined arrangement, and the remaining light-emitting units may be mounted within the housing of the motion capture device body in a predetermined arrangement. The above predetermined arrangements can be flexibly configured according to the shapes of the motion capture device body and the base, and no limitations are imposed here.

[0098] In S203, the motion capture device is calibrated based on target posture information.

[0099] In this invention, after determining the target orientation information of the device coordinate system relative to the world coordinate system, the orientation of the motion capture device relative to the world is calibrated based on the target orientation information, further improving the accuracy of limb movements captured by the calibrated motion capture device.

[0100] In some selective embodiments, the orientation of the motion capture device relative to the world is calibrated, selectively calibrating the initial orientation of the motion capture device relative to the world, or calibrating the real-time orientation of the motion capture device relative to the world. No limitations are imposed in this application.

[0101] In the calibration method for a motion capture device provided in the embodiment of the present invention, a camera collects light spot images of light-emitting units on the motion capture device attached to the limbs of a human body, and based on the light spot images, target orientation information of the device coordinate system relative to the world coordinate system is determined, and the motion capture device is calibrated based on the target orientation information. By performing a calibration operation on the motion capture device using light spot images of light-emitting units on the motion capture device, the present invention can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve the calibration accuracy of the motion capture device. As a result, the motion capture effect of the limbs can be improved by the calibrated motion capture device, contributing to improved accuracy in reproducing human body posture.

[0102] The calibration method for a motion capture device provided by the embodiments of the present application will be further interpreted and described below with reference to Figure 6. As shown in Figure 6, the method includes the following steps.

[0103] In S301, the initial attitude information of the device coordinate system relative to the reference coordinate system is determined based on the inertial measurement unit data transmitted from the motion capture device.

[0104] The reference coordinate system may be a geographic coordinate system, or any coordinate system that differs from the user-defined world coordinate system. In this application, the rolling angle (also called the roll angle) and pitch angle in the reference coordinate system are the same as the rolling angle and pitch angle in the world coordinate system.

[0105] The above reference coordinate system is denoted as "ref". Furthermore, if the reference coordinate system is a geographic coordinate system, it is a NED coordinate system.

[0106] In the embodiment of this application, the relationship between the world coordinate system, the reference coordinate system, and the device coordinate system is shown in Figure 7. GO is the world coordinate system. ref O is the reference coordinate system. L1 This is the coordinate system of a motion capture device attached to the thigh of the lower limb of the human body, O L2 This is the coordinate system of another motion capture device attached to the shin, corresponding to the thigh mentioned above.

[0107] In some selective embodiments, taking into account that the motion capture device is in operation, the inertial sensors (i.e., inertial measurement units) on the motion capture device transmit inertial measurement unit data to the head-mounted display device in real time, and the head-mounted display device calculates the initial attitude information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data. That is, it calculates the attitude transformation relationship between the device coordinate system and the reference coordinate system.

[0108] The inertial measurement unit data transmitted from the motion capture device includes accelerometer data, gyroscope data, and magnetometer data. Accordingly, in this application, an algorithm such as a complementary filter algorithm, which is pre-configured, determines the initial attitude information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, gyroscope data, and magnetometer data transmitted from the motion capture device.

[0109] The inertial measurement unit on the motion capture device transmits inertial measurement unit data to a head-mounted display device, and the motion capture device may acquire and transmit inertial measurement unit data using the inertial measurement unit when the camera on the head-mounted display device collects a light spot image, or the motion capture device may continuously acquire and transmit inertial measurement unit data using the inertial measurement unit from any timing before the camera on the head-mounted display device collects a light spot image. No limitations are imposed on this in this application.

[0110] In the present application, the number of initial attitude information of the device coordinate system with respect to the reference coordinate system determined based on the inertial measurement unit data transmitted from the motion capture device is one or more. Specifically, when the inertial sensor on the motion capture device transmits the inertial measurement unit data only once, the calculated initial attitude information of the device coordinate system with respect to the reference coordinate system is one. When the inertial sensor on the motion capture device transmits the inertial measurement unit data multiple times, the calculated initial attitude information of the device coordinate system with respect to the reference coordinate system is multiple. It can be understood that for each reception of the inertial measurement unit data, one initial attitude information of the device coordinate system with respect to one reference coordinate system can be calculated.

[0111] Also, the above-mentioned multiple initial attitude information may be the same or different, and specifically, it is determined according to the user's motion state.

[0112] In the present application, "multiple" is understood to mean two or more, that is, at least two.

[0113] Optionally, in the present application, the initial attitude information of the i-th device coordinate system with respect to the reference coordinate system is

Number

[0114] In S302, the camera collects the light spot image of the light emitting unit on the motion capture device.

[0115] In S303, based on the light spot image, the target attitude information of the device coordinate system with respect to the world coordinate system is determined, and the device coordinate system is the motion capture device coordinate system.

[0116] In S304, the attitude transformation relationship between the world coordinate system and the reference coordinate system is corrected based on the target attitude information and the initial attitude information.

[0117] In some selective embodiments, consider that while a camera on a head-mounted display device collects a spot light image, a motion capture device synchronously transmits inertial measurement unit data to the head-mounted display device. Therefore, as an option, the present invention corrects the attitude transformation relationship between the world coordinate system and the reference coordinate system based on target attitude information determined from the spot light image at the same time, and initial attitude information determined from the inertial measurement unit data. This is selectively achieved by the following equation (1).

number

[0118] Here,

number

number

number

[0119] It is considered that the attitude transformation relationship between the world coordinate system and the reference coordinate system can be expressed by Euler angles. Furthermore, Euler angles include azimuth, elevation, and roll angles. Accordingly, this application corrects the attitude transformation relationship between the world coordinate system and the reference coordinate system based on target attitude information and initial attitude information, and selectively corrects the azimuth, elevation, and / or roll angles between the world coordinate system and the reference coordinate system.

[0120] The above-mentioned azimuth angle, elevation angle, and / or roll angle may be an azimuth angle, or an elevation angle, or a roll angle, or an azimuth angle and elevation angle, or an azimuth angle and roll angle, or an azimuth angle, elevation angle, and roll angle. It should be understood that this application does not impose any restrictions on these.

[0121] In this application, the azimuth angle is understood as the angle around the X-axis of the motion capture device, where the X-axis is perpendicular to the ground. The elevation angle is understood as the angle around the Y-axis of the motion capture device, and the roll angle is understood as the angle around the Z-axis of the motion capture device.

[0122] In some selective embodiments, when the attitude transformation relationship between the world coordinate system and the reference coordinate system is expressed in terms of Euler angles, the elevation and roll angles in the world coordinate system and the reference coordinate system are completely identical, with only the azimuth angle parameter exhibiting a deviation. Therefore, in this application, the attitude transformation relationship between the world coordinate system and the reference coordinate system is corrected based on the target attitude information and the initial attitude information. Specifically, this refers to correcting the azimuth angle parameter in the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information.

[0123] In this application, correcting the azimuth angle in the attitude transformation relationship between the world coordinate system and the reference coordinate system specifically includes correcting the azimuth angle transformation relationship in the attitude transformation relationship between the world coordinate system and the reference coordinate system based on target attitude information and initial attitude information.

[0124] In S305, the motion capture device is calibrated based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system.

[0125] In this application, after selectively correcting the attitude transformation relationship between the world coordinate system and the reference coordinate system, the corrected attitude transformation relationship is combined with the initial attitude information of the device coordinate system relative to the reference coordinate system, thereby enabling the device to determine new target attitude information that has high accuracy with respect to the world coordinate system. Furthermore, the initial or real-time attitude of the motion capture device can be calibrated based on this new target attitude information, thereby further improving the calibration accuracy of the motion capture device.

[0126] In some selective embodiments, the corrected attitude transformation relationship can be determined based on the initial attitude information of the device coordinate system relative to the reference coordinate system, and can be realized by the following equation (2).

number

[0127] Here,

number

number

number

[0128] According to the calibration method for a motion capture device provided in the embodiment of the present invention, based on the light spot image of a light-emitting unit on a motion capture device attached to the limbs of a human body by a camera, target orientation information of the device coordinate system relative to the world coordinate system is determined based on the light spot image, and the motion capture device is calibrated based on the target orientation information. In the present invention, by performing a calibration operation on the motion capture device using the light spot image of a light-emitting unit on the motion capture device, the difficulty of calibration of the motion capture device can be effectively reduced, the calibration time can be shortened, and calibration errors can be reduced, improving the calibration accuracy of the motion capture device. As a result, the motion capture effect of the limbs can be improved by the calibrated motion capture device, contributing to improved accuracy in reproducing human body posture. Furthermore, when the orientation of the motion capture device relative to the world is calibrated based on the corrected orientation transformation relationship between the world coordinate system and the reference coordinate system by determining the initial orientation information of the device coordinate system relative to the reference coordinate system and correcting the orientation transformation relationship between the world coordinate system and the reference coordinate system based on the target orientation information and the initial orientation information, calibration errors can be further reduced, and the calibration accuracy of the motion capture device can be improved.

[0129] In selective implementation scenarios, the present invention can also determine the pose transformation relationship between the limb coordinate system and the world coordinate system after calibrating the motion capture device, and then determine the pose of the limbs corresponding to the motion capture device in the world coordinate system based on this pose transformation relationship. Below, with reference to Figure 8, the pose transformation relationship between the limb coordinates and the world coordinate system provided by the embodiment of the present invention will be determined, and the process of determining the limb pose in the world coordinate system based on this pose transformation relationship will be specifically described.

[0130] As shown in Figure 8, the method includes the following steps.

[0131] In S401, the orientation information of the device coordinate system relative to the limb coordinate system is determined.

[0132] In the embodiments of the present application, the limb coordinate system specifically refers to the limb coordinate system of the four limbs of the human body, and the limb coordinate system is B i This is how it is written. B is an abbreviation for Body, and i is the identification information of the limbs corresponding to the limb coordinate system. This identification information is any information that can uniquely determine the identity of a limb, such as the number or name of a major bone in the human body, and there are no specific restrictions on this.

[0133] As a selective implementation method, the relationship between the device coordinate system, the world coordinate system, and the limb coordinate system is shown in Figure 9. G This represents the world coordinate system, O B1 This represents the quadriplegic coordinate system of the thigh, O B2 This represents the limb coordinate system of the shin corresponding to the thigh, O L1 This represents the device coordinate system of the motion capture device attached to the thigh, O L2 This represents the device coordinate system of another motion capture device attached to the lower thigh.

[0134] Selectively, in this application, before determining the orientation information of the device coordinate system relative to the limb coordinate system, the user is first guided to perform a pre-set calibration action (or a pre-set calibration posture) and hold it for a certain period of time, thereby ensuring the relative posture between the user's face and the limbs to which the motion capture device is attached. The pre-set calibration action can be selected from, but is not limited to, a T-pose, an N-pose, other actions that ensure the user's face, chest, and lower limbs (i.e., legs) are facing the same direction, or actions that ensure the relative posture between the user's face and the limbs to which the motion capture device is attached. Subsequently, the orientation information of the device coordinate system relative to the limb coordinate system is determined based on the pre-set action. The certain period of time in this application can be flexibly set according to the actual calculation requirements. For example, it may be 1 second, 2 seconds, or 3 seconds, but this application does not impose any restrictions on this.

[0135] The T-pose described above is understood as a posture in which both arms are extended horizontally to the left and right, forming a T-shape. The N-pose described above is understood as a posture similar to an upright posture.

[0136] In a selective implementation method, determining the orientation information of the device coordinate system relative to the limb coordinate system includes the following steps:

[0137] In Step 1, the attitude transformation relationship between the limb coordinate system and the head-mounted coordinate system, as well as the attitude information of the head-mounted coordinate system relative to the world coordinate system, are determined using a pre-set calibration posture. The head-mounted coordinate system is the coordinate system of the head-mounted display device.

[0138] A pre-set calibration posture can be understood as the pre-set calibration actions described above, such as T-pose, N-pose, other actions that ensure the user's face, chest, and lower limbs (i.e., legs) are facing the same direction, or other actions that determine the relative posture between the user's face and the limbs to which the motion capture device is attached.

[0139] In some selective embodiments, after the user performs a pre-set calibration posture, the positions of the human limbs and head remain fixed. Correspondingly, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is also fixed. That is, at the point when the user performs a pre-set action, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is a known quantity, which is usually the localization number, and this constant is the identity matrix.

[0140] In some implementations, a camera on a head-mounted display device collects ambient environmental data in real time, and the head-mounted display device can determine its position and orientation information in real time based on the ambient environmental data collected by the camera using a computer vision SLAM algorithm. Therefore, in this application, when determining the orientation information of the head-mounted coordinate system relative to the world coordinate system, it can be determined using a computer vision SLAM algorithm based on the ambient environmental data collected by the camera. The specific determination process is a technique commonly used in this field and will not be described again here.

[0141] In step 2, the attitude information of the device coordinate system relative to the limb coordinate system is determined based on the attitude transformation relationship between the limb coordinate system and the head-mounted coordinate system, the attitude information of the head-mounted coordinate system relative to the world coordinate system, and the attitude information of the device coordinate system relative to the world coordinate system.

[0142] Selectively, the orientation information of the device coordinate system relative to the limb coordinate system can be realized by the following equation (3).

number

[0143] Here,

number

number

number

number

[0144] The above

number

number

number

number

[0145] The above t1 is the collection time of the motion capture device image or the light spot image, or the time when the motion capture device image or the light spot image is collected and the calibration operation of the motion capture device is completed. t2 is the time to determine the pose of the device coordinate system with respect to the limb coordinate system in a preset calibration pose. It should be understood that t2 may be later than t1, that is, t1 < t2 may also be true.

[0146] In S304 of the above embodiment, considering that the pose transformation relationship between the corrected world coordinate system and the reference coordinate system has already been determined, this pose transformation relationship is fixed during calibration or until the next calibration after the previous calibration. Also, since the inertial measurement unit of the motion capture device transmits inertial measurement unit data in real time, based on the received inertial measurement unit data, the initial pose information of the device coordinate system with respect to the reference coordinate system at time t2 can be calculated. Furthermore, in the present application, based on the initial pose information of the device coordinate system with respect to the reference coordinate system at time t2 and the pose transformation relationship between the corrected world coordinate system and the reference coordinate system, the pose information of the i-th world coordinate system with respect to the device coordinate system corresponding to time t2 when the user performs the preset calibration pose can be determined.

[0147] To determine the pose information of the device coordinate system with respect to the world coordinate system based on the initial pose information of the device coordinate system with respect to the reference coordinate system at time t2 and the pose transformation relationship between the corrected world coordinate system and the reference coordinate system, specifically, reference can be made to S305 of the above embodiment, and it will not be repeatedly described here.

[0148] In S402, based on the pose information of the device coordinate system with respect to the limb coordinate system and the pose information of the device coordinate system with respect to the world coordinate system, the pose transformation relationship between the limb coordinate system and the world coordinate system is determined.

[0149] In several selective implementation methods, determining the attitude transformation relationship between the limb coordinate system and the world coordinate system can be achieved by the following equation (4).

number

[0150] Here,

number

number

number

[0151] Furthermore, the limb coordinate system may be calibrated based on the attitude transformation relationship between the limb coordinate system and the world coordinate system.

[0152] The attitude transformation relationship between the limb coordinate system and the world coordinate system described above should be understood as attitude information of the limb coordinate system relative to the world coordinate system, and it should be understood that this attitude information represents the attitude of the limbs in the world coordinate system.

[0153] In several selective implementation methods, regarding the execution order of the orientation information of the device coordinate system relative to the limb coordinate system and the calibration of the motion capture device in this application, the calibration operation for the motion capture device may be performed first, and then the orientation information of the device coordinate system relative to the limb coordinate system may be determined, thereby acquiring the orientation information of the limb coordinate system relative to the world coordinate system and realizing the motion capture function of the corresponding limbs. Alternatively, the orientation information of the device coordinate system relative to the limb coordinate system may be determined first, and then the calibration operation for the motion capture device may be performed, thereby acquiring the orientation information of the limb coordinate system relative to the world coordinate system and realizing the motion capture function of the corresponding limbs, but this application does not specifically limit itself to this.

[0154] When first determining the orientation information of the device coordinate system relative to the limb coordinate system, and then performing a calibration operation on the motion capture device, in this application, first, the orientation information of the device coordinate system relative to the limb coordinate system at time t2 is cached, and then, during the calibration operation, the calibration operation is performed on the motion capture device based on the cached orientation information of the device coordinate system relative to the limb coordinate system at time t2.

[0155] As described above, determining the orientation information of the device coordinate system relative to the limb coordinate system can be achieved by the following equation (5).

number

[0156] Here,

number

number

number

number

number

[0157] Furthermore, the motion capture device can be fixed relative to the motion capture device coordinate system, and the relationship between the motion capture device coordinate system and the reference coordinate system or world coordinate system is equivalent to the relationship of the motion capture device relative to the reference coordinate system or world coordinate system.

[0158] In the calibration method for a motion capture device provided in the embodiment of the present invention, a camera collects light spot images of light-emitting units on the motion capture device attached to the limbs of the body. Based on the light spot images, target orientation information of the device coordinate system relative to the world coordinate system is determined, and the motion capture device is calibrated based on the target orientation information. In this invention, by performing a calibration operation on the motion capture device using light spot images of light-emitting units on the motion capture device, the difficulty of calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, and calibration errors can be reduced, improving the calibration accuracy of the motion capture device. As a result, the motion capture effect of the limbs can be improved based on the calibrated motion capture device, contributing to improved accuracy in human body posture reproduction. Furthermore, in this invention, the limb coordinate system can also be calibrated based on the posture transformation relationship between the limb coordinate system and the world coordinate system, and the posture of the limbs in the world coordinate system can be determined based on the calibrated limb coordinate system. As a result, the accuracy and validity of determining the posture of the human limbs in the world coordinate system can be improved, and the motion capture effect of the limbs can be enhanced.

[0159] The calibration apparatus for a motion capture device presented in the embodiment of the present application will be described below with reference to Figure 10. Figure 10 is a schematic block diagram of the calibration apparatus for a motion capture device provided in the embodiment of the present application.

[0160] The motion capture device is attached to the limbs of a human body, the device is placed on a head-mounted display device, and the head-mounted display device includes a camera. As shown in Figure 10, the calibration device 500 of the motion capture device includes an image acquisition module 510, a posture determination module 520, and a device calibration module 530.

[0161] The image acquisition module 510 is used to collect images from the motion capture device using the camera.

[0162] The attitude determination module 520 is used to determine target attitude information for the device coordinate system relative to the world coordinate system based on the motion capture device image. The device coordinate system is the motion capture device coordinate system.

[0163] The device calibration module 530 is used to calibrate the motion capture device based on the target attitude information.

[0164] In a selective implementation method of the embodiment of the present application, the attitude determination module 520 specifically, Image feature points are extracted from the motion capture device image, Based on the aforementioned image feature points, the target attitude information of the device coordinate system relative to the world coordinate system is used to determine the target attitude information of the device coordinate system.

[0165] In a selective implementation method of the embodiment of the present application, the motion capture device includes a light-emitting unit, an image acquisition module 510, and specifically, The aforementioned camera is used to collect a light spot image of the light-emitting unit on the motion capture device.

[0166] Relatively speaking, the attitude determination module 520 is, specifically, The aforementioned light spot image is used to determine the target attitude information of the device coordinate system relative to the world coordinate system.

[0167] In a selective implementation method of the embodiment of the present application, the attitude determination module 520 is A position and orientation estimation algorithm determines the target position and orientation information of the device coordinate system relative to the world coordinate system based on the light spot image. It is also used to extract attitude information from the aforementioned target position and attitude information, and to determine the attitude information as the target attitude information for the device coordinate system relative to the world coordinate system.

[0168] In a selective implementation method of the embodiment of the present application, the apparatus 500 is The system further includes a first determination module used to determine the initial attitude information of the device coordinate system relative to a reference coordinate system, based on inertial measurement unit data transmitted from the motion capture device.

[0169] Relatively speaking, the device calibration module 530 specifically, Based on the target attitude information and the initial attitude information, the attitude transformation relationship between the world coordinate system and the reference coordinate system is corrected. This is used to calibrate the motion capture device based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system.

[0170] In a selective implementation method of the embodiment of the present application, the device calibration module 530 is It is also used to correct the azimuth angle in the attitude transformation relationship between the aforementioned world coordinate system and the aforementioned reference coordinate system.

[0171] In a selective implementation method of the embodiment of the present application, the inertial measurement unit data includes accelerometer data, gyroscope data, and magnetometer data. The first decision module specifically includes, A complementary filter algorithm is used to determine the initial attitude information of the device coordinate system relative to the reference coordinate system, based on the accelerometer data, gyroscope data, and magnetometer data transmitted from the motion capture device.

[0172] In a selective implementation method of the embodiment of the present application, the apparatus 500 is A second determination module for determining the orientation information of the device coordinate system relative to the limb coordinate system, The system includes a third determination module used to determine the attitude transformation relationship between the limb coordinate system and the world coordinate system based on attitude information of the device coordinate system relative to the limb coordinate system and attitude information of the device coordinate system relative to the world coordinate system.

[0173] In the selective implementation method of the embodiment of the present application, the second decision module specifically, It is used to determine the attitude transformation relationship between the limb coordinate system and the head-mounted coordinate system, and the attitude information of the head-mounted coordinate system relative to the world coordinate system, in a pre-set calibration posture, and the head-mounted coordinate system is the head-mounted display device coordinate system. This is used to determine the attitude information of the device coordinate system relative to the limb coordinate system, based on the attitude transformation relationship between the limb coordinate system and the head-mount coordinate system, the attitude information of the head-mount coordinate system relative to the world coordinate system, and the attitude information of the device coordinate system relative to the world coordinate system.

[0174] In a selective implementation method of the embodiment of the present application, the second decision module is: This further includes determining the attitude information of the device coordinate system relative to the world coordinate system based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system.

[0175] In the selective implementation method of the embodiment of the present invention, the number of light spots on the motion capture device is at least three.

[0176] A selective implementation method of the embodiments of this application is: The system further includes a display module used to display an image perspective view and, in the image perspective view, to display guide information for guiding a user to position the motion capture device in the image perspective view.

[0177] The calibration device for a motion capture device provided in the embodiment of this application collects images of the motion capture device attached to the limbs of a human body using a camera, determines target orientation information of the device coordinate system relative to the world coordinate system based on the motion capture device images, and then calibrates the motion capture device based on the target orientation information. In this application, by performing calibration operations on the motion capture device using the motion capture device images, the difficulty of calibrating the motion capture device is effectively reduced, the calibration time is shortened, calibration errors are reduced, and the calibration accuracy of the motion capture device can be improved. As a result, the motion capture effect of the limbs can be improved based on the calibrated motion capture device, contributing to improved accuracy in reproducing human body posture.

[0178] The embodiments of the apparatus and the embodiments of the method correspond to each other, and it should be understood that similar descriptions can be found by referring to the embodiments of the method; therefore, to avoid duplication, they will not be repeated here. Specifically, the apparatus 500 shown in Figure 10 is capable of realizing the embodiment of the method corresponding to Figure 3, and the aforementioned and other operations and / or functions of each module within the apparatus 500 are used to realize the corresponding processes in each method of Figure 3, respectively. For the sake of brevity, they will not be repeated here.

[0179] In the above description, the apparatus 500 of the embodiment of the present application has been described from the perspective of a functional module with reference to the attached drawings. It should be understood that the functional module may be implemented in hardware form, in software form by instructions, or in combination of hardware and software modules. Specifically, each step of the method embodiment of the first embodiment of the present application can be completed by integrated logic circuits, which are hardware in the processor, and / or by instructions, which are in software form, and the steps of the method of the first embodiment disclosed in the present application are immediately embodied by the execution of a hardware decode processor or by a combination of hardware and software modules in the decode processor. Selectively, the software module can be located in a storage medium commonly used in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory, and the processor reads information in the memory and, in combination with its hardware, implements the steps of the method embodiment of the first embodiment.

[0180] Figure 11 is a schematic block diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 11, the electronic device 600 includes a memory 610 and a processor 620, the memory 610 being used to store a computer program and to transfer the program code to the processor 620. In other words, the processor 620 can implement the calibration method for the motion capture device in an embodiment of the present application by calling and executing the computer program from the memory 610.

[0181] For example, the processor 620 can be used to perform an embodiment of the calibration method for the motion capture device in accordance with instructions in the computer program.

[0182] In some embodiments of the present application, the processor 620 includes, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistor logic devices, discrete hardware components.

[0183] In some embodiments of the present application, the memory 610 includes, but is not limited to, volatile memory and / or non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), or flash memory. Volatile memory includes random access memory (RAM) and is used as external cache memory. By illustrative and non-exclusive description, various forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synch-link dynamic random access memory (synch-link DRAM, SLDRAM), and direct rambus random access memory (Direct Rambus RAM, DR RAM).

[0184] In some embodiments of the present invention, the computer program is divisible into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the calibration method for a motion capture device provided by the present invention. The one or more modules may be a series of computer program instruction segments capable of completing a specific function, which are used to record the execution process of the computer program in the electronic device.

[0185] As shown in Figure 11, the electronic device 600 is The system may further include a transceiver 630 that can be connected to the processor 620 or memory 610.

[0186] The processor 620 can control the transceiver 630 to communicate with other devices, specifically by transmitting information or data to or receiving information or data transmitted from other devices. The transceiver 630 includes a transmitter and a receiver. The transceiver 630 may further include antennas, one or more in number.

[0187] It should be understood that each assembly within the electronic device is connected via a bus system, and that the bus system further includes a power bus, a control bus, and a status signal bus, in addition to the data bus.

[0188] The present invention further provides a computer storage medium on which a computer program is stored, and which, when the computer program is executed by a computer, causes the computer to execute the method described in the above embodiment of the method.

[0189] Embodiments of the present application provide a computer program product including program instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in the embodiment of the method.

[0190] When implemented by software, the entire or partial implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. Loading and executing the computer program instructions on a computer generates all or part of the flow or function relating to the embodiments of this application. The computer is a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions are stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions are transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired connection (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless connection (e.g., infrared, radio, microwave, etc.). The computer-readable storage medium is any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The usable media include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid state drives (SSDs)).

[0191] Those skilled in the art will understand that the steps of the modules and algorithms described in the embodiments disclosed herein can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the art. Those skilled in the art can implement the described functions using different methods depending on the specific application, and such implementations should not be considered beyond the scope of this application.

[0192] In some embodiments provided herein, it should be understood that the disclosed systems, apparatus, and methods are achievable in other forms. For example, the embodiments of the apparatus described above are merely illustrative, and the division of the modules is merely a division of logical functions; other division methods may be employed in actual implementation. For example, multiple modules or assemblies may be combined or integrated into another system, or some features may be ignored or omitted. Furthermore, the mutual coupling, direct coupling, or communication connection shown or considered may be an indirect coupling or communication connection via some interface, apparatus, or module, and may be electrical, mechanical, or otherwise. Modules described as separate components may or may not be physically separated. Components shown as modules may or may not be physical modules. That is, they may be located in one place or distributed across multiple network units. In practice, some or all of the modules can be selected as needed to achieve the objectives of this embodiment. For example, each functional module in each embodiment of this application may be integrated into a single processing module, or each module may exist physically independently, or two or more modules may be integrated into a single module.

[0193] The above are merely specific embodiments of the present application, and the scope of protection of this application is not limited thereto. Any modification or substitution that a person skilled in the art can easily conceive within the technical scope disclosed in this application should be included in the scope of protection of this application. Accordingly, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for calibrating a motion capture device, The steps include: collecting images of the motion capture device attached to the limbs of a human body using a camera included in a head-mounted display device to which the above method is applied; The step of determining target attitude information of the device coordinate system relative to the world coordinate system based on the image of the motion capture device, wherein the device coordinate system is the coordinate system of the motion capture device, The steps include calibrating the motion capture device based on the target posture information, Methods that include...

2. The step of determining the target attitude information of the device coordinate system relative to the world coordinate system based on the image of the motion capture device is: The steps include extracting image feature points from the image of the motion capture device, The steps include determining the target attitude information of the device coordinate system relative to the world coordinate system based on the aforementioned image feature points, The method according to claim 1, including the method described in claim 1.

3. The motion capture device includes a light-emitting unit, The step of collecting images of the motion capture device using the aforementioned camera is: The step includes collecting a light spot image of the light-emitting unit on the motion capture device using the camera, The step of determining the target attitude information of the device coordinate system relative to the world coordinate system based on the image of the motion capture device is: The process includes the step of determining target attitude information of the device coordinate system relative to the world coordinate system based on the aforementioned light spot image. The method according to claim 1.

4. The step of determining the target attitude information of the device coordinate system relative to the world coordinate system based on the aforementioned light spot image is: The steps include determining target position and orientation information of the device coordinate system relative to the world coordinate system based on the light spot image using a position and orientation estimation algorithm, The steps include extracting attitude information from the aforementioned target position and attitude information, and determining the said attitude information as the target attitude information for the device coordinate system relative to the world coordinate system, The method according to claim 3, including the method described in claim 3.

5. The above method further, The step includes determining the initial attitude information of the device coordinate system relative to a reference coordinate system based on inertial measurement unit data transmitted from the motion capture device, The step of calibrating the motion capture device is: A step of correcting the attitude transformation relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information, A step of calibrating the motion capture device based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, The method according to claim 3, including the method described in claim 3.

6. The step of correcting the attitude transformation relationship between the world coordinate system and the reference coordinate system is: A step of correcting the azimuth angle in the attitude transformation relationship between the world coordinate system and the reference coordinate system, The method according to claim 5, including the method described in claim 5.

7. The inertial measurement unit data includes accelerometer data, gyroscope data, and magnetometer data. The step of determining the initial attitude information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data transmitted from the motion capture device is: A step of determining the initial attitude information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, gyroscope data, and magnetometer data transmitted from the motion capture device using a complementary filter algorithm. The method according to claim 5, including the method described in claim 5.

8. The aforementioned method, A step of determining the orientation information of the device coordinate system with respect to the limb coordinate system, A step of determining the attitude transformation relationship between the limb coordinate system and the world coordinate system based on the attitude information of the device coordinate system with respect to the limb coordinate system and the attitude information of the device coordinate system with respect to the world coordinate system. The method according to any one of claims 1 to 7, including

9. The step of determining the orientation information of the device coordinate system relative to the limb coordinate system is: The steps include determining the attitude transformation relationship between the limb coordinate system and the head-mounted coordinate system, and the attitude information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is the head-mounted display device coordinate system, and A step of determining the attitude information of the device coordinate system relative to the limb coordinate system based on the attitude transformation relationship between the limb coordinate system and the head-mount coordinate system, the attitude information of the head-mount coordinate system relative to the world coordinate system, and the attitude information of the device coordinate system relative to the world coordinate system. The method according to claim 8, including the method described in claim 8.

10. The step of determining the attitude information of the device coordinate system with respect to the world coordinate system is: Based on the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, determine the attitude information of the device coordinate system relative to the world coordinate system. The method according to claim 9, including the method described in claim 9.

11. The motion capture device has at least three light spots. The method according to claim 3.

12. The above method further, A step of displaying an image perspective view, and displaying guide information in the image perspective view to guide the user to position the motion capture device within the image perspective view. The method according to claim 1, including the method described in claim 1.

13. A calibration device for motion capture devices, An image acquisition module used to collect images of the motion capture device attached to the limbs of a human body by a camera included in the head-mounted display device to which the device is applied, An attitude determination module used to determine target attitude information of the device coordinate system relative to the world coordinate system based on the image of the motion capture device, wherein the device coordinate system is the coordinate system of the motion capture device, and A device calibration module used to calibrate the motion capture device based on the target posture information, A device including a device.

14. It is an electronic device, Including the processor and memory, The aforementioned memory stores computer programs. The processor, upon calling and executing the computer program, causes the electronic device to perform the method according to any one of claims 1 to 12. Electronic devices.

15. A computer used to store a computer program that causes a computer to carry out the method according to any one of claims 1 to 12, A computer-readable storage medium.

16. A computer program product that, when executed on an electronic device, includes a program instruction that causes the electronic device to perform the method according to any one of claims 1 to 12.

Images