Calibration method and apparatus for motion capture device, device, and medium

Abstract

The present application provides a calibration method and apparatus for a motion capture device, a device, and a medium. The motion capture device is worn on a human limb, the method is applied to a head-mounted display device, and the head-mounted display device comprises a camera. The method comprises: capturing an image of the motion capture device by means of the camera; on the basis of the image of the motion capture device, determining target attitude information of a device coordinate system relative to a world coordinate system; and calibrating the motion capture device on the basis of the target attitude information. According to the present application, by using an image of a motion capture device to calibrate the motion capture device, the difficulty in calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, the calibration error can be reduced, and the calibration accuracy of the motion capture device can be improved.

Description

Motion capture device calibration method, apparatus, equipment and medium

[0001] This application claims priority to the Chinese invention patent application entitled “Calibration method, apparatus, device and medium for motion capture equipment” filed on August 30, 2023, with application number 202311110278.X. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of motion capture technology, and in particular to a calibration method, apparatus, device, and medium for a motion capture device. Background Art

[0003] Human motion capture technology has enormous potential for applications in virtual reality (VR), human-computer interaction, and other fields. Current human motion capture solutions primarily use motion capture devices (such as inertial sensors or tracking devices equipped with inertial sensors) to capture real-world human motion data.

[0004] Before using a motion capture device to collect human motion data, it must be calibrated to ensure the accuracy and precision of the captured motion data. In related technologies, calibration requires the use of reference objects and a fixed position for the device. This approach undoubtedly increases user complexity and can be frustrating. It can also introduce calibration errors due to inaccurate positioning of the motion capture device, resulting in low calibration accuracy and poor results.

[0005] Summary of the Invention

[0006] The present application provides a motion capture device calibration method, apparatus, equipment and medium. By utilizing the motion capture device image to calibrate the motion capture device, the difficulty of calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, and the calibration error can be reduced, thereby improving the calibration accuracy of the motion capture device.

[0007] In a first aspect, an embodiment of the present application provides a calibration method for a motion capture device, wherein the motion capture device is worn on a human limb, and the method is applied to a head-mounted display device, wherein the head-mounted display device includes a camera, and the method includes:

[0008] Collecting the motion capture device image through the camera;

[0009] determining target posture information of a device coordinate system relative to a world coordinate system based on the motion capture device image, the device coordinate system being the motion capture device coordinate system;

[0010] The motion capture device is calibrated according to the target posture information.

[0011] In a second aspect, an embodiment of the present application provides a calibration device for a motion capture device, wherein the motion capture device is worn on a human limb, and the device is configured on a head-mounted display device, wherein the head-mounted display device includes a camera, including:

[0012] An image acquisition module, configured to acquire images of the motion capture device through the camera;

[0013] a posture determination module, configured to determine target posture information of a device coordinate system relative to a world coordinate system based on the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system;

[0014] The device calibration module is used to calibrate the motion capture device according to the target posture information.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0016] A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the motion capture device calibration method described in the first aspect embodiment or its various implementations.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the motion capture device calibration method as described in the embodiment of the first aspect or its various implementations.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product comprising program instructions, which, when executed on an electronic device, enables the electronic device to execute the motion capture device calibration method as described in the embodiment of the first aspect or its various implementations.

[0019] The technical solutions disclosed in the embodiments of this application have at least the following beneficial effects:

[0020] The camera captures images of a motion capture device worn on a human limb, determines the target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device images, and then calibrates the motion capture device based on the target pose information. This application utilizes the motion capture device images to calibrate the motion capture device, effectively reducing the difficulty of calibrating the motion capture device, shortening the calibration time, reducing calibration errors, and improving the calibration accuracy of the motion capture device. The calibrated motion capture device can thus improve the limb motion capture effect, providing conditions for improving the accuracy of human posture restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0023] FIG2 is a schematic diagram of a head-mounted display device, a motion capture device, and peripheral devices worn on different parts of a human body, provided by an embodiment of the present application;

[0024] FIG3 is a flow chart of a method for calibrating a motion capture device according to an embodiment of the present application;

[0025] FIG4 is a flow chart of another method for calibrating a motion capture device according to an embodiment of the present application;

[0026] FIG5 is a schematic diagram of the relationship between the world coordinate system and the device coordinate system provided in an embodiment of the present application;

[0027] FIG6 is a flow chart of another method for calibrating a motion capture device according to an embodiment of the present application;

[0028] FIG7 is a schematic diagram of the relationship between the world coordinate system, the reference coordinate system, and the device coordinate system provided in an embodiment of the present application;

[0029] FIG8 is a flow chart of another method for calibrating a motion capture device according to an embodiment of the present application;

[0030] FIG9 is a schematic diagram of the relationship between the device coordinate system, the world coordinate system, and the limb coordinate system provided in an embodiment of the present application;

[0031] FIG10 is a schematic block diagram of a calibration apparatus for a motion capture device provided in an embodiment of the present application;

[0032] FIG11 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0035] The present application is applicable to human motion capture scenarios. The current human motion capture solutions mainly use motion capture devices to capture motions in order to collect human motion data in the real world. The motion capture devices are basically inertial sensors or tracking devices with inertial sensors. In order to ensure the accuracy and precision of the motion data collected by the motion capture device, the motion capture device needs to be calibrated. Considering that the relevant technology requires the use of reference objects and the wearing position of the motion capture device to be fixed when calibrating the motion capture device, it will increase the difficulty of operation for the user and cause trouble to the user. It may even introduce calibration errors due to the inaccurate wearing position of the motion capture device, resulting in the problem of low calibration accuracy and poor effect of the motion capture device. Therefore, the present application designs a motion capture device calibration method, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error and improve the calibration accuracy of the motion capture device.

[0036] To facilitate understanding of the embodiments of the present application, before describing each embodiment of the present application, some concepts involved in all embodiments of the present application are first appropriately explained as follows:

[0037] 1) Virtual Reality (VR), a technology for creating and experiencing a virtual world, determines the generation of a virtual environment, which is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.), realizing the fusion of virtual environments, interactive three-dimensional dynamic vision and simulation of entity behavior, allowing users to immerse themselves in a simulated virtual reality environment, and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0038] 2) Virtual reality devices (VR devices), terminals that achieve virtual reality effects, can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this and can be further miniaturized or enlarged according to actual needs.

[0039] Optionally, the virtual reality devices described in the embodiments of this application may include but are not limited to the following types:

[0040] 2.1) PC-based virtual reality (PCVR) devices use the PC to perform calculations and output data related to virtual reality functions. External PC-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

[0041] 2.2) Mobile VR devices support the configuration of mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for VR functions and outputs data to the mobile VR device, such as viewing VR videos through the mobile terminal's app.

[0042] 2.3) All-in-one virtual reality devices have a processor for performing relevant calculations for virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom of use.

[0043] 3) Virtual field of view: the area in the virtual environment that the user can perceive through the lens in the virtual reality device. The field of view (FOV) of the virtual field of view is used to represent the perceived area.

[0044] 4) Augmented Reality (AR): A technology that uses real-time calculations of the camera's pose parameters in the real world (also known as the 3D world or the real world) as the camera captures images. Based on these pose parameters, virtual elements are added to the captured image. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world on the real world for interactive viewing.

[0045] 5) Mixed Reality (MR): A simulated setting that integrates computer-generated sensory input (e.g., virtual objects) with sensory input from a physical setting or its representation. In some MR settings, the computer-generated sensory input can adapt to changes in the sensory input from the physical setting. Furthermore, some electronic systems used to render MR settings can monitor the orientation and / or position relative to the physical setting to enable virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0046] 6) Extended Reality (XR) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices, including virtual reality (VR), augmented reality (AR) and mixed reality (MR).

[0047] 7) A virtual scene is a virtual scene displayed (or provided) when an application is running on an electronic device. The virtual scene can be a simulation of the real world, a semi-simulation and semi-fictitious virtual scene, or a purely fictitious virtual scene. The virtual scene can be any of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene. The embodiments of the present application do not limit the dimensions of the virtual scene. For example, a virtual scene can include the sky, land, ocean, etc., and the land can include environmental elements such as deserts and cities. Users can control virtual objects to move in the virtual scene.

[0048] 8) Virtual objects are objects that interact in virtual scenes and are controlled by users or robot programs (for example, robot programs based on artificial intelligence). They can be still, move, and perform various behaviors in the virtual scene, such as various characters in games.

[0049] In order to clearly illustrate the technical solution of this application, the application scenarios of the technical solution of this application are described below. It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited thereto:

[0050] For example, Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. As shown in Figure 1, the application scenario 1000 may include: a head-mounted display device 100 and a motion capture device 200. Furthermore, the head-mounted display device 100 and the motion capture device 200 can communicate with each other.

[0051] In some implementations, the head-mounted display device 100 may be an HMD, such as a head-mounted display in a VR all-in-one machine, etc. This embodiment does not impose any restrictions on this.

[0052] Furthermore, the head-mounted display device 100 is provided with a camera to collect information about the surrounding environment through the camera. Based on the collected surrounding environment data, the camera performs tracking and positioning using a simultaneous localization and mapping (SLAM) algorithm in computer vision. The number of the camera can be at least one, and FIG1 illustrates an example of four cameras. Furthermore, the type of the camera can be a fisheye camera, a standard camera, or other types of cameras, and this application does not impose any restrictions thereon.

[0053] In some implementations, the motion capture device 200 may be an inertial sensor, or any device having an inertial sensor.

[0054] In some possible implementations, the motion capture device 200 can be worn on human limbs, such as the limbs and torso. The limbs include upper limbs and lower limbs. The torso includes the waist, etc. This application does not impose any restrictions on human limbs. For example, as shown in Figure 2, the motion capture device 200 is worn on the lower limbs of the human body. The lower limbs of the human body include thighs and calves. In other words, four motion capture devices 200 can be worn on the thighs and calves of the human body respectively, so as to use the motion capture device 200 to collect the thigh motion data and calf motion data of the human body (collectively referred to as lower limb motion data), and send the lower limb motion data to the head-mounted display device 100 to achieve the purpose of tracking the movement of the lower limbs of the human body.

[0055] It should be noted that the limb data collected by the motion capture device 200 worn on the limbs of the human body may be 3-degree of freedom (Dof) data or 6-DoF data.

[0056] Considering that the upper limbs of the human body can also make different movements, in addition to wearing the motion capture device 200 on the upper limbs of the human body, the present application optionally allows the external device 300 to be worn on the upper limbs of the human body. For example, it can be worn on the hand and / or arm of the human body. Then, the upper limb motion data of the human body is collected by the external device 300 and sent to the head-mounted display device 100, thereby achieving the tracking of the upper limb motion of the human body. The wearing method of the external device 300 is specifically shown in Figure 2.

[0057] In some implementations, the peripheral device 300 may be, but is not limited to, a handle, a glove, a bracelet, a wristband, a finger ring, and other wearable devices.

[0058] In addition, the peripheral device 300 is provided with an inertial sensor, and the inertial sensor can provide 6-degree-of-freedom data including the position and posture of the upper limbs of the human body.

[0059] It should be noted that the inertial sensor involved in this embodiment specifically refers to an inertial measurement unit (IMU). Moreover, the IMU can be a six-axis IMU or a nine-axis IMU, which is not specifically limited here.

[0060] It can be understood that the embodiments of the present application can achieve full-body tracking of the human body by wearing motion capture devices 200 on the human body's limbs, or wearing external devices 300 on the human body's upper limbs and motion capture devices on the human body's lower limbs.

[0061] It should be understood that the head-mounted display device 100, the motion capture device 200, and the peripheral device 300 shown in Figures 1 and 2 are only illustrative and do not constitute specific limitations on the present application.

[0062] After introducing the application scenarios of the embodiments of the present application, the calibration method, apparatus, device and medium of a motion capture device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0063] Figure 3 is a flow chart of a motion capture device calibration method provided in an embodiment of the present application. This embodiment of the present application is applicable to motion capture device calibration scenarios for human motion capture. The motion capture device calibration method can be performed by a motion capture device calibration device. The motion capture device calibration device can be composed of hardware and / or software and can be integrated into a head-mounted display device.

[0064] In the embodiments of the present application, the head-mounted display device can be any electronic device that can reproduce human body movements. Optionally, the electronic device can be an extended reality (XR) device. The XR device can be a VR device, an augmented reality (AR) device, or a mixed reality (MR) device. This application does not specifically limit the type of electronic device.

[0065] As shown in FIG3 , the method may include the following steps:

[0066] S101, collecting images of a motion capture device through a camera.

[0067] Considering that a head-mounted display device may be equipped with multiple cameras, such as shown in Figure 1, when there are multiple cameras on the head-mounted display device, this application gives priority to using the downward-looking camera located on the head-mounted display device to capture the motion capture device image, thereby making it easier to capture the motion capture device image worn on the human limb using the camera closer to the side of the human limb.

[0068] Optionally, before executing S101, the user first wears the head-mounted display device and the motion capture device on different parts of the human body, such as wearing the head-mounted display device on the human head, wearing the motion capture device on the human lower limbs, and wearing the external device on the human upper limbs. For details, please refer to the aforementioned Figure 2. For another example, the head-mounted display device is worn on the human head, and the motion capture device is worn on the human upper limbs and the human lower limbs. For another example, the head-mounted display device is worn on the human head, and the motion capture device is worn on the human upper limbs, waist, and human lower limbs, etc.

[0069] After wearing the head-mounted display device, motion capture device and external devices, the user can start the head-mounted display device, motion capture device and external devices to establish communication connections between the head-mounted display device and each motion capture device, as well as communication connections between the head-mounted display device and external devices, laying the foundation for subsequent data communication.

[0070] Among them, the specific implementation methods of establishing the communication connection between the head-mounted display device and each motion capture device, as well as the communication connection between the head-mounted display device and the external device can refer to the existing establishment method, and this application will not go into details about this.

[0071] Furthermore, the head-mounted display device can capture images of motion capture devices worn on human limbs through a camera. Considering that the captured motion capture device images include not only the motion capture device but also the human limb on which the motion capture device is worn, a frame of motion capture device images captured by the camera on the head-mounted display device in this application may include at least one motion capture device. Thus, the head-mounted display device can distinguish each motion capture device in the motion capture device image based on the motion capture devices in the motion capture device image and the human limb to which each motion capture device belongs. Furthermore, based on each motion capture device in the motion capture device image, the motion capture device worn on the human limb can be calibrated.

[0072] In some optional embodiments, when the present application uses a camera on a head-mounted display device to capture a motion capture device image, it is optional to only capture one motion capture device image in a time period, and calibrate the motion capture device based on the motion capture device image. This can reduce the image processing complexity of the head-mounted display device and improve the calibration efficiency of the motion capture device.

[0073] For example, in a first time period, a motion capture device image is captured on the right calf. In a second time period, a motion capture device image is captured on the right thigh. In a third time period, a motion capture device image is captured on the left calf. In a fourth time period, a motion capture device image is captured on the left thigh. The order of the first, second, third, and fourth time periods can be first time period → second time period → third time period → fourth time period. This order can be flexibly adjusted according to actual needs, and this application does not impose any restrictions on this.

[0074] In some optional embodiments, considering that the head-mounted display device can support the Video See-Through (VST) function, based on the VST function, the user can see a virtualized real space and real objects in proportion to the real world through the head-mounted display device. The virtualized real objects include but are not limited to tables, chairs, etc. Therefore, before the present application uses the camera to capture the image of the motion capture device, the optional head-mounted display device can use the VST function to display a video perspective screen in proportion to the real world. At the same time, guidance information guiding the user to place any motion capture device in the video perspective screen is displayed in any area of ​​the video perspective screen, so that the user can place the above-mentioned motion capture device in the video perspective screen based on the guidance information. Therefore, when the user places any motion capture device in the video perspective screen according to the guidance information, the camera on the head-mounted display device can capture the above-mentioned motion capture device image. The advantage of this setting is that it uses the VST function to output guidance information to the user, so as to guide the user to make corresponding actions according to the guidance information, and the user can see his own actions through the image perspective function, so that the user's actions can be visualized, and the interaction between the user and the head-mounted display device is more intuitive, which improves the ease of motion capture device calibration.

[0075] It should be noted that, considering that the calibration principles and calibration processes of various motion capture devices worn on human limbs are the same, in order to simplify the description, this application will describe the technical solution of this application in detail in the following embodiments by taking the calibration of one of the motion capture devices as an example.

[0076] S102 : Determine target posture information of a device coordinate system relative to a world coordinate system based on the motion capture device image, where the device coordinate system is a motion capture device coordinate system.

[0077] Among them, the device coordinate system can be represented by Li, where L stands for Local, and i is the identification information of the motion capture device. The identification information can be any information that can uniquely determine the identity of the motion capture device, such as a number or name, etc., and there is no specific restriction on it here.

[0078] The world coordinate system refers to the global world coordinate system, which can be represented by G, where the full name of G is Global.

[0079] In some possible implementations, the relationship between the world coordinate system and the device coordinate system may be as shown in FIG4 . G is the world coordinate system, O L1 is the coordinate system of a motion capture device worn on a thigh of the human lower limb, and O L2 Represents the coordinate system of another motion capture device worn on the calf associated with the above thigh.

[0080] Optionally, after acquiring the motion capture device image captured by the camera, the head-mounted display device can extract image feature points from the motion capture device image and then determine the target pose information of the device coordinate system relative to the world coordinate system based on the extracted image feature points.

[0081] Considering that the motion capture device image captured by the camera may include at least one motion capture device, the head-mounted display device may use different methods to extract image feature points from the motion capture device image according to the number of motion capture devices in the motion capture device image, as follows:

[0082] In a first manner, when there is only one motion capture device in the motion capture device image, the head mounted display device extracts image feature points from the motion capture device image.

[0083] In a second manner, when the motion capture device image includes at least two motion capture devices, the head mounted display device extracts image feature points of each motion capture device from the motion capture device image.

[0084] In the present application, the above-mentioned image feature points can be relatively prominent or representative points in the motion capture device image, such as the corner points, contour points, dark points in brighter areas, and bright points in darker areas of the motion capture device in the motion capture device image. This application does not impose any restrictions on image feature points.

[0085] Furthermore, the image feature points extracted from the motion capture device image are 2-dimensional points (ie, 2D points).

[0086] As an optional implementation, the present application may employ a feature point extraction algorithm to extract image feature points from motion capture device images. Exemplary feature point extraction algorithms include, but are not limited to, SIFT (Scale Invariant Feature Transform) extraction algorithms, SURF (Speeded-Up Robust Features) extraction algorithms, and ORB (Oriented FAST and Rotated BRIEF) extraction algorithms.

[0087] After extracting image feature points from the motion capture device image, the head-mounted display device can determine the target pose information of the device coordinate system corresponding to the motion capture device relative to the world coordinate system based on the image feature points extracted from the motion capture device image. In other words, it determines the pose conversion relationship between the device coordinate system and the world coordinate system.

[0088] Considering that the model, specifications and shape of the motion capture device worn on the human body are known. Therefore, it can be determined that the 3D model (i.e., 3D model) of the motion capture device is also known. Therefore, the present application is based on extracting the image feature points of any motion capture device from the motion capture device image, and the 3D model of the motion capture device, and can determine the 3D points corresponding to the 2D image feature points of the motion capture device. Then, using the preset calculation rules, the posture conversion relationship between the device coordinate system and the world coordinate system is calculated based on the 2D points and 3D points of the motion capture device.

[0089] The above-mentioned preset calculation rules can be selected as any algorithm or strategy that can calculate the posture conversion relationship between the device coordinate system and the world coordinate system based on 2D points and 3D points. This application does not impose any restrictions on this.

[0090] In this application, the above-mentioned 2D points can be understood as the projections of the 3D points on the imaging plane, and the 3D points can be understood as the 3D coordinate points in the world coordinate system.

[0091] Because the pose information of the head-mounted coordinate system relative to the world coordinate system is known, when the preset calculation rule is a perspective-n-point (PnP) algorithm in computer vision, the head-mounted display device can use the PnP algorithm to calculate the target pose information of the device coordinate system relative to the world coordinate system based on n points (2D and 3D points of the motion capture device in the motion capture device image). Then, the target pose information of the device coordinate system relative to the world coordinate system is extracted from the target pose information.

[0092] Generally, the PnP algorithm may include: P3P algorithm, P5P algorithm and other algorithms. Moreover, the above-mentioned P3P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on three points. The above-mentioned P5P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on five points. Therefore, in the present application, the number of motion capture device feature points extracted from the motion capture device image is at least three, and correspondingly, at least three 3D points need to be determined based on the above-mentioned at least three 2D points. Therefore, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device image captured by the camera, different PnP algorithms can be selected based on the number of image feature points of the motion capture device in the motion capture device image.

[0093] S103: Calibrate the motion capture device according to the target posture information.

[0094] After determining the target posture information of the device coordinate system relative to the world coordinate system, the present application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the subsequent limb movements captured by the calibrated motion capture device are more accurate.

[0095] In some optional embodiments, the posture of the motion capture device relative to the world is calibrated. Optionally, the initial posture of the motion capture device relative to the world is calibrated, or the real-time posture of the motion capture device relative to the world is calibrated. This application does not impose any restrictions on this.

[0096] The motion capture device calibration method provided in an embodiment of the present application uses a camera to capture an image of the motion capture device worn on a human limb, determines target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device image, and then calibrates the motion capture device based on the target pose information. By utilizing the motion capture device image to calibrate the motion capture device, the present application can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve the calibration accuracy of the motion capture device. As a result, the calibrated motion capture device can improve the effect of capturing limb motion, providing conditions for improving the accuracy of human posture restoration.

[0097] In another optional implementation scenario, the motion capture device may include a light-emitting unit. Specifically, the motion capture device may be an inertial sensor with a light-emitting unit, or any device with both an inertial sensor and a light-emitting unit, such as an optical tracker. Therefore, the present application optionally uses a camera to capture a light spot image of the light-emitting unit on the motion capture device image, and calibrates the motion capture device based on the light spot image.

[0098] It should be noted that the light-emitting unit on the above-mentioned motion capture device can be selected as an LED or other light-emitting device, and the light emitted can be visible light or invisible light, as long as it can be captured by the camera. In addition, the above-mentioned light-emitting unit can be distributed in a preset deployment manner inside the shell of the motion capture device body or in an accessory rigidly connected to the motion capture device body, such as a base. In this application, the above-mentioned preset deployment method can be flexibly set according to the shape of the motion capture device body or the accessory rigidly connected to the motion capture device body, and there is no restriction on it here.

[0099] Furthermore, when the motion capture device worn on a human limb includes a light-emitting unit, the limb data collected by the motion capture device includes two categories:

[0100] The first category is that when the light-emitting unit on the motion capture device is in the light-on state and is observed by the camera on the head-mounted display device, the limb motion data collected by the motion capture device is 6-degree-of-freedom (Dof) data including the position and posture of the human limbs.

[0101] The second category is when the light-emitting unit on the motion capture device is in the lit state and cannot be observed by the camera on the head-mounted display device. At this time, the limb motion data collected by the motion capture device is 3-degree-of-freedom (Dof) data including the human limb posture.

[0102] As shown in the example of FIG2 , the lower limb motion data collected by the motion capture device 200 includes two categories: the first category, when the light-emitting unit on the motion capture device 200 is in the illuminated state and is observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 6-degree-of-freedom (Dof) data including the position and posture of the lower limbs of the human body. The second category, when the light-emitting unit on the motion capture device 200 is in the illuminated state and cannot be observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 is 3-degree-of-freedom (Dof) data including the posture of the lower limbs of the human body.

[0103] The process of calibrating the motion capture device based on the spot image described above is described in detail below with reference to FIG5 . As shown in FIG5 , the method includes the following steps:

[0104] S201, collecting a light spot image of a light-emitting unit on a motion capture device through a camera.

[0105] Considering that each motion capture device worn on a human limb is equipped with a light-emitting unit, if the light-emitting units on each motion capture device are lit simultaneously, and the camera on the head-mounted display device simultaneously captures the light spot images of the light-emitting units on all motion capture devices, the head-mounted display device may not be able to accurately distinguish which motion capture device each light spot corresponds to based on the light spot images captured by the camera, thereby causing confusion in the calibration of the motion capture devices, or even calibration errors.

[0106] Based on the above reasons, when calibrating all motion capture devices, this application adopts time-sharing lighting of the light-emitting units on the motion capture devices, that is, each motion capture device is lit in turn in a certain order in each cycle, so that the camera on the head-mounted display device only collects the light spot image of the light-emitting unit on one motion capture device in a time period, and uses the light spot image to calibrate the motion capture device, thereby ensuring that the calibration of each motion capture device is more accurate.

[0107] For example, in time period 1, the light-emitting unit in the motion capture device worn on the left thigh is lit, and the light-emitting units in the motion capture devices worn on the right thigh, right calf, and left calf are controlled to be in an off state. In time period 2, the light-emitting unit in the motion capture device worn on the left calf is lit, and the light-emitting units in the motion capture devices worn on the right thigh, right calf, and left thigh are controlled to be in an off state. In time period 3, the light-emitting unit in the motion capture device worn on the right thigh is lit, and the light-emitting units in the motion capture devices worn on the right calf, left thigh, and left calf are controlled to be in an off state. In time period 4, the light-emitting unit in the motion capture device worn on the right calf is lit, and the light-emitting units in the motion capture devices worn on the left thigh, left calf, and right thigh are controlled to be in an off state. Among them, the order of time period 1, time period 2, time period 3, and time period 4 can be selected as time period 1 → time period 2 → time period 3 → time period 4. Of course, it can also be other orders, and there is no specific limitation on it here.

[0108] The above-mentioned lighting of the light-emitting unit on the motion capture device can be manually triggered by the user, or the head-mounted display device can light up the light-emitting unit on the motion capture device by sending a lighting instruction to the motion capture device. This application does not impose any restrictions on this.

[0109] In some optional implementations, when the camera is used to capture the light spot image of the light-emitting unit on the motion capture device in the lit state, if the user's top is loose or other conditions may cause the camera on the head-mounted display device to be unable to capture the light spot image of the light-emitting unit on the motion capture device in the lit state.

[0110] To this end, after lighting up the light-emitting unit on any motion capture device, the present application guides the user to make a head-lowering action, a kicking action, and / or an arm-raising action, so that the camera on the head-mounted display device can capture the light spot image of the light-emitting unit on the motion capture device. Among them, the lower limb that makes the kicking action or the arm-raising action should be the lower limb or upper limb that is wearing the motion capture device in a lit state. For example, assuming that the light-emitting unit in the motion capture device on the left calf is in a lit state, then the user is guided to make a kicking action with his left calf. For another example, assuming that the light-emitting unit in the motion capture device on the right forearm is in a lit state, then the user is guided to raise his right forearm.

[0111] It can be understood that the above-mentioned lowering head action, kicking leg action and / or raising arm action can be a lowering head action; or, it can be a kicking leg action; or, it can be an arm raising action; or, it can be a lowering head action and an arm raising action; or, it can be a lowering head action and a kicking leg action, etc., and there is no limitation on it here.

[0112] In addition, in the present application, when guiding the user to lower the head, kick the legs and / or raise the arms, the user may be output with guiding voice, guiding text and / or guiding animation, etc., and there is no limitation on this here.

[0113] In some optional embodiments, considering that the head-mounted display device can support the Video See-Through (VST) function, based on the VST function, the user can see a virtualized real space and real objects in proportion to the real world through the head-mounted display device. Among them, the virtualized real objects include but are not limited to tables, chairs, etc. Therefore, before the present application uses the camera to capture the light spot image of the light-emitting unit on the motion capture device, the optional head-mounted display device can use the VST function to display a video perspective screen in proportion to the real world. At the same time, guidance information guiding the user to place any motion capture device with a lit light-emitting unit in the video perspective screen is displayed in any area of ​​the video perspective screen, so that the user can place the motion capture device with the light-emitting unit in the lit state in the video perspective screen based on the guidance information. Therefore, when the user places any motion capture device with a lit light-emitting unit in the video perspective screen according to the guidance information, the camera on the head-mounted display device can capture the light spot image of the light-emitting unit on the motion capture device. The advantage of this setting is that it uses the VST function to output guidance information to the user, so as to guide the user to make corresponding actions according to the guidance information, and the user can see his or her own actions through the image perspective function, so that the user's lowering head, kicking and / or arm raising movements are visualized, and the interaction with the head-mounted display device is more intuitive, which improves the ease of calibration of the motion capture device.

[0114] S202: Determine target posture information of the device coordinate system relative to the world coordinate system based on the light spot image.

[0115] Optionally, the head-mounted display device of the present application may adopt preset calculation rules to calculate the target posture information of the device coordinate system relative to the world coordinate system based on the spot image.

[0116] Taking into account that the pose information of the head-mounted coordinate system relative to the world coordinate system is known, the present application can use the pose estimation (Perspective-n-Point, abbreviated as PnP) algorithm in computer vision to calculate the pose information of the device coordinate system relative to the world coordinate system based on n points (n points corresponding to the light spot in the light spot image). Therefore, the present application can use the pose estimation algorithm to calculate the target pose information of the device coordinate system relative to the world coordinate system based on the light spot image captured by the camera when the deployment position of the light-emitting unit on the motion capture device is known. Then, the posture information is extracted from the target posture information, and the extracted posture information is determined as the target posture information of the device coordinate system relative to the world coordinate system.

[0117] Generally, PnP algorithms may include: P3P algorithm, P5P algorithm and other algorithms. Moreover, the above-mentioned P3P algorithm is based on three points to calculate the pose information of the device coordinate system relative to the world coordinate system. The above-mentioned P5P algorithm is based on five points to calculate the pose information of the device coordinate system relative to the world coordinate system. Therefore, in this application, the number of light spots on the motion capture device is at least three. Therefore, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the light spot image captured by the camera, different PnP algorithms can be selected based on the number of light spots in the light spot image.

[0118] The number of light spots on the motion capture device may be the same as or different from the number of light-emitting units on the motion capture device. If the number of light spots on the motion capture device is the same as the number of light-emitting units, the number of light-emitting units on the motion capture device is at least three. If the number of light spots on the motion capture device is different from the number of light-emitting units, the number of light-emitting units on the motion capture device is at least one.

[0119] In some optional embodiments, it is considered that the motion capture device may include a motion capture device body or include a motion capture device body and accessories rigidly connected to the motion capture device body, such as a base. Then, when the number of light-emitting units on the motion capture device is at least three, as an optional implementation method, the above-mentioned at least three light-emitting units may be distributed in the shell of the motion capture device body according to a preset deployment method. Alternatively, one light-emitting unit is installed in the shell of the motion capture device body according to a preset deployment method, and the remaining light-emitting units are installed in the base according to a preset deployment method. Alternatively, one light-emitting unit is installed in the base according to a preset deployment method, and the remaining light-emitting units are installed in the shell of the motion capture device body according to a preset deployment method. The above-mentioned preset deployment method can be flexibly set according to the shape of the motion capture device body and the base, and there is no restriction on it here.

[0120] S203: Calibrate the motion capture device according to the target posture information.

[0121] After determining the target posture information of the device coordinate system relative to the world coordinate system, the present application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the subsequent limb movements captured by the calibrated motion capture device are more accurate.

[0122] In some optional embodiments, the posture of the motion capture device relative to the world is calibrated. Optionally, the initial posture of the motion capture device relative to the world is calibrated, or the real-time posture of the motion capture device relative to the world is calibrated. This application does not impose any restrictions on this.

[0123] The motion capture device calibration method provided in an embodiment of the present application uses a camera to capture a spot image of a light-emitting unit on a motion capture device worn on a human limb, to determine target posture information of the device coordinate system relative to the world coordinate system based on the spot image, and then calibrates the motion capture device based on the target posture information. By utilizing the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, the present application can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve the calibration accuracy of the motion capture device. Therefore, the calibrated motion capture device can improve the limb motion capture effect, providing conditions for improving the accuracy of human posture restoration.

[0124] The following further explains the calibration method of the motion capture device provided in the embodiment of the present application in conjunction with Figure 6. As shown in Figure 6, the method includes the following steps:

[0125] S301 : Determine initial posture information of a device coordinate system relative to a reference coordinate system based on inertial measurement unit data sent by a motion capture device.

[0126] The reference coordinate system may be a geographic coordinate system or any user-defined coordinate system that differs from the world coordinate system. In this application, the roll angle (also called the roll angle) and pitch angle in the reference coordinate system are the same as the roll angle and pitch angle in the world coordinate system.

[0127] The reference coordinate system may be represented as ref. Furthermore, when the reference coordinate system is a geographic coordinate system, the geographic coordinate system may be a north-east geographic coordinate system.

[0128] In the embodiment of the present application, the relationship between the world coordinate system, the reference coordinate system and the device coordinate system can be shown in FIG7. G is the world coordinate system, O ref is the reference coordinate system, O L1is the coordinate system of a motion capture device worn on a thigh of the human lower limb, and O L2 Represents the coordinate system of another motion capture device worn on the calf associated with the above thigh.

[0129] In some optional embodiments, when the motion capture device is in operation, the inertial sensor (i.e., inertial measurement unit) on the motion capture device transmits inertial measurement unit data to the head-mounted display device in real time, so that the head-mounted display device calculates the initial posture information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data. In other words, the posture conversion relationship between the device coordinate system and the reference coordinate system.

[0130] Since the inertial measurement unit data sent by the motion capture device may include accelerometer data, gyroscope data, and magnetometer data, the present application may utilize a preset attitude solution algorithm, such as a complementary filtering algorithm, to determine the initial attitude information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, gyroscope data, and magnetometer data sent by the motion capture device.

[0131] The inertial measurement unit on the above-mentioned motion capture device sends the inertial measurement unit data to the head-mounted display device. The motion capture device may start to obtain the inertial measurement unit data through the inertial measurement unit and send the inertial measurement unit data when the camera on the head-mounted display device captures the light spot image; or, it may start at any time before the camera on the head-mounted display device captures the light spot image, and then the motion capture device continues to obtain the inertial measurement unit data through the inertial measurement unit and send the inertial measurement unit data. This application does not impose any restrictions on this.

[0132] In the present application, based on the inertial measurement unit data sent by the motion capture device, the number of initial posture information of the device coordinate system relative to the reference coordinate system can be one or more. Specifically, when the inertial sensor on the motion capture device only sends the inertial measurement unit data once, then the initial posture information of the device coordinate system relative to the reference coordinate system calculated at this time is one. When the inertial sensor on the motion capture device sends the inertial measurement unit data multiple times, then the initial posture information of the device coordinate system relative to the reference coordinate system calculated at this time is multiple. Among them, multiple initial posture information can be understood as the initial posture information of a device coordinate system relative to the reference coordinate system can be calculated each time the inertial measurement unit data is received.

[0133] Furthermore, the above-mentioned multiple initial posture information may be the same or different, which is specifically determined based on the user's motion state.

[0134] In this application, a plurality can be understood as two or more than two, that is, at least two.

[0135] Optionally, the initial posture information of the i-th device coordinate system relative to the reference coordinate system in this application can be expressed as: Where ref represents the reference coordinate system, t1 represents the t1th moment, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, and R represents the posture information.

[0136] S302: Capture a light spot image of a light-emitting unit on a motion capture device through a camera.

[0137] S303 : Determine target posture information of a device coordinate system relative to a world coordinate system according to the light spot image, where the device coordinate system is a motion capture device coordinate system.

[0138] S304: Correct the posture conversion relationship between the world coordinate system and the reference coordinate system according to the target posture information and the initial posture information.

[0139] In some optional embodiments, considering that the camera on the head-mounted display device is collecting the spot image, the motion capture device can synchronously send the inertial measurement unit data to the head-mounted display device. Therefore, the present application optionally corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system based on the target attitude information determined by the spot image at the same time and the initial attitude information determined by the inertial measurement unit data. This can be optionally achieved by the following formula (1):

[0140] in, G R ref Represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, Indicates the target posture information of the i-th device coordinate system relative to the world coordinate system at time t1, Indicates the initial posture information of the i-th device coordinate system relative to the reference coordinate system at time t1, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, R represents the posture information, and -1 represents inversion.

[0141] Considering the attitude conversion relationship between the world coordinate system and the reference coordinate system, it can be expressed by Euler angles. In addition, Euler angles may include heading angle, pitch angle, and roll angle. Therefore, the present application corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information, and may optionally correct the heading angle, pitch angle, and / or roll angle between the world coordinate system and the reference coordinate system.

[0142] It can be understood that the above-mentioned heading angle, pitch angle and / or roll angle can be the heading angle; or, it can be the pitch angle; or, it can be the roll angle; or, it can be the heading angle and the pitch angle; or, it can be the heading angle and the roll angle; or, it can be the pitch angle and the roll angle; or, it can be the heading angle, the pitch angle and the roll angle, and this application does not impose any specific restrictions on this.

[0143] In this application, the above-mentioned heading angle can be understood as the angle of rotation of the motion capture device around the X-axis, where the X-axis is perpendicular to the ground. The pitch angle can be understood as the angle of rotation of the motion capture device around the Y-axis, and the roll angle can be understood as the angle of rotation of the motion capture device around the Z-axis.

[0144] In some optional embodiments, when the attitude conversion relationship between the world coordinate system and the reference coordinate system is expressed using Euler angles, the pitch angle and roll angle in the world coordinate system and the reference coordinate system are exactly the same, and only the heading angle parameter item deviates. Therefore, the present application corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information. Specifically, the present application corrects the heading angle parameter item in the attitude conversion relationship between the world coordinate system and the reference coordinate system based on the target attitude information and the initial attitude information.

[0145] In the present application, correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system specifically includes: correcting the heading angle conversion relationship in the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information.

[0146] S305 , calibrating the motion capture device according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0147] Optionally, after correcting the posture conversion relationship between the world coordinate system and the reference coordinate system, the present application can determine new target posture information of the device relative to the world coordinate system with higher accuracy based on the corrected posture conversion relationship combined with the initial posture information of the device coordinate system relative to the reference coordinate system. Furthermore, based on this new target posture information, the motion capture device can be calibrated for the initial posture or real-time posture, thereby further improving the calibration accuracy of the motion capture device.

[0148] In some optional embodiments, based on the corrected posture conversion relationship and the initial posture information of the device coordinate system relative to the reference coordinate system, the new target posture information between the device and the world coordinate system is determined, which can be achieved by the following formula (2):

[0149] in, Represents the new target pose information of the i-th device coordinate system relative to the world coordinate system at any time after the t1-th time. G R ref Represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, Indicates the initial posture information of the i-th device coordinate system relative to the reference coordinate system at any time after the t1-th time, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, and R represents the posture information.

[0150] The calibration method of the motion capture device provided in the embodiment of the present application uses a camera to capture the spot image of the light-emitting unit on the motion capture device worn on the human limb, so as to determine the target posture information of the device coordinate system relative to the world coordinate system based on the spot image, and then calibrate the motion capture device based on the target posture information. The present application uses the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human posture restoration. In addition, by determining the initial posture information of the device coordinate system relative to the reference coordinate system, and correcting the posture conversion relationship between the world coordinate system and the reference coordinate system based on the target posture information and the initial posture information, and then based on the corrected posture conversion relationship between the world coordinate system and the reference coordinate system, when calibrating the posture of the motion capture device relative to the world, the calibration error can be further reduced and the calibration accuracy of the motion capture device can be improved.

[0151] As an optional implementation scenario, after calibrating the motion capture device, the present application may also determine the posture conversion relationship between the limb coordinate system and the world coordinate system, and then determine the posture of the limb corresponding to the motion capture device in the world coordinate system based on the posture conversion relationship. Below, in conjunction with Figure 8, the process of determining the posture conversion relationship between the limb coordinates and the world coordinate system, and determining the posture of the limb in the world coordinate system based on the posture conversion relationship, provided in an embodiment of the present application, is specifically described.

[0152] As shown in FIG8 , the method includes the following steps:

[0153] S401, determining posture information of a device coordinate system relative to a limb coordinate system.

[0154] In the embodiment of the present application, the limb coordinate system specifically refers to the limb coordinate system of the human limb, which can be expressed as B i, where B stands for Body, and i is the identification information of the limb corresponding to the limb coordinate system. The identification information can be any information that can uniquely determine the identity of the limb, such as the number or name of an important bone in the human body, etc. There is no specific restriction on it here.

[0155] As an optional implementation, the relationship between the device coordinate system, the world coordinate system, and the limb coordinate system can be exemplarily shown in FIG9. G Represents the world coordinate system, O B1 Represents the limb coordinate system of a thigh, O B2 Represents the limb coordinate system of a lower leg associated with the thigh, O L1 represents the device coordinate system of a motion capture device worn on the thigh, and O L2 Represents the device coordinate system of another motion capture device worn on the above calf.

[0156] Optionally, before determining the posture information of the device coordinate system relative to the limb coordinate system, the present application first guides the user to perform a preset calibration action (or a preset calibration posture) and maintain it for a certain time, so as to ensure that the user's face and the limb wearing the motion capture device have a certain relative posture through the preset calibration action. Among them, the preset calibration action can be selected as but not limited to: T-pose, N-pose, other actions that can ensure that the user's face, chest and lower limbs (i.e. legs) are facing the same direction, or other actions that ensure that the user's face and the limb wearing the motion capture device have a certain relative posture, etc. Afterwards, the posture information of the device coordinate system relative to the limb coordinate system is determined based on the preset action. In this application, a certain time can be flexibly set according to actual calculation requirements, such as 1 second, 2 seconds or 3 seconds, etc., and this application does not impose any restrictions on this.

[0157] It should be noted that the T-pose mentioned above can be understood as a posture with both arms stretched out horizontally to the sides in a T-shape. The N-pose mentioned above can be understood as a posture similar to standing at attention.

[0158] In some optional implementations, determining the posture information of the device coordinate system relative to the limb coordinate system may include the following steps:

[0159] Step 1: Under a preset calibration posture, determine the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, as well as the posture information of the head-mounted coordinate system relative to the world coordinate system, where the head-mounted coordinate system is the coordinate system of the head-mounted display device.

[0160] Among them, the preset calibration posture can be understood as the above-mentioned preset calibration actions, such as T-pose, N-pose, and other actions that can ensure that the user's face, chest and lower limbs (i.e., legs) are facing the same direction, or other actions that ensure that the user's face and the limbs wearing the motion capture device have a certain relative posture.

[0161] In some optional embodiments, after the user performs a preset calibration gesture, the positions of the human limbs and head remain fixed. Accordingly, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is also fixed. That is, at the moment the user performs the preset gesture, the posture transformation relationship between the limb coordinate system and the head-mounted coordinate system is a known quantity, and this known quantity is typically a constant, which is a unit matrix.

[0162] In some implementations, the camera on the head-mounted display device can collect surrounding environment data in real time, and the head-mounted display device can use a computer vision SLAM algorithm to determine the head-mounted display device's position information in real time based on the surrounding environment data collected by the camera. Therefore, when determining the pose information of the head-mounted coordinate system relative to the world coordinate system, the computer vision SLAM algorithm can be used to determine the pose information based on the surrounding environment data collected by the camera. The specific determination process is conventional technology in the field and will not be described in detail here.

[0163] Step 2: Determine the posture information of the device coordinate system relative to the limb coordinate system based on the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

[0164] Optionally, determining the posture information of the device coordinate system relative to the limb coordinate system can be achieved by the following formula (3):

[0165] in, Represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2, represents the posture conversion relationship between the i-th limb coordinate system and the head-mounted coordinate system at time t2, G R HMD Indicates the posture information of the head-mounted coordinate system relative to the world coordinate system at time t2, Indicates the posture information of the i-th device coordinate system relative to the world coordinate system at time t2, B i represents the coordinate system of the ith limb, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, HMD represents the head-mounted coordinate system, R represents the posture information, and -1 represents the inversion.

[0166] above in, Indicates the posture information of the i-th device coordinate system relative to the world coordinate system at time t2, G R ref Represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, Indicates the initial posture information of the device coordinate system relative to the reference coordinate system at time t2, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, and R represents the posture information.

[0167] It should be understood that the above t1 refers to the time of acquiring the motion capture device image or the spot image, or the time of acquiring the motion capture device image or the spot image and completing the calibration of the motion capture device, and t2 refers to the time of determining the posture of the device coordinate system relative to the limb coordinate system under the preset calibration posture, and t2 may be later than t1, that is, t1 <t2。

[0168] Taking into account that the posture conversion relationship between the corrected world coordinate system and the reference coordinate system has been determined in S304 in the aforementioned embodiment, the posture conversion relationship is fixed during calibration or after the last calibration and before the next calibration. In addition, the inertial measurement unit in the motion capture device will send inertial measurement unit data in real time, so the initial posture information of the device coordinate system relative to the reference coordinate system at time t2 can be calculated based on the received inertial measurement unit data. Furthermore, based on the initial posture information of the device coordinate system relative to the reference coordinate system at time t2 and the posture conversion relationship between the corrected world coordinate system and the reference coordinate system, the present application can determine the posture information of the i-th device coordinate system relative to the world coordinate system at time t2 when the user makes the preset calibration posture.

[0169] Among them, based on the initial posture information of the device coordinate system relative to the reference coordinate system at time t2, and the posture conversion relationship between the corrected world coordinate system and the reference coordinate system, the posture information of the device coordinate system relative to the world coordinate system is determined. For details, please refer to the S305 part of the aforementioned embodiment, which will not be elaborated here.

[0170] S402 : Determine a posture conversion relationship between the limb coordinate system and the world coordinate system based on posture information of the device coordinate system relative to the limb coordinate system and posture information of the device coordinate system relative to the world coordinate system.

[0171] In some optional implementations, the posture transformation relationship between the limb coordinate system and the world coordinate system can be determined by the following formula (4):

[0172] in, Indicates the posture transformation relationship between the limb coordinate system and the world coordinate system at time t2, Indicates the posture information of the i-th device relative to the world coordinate system at time t2, represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2, B i represents the coordinate system of the ith limb, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, R represents the posture information, and -1 represents inversion.

[0173] Furthermore, the limb coordinate system can be calibrated based on the posture conversion relationship between the limb coordinate system and the world coordinate system.

[0174] It should be understood that the posture conversion relationship between the above-mentioned limb coordinate system and the world coordinate system can be understood as the posture information of the limb coordinate system relative to the world coordinate system, and the posture information can represent the posture of the limb in the world coordinate system.

[0175] In some optional implementations, the present application determines the posture information of the device coordinate system relative to the limb coordinate system, and the execution order of calibrating the motion capture device can be to first perform the calibration operation on the motion capture device, and then perform the determination of the posture information of the device coordinate system relative to the limb coordinate system, and then obtain the posture information of the limb coordinate system relative to the world coordinate system, and realize the corresponding limb motion capture function; or, it can also first perform the determination of the posture information of the device coordinate system relative to the limb coordinate system, and then perform the calibration operation on the motion capture device, and then obtain the posture information of the limb coordinate system relative to the world coordinate system, and realize the corresponding limb motion capture function. The present application does not impose specific restrictions on this.

[0176] Among them, when first determining the posture information of the device coordinate system relative to the limb coordinate system and then performing the calibration operation on the motion capture device, this application first caches the posture information of the device coordinate system relative to the limb coordinate system at the time t2, and then performs the calibration operation on the motion capture device based on the cached posture information of the device coordinate system relative to the limb coordinate system at the time t2 when performing the calibration operation.

[0177] The above first determines the posture information of the device coordinate system relative to the limb coordinate system, which can be achieved by the following formula (5):

[0178] in, represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2, represents the posture conversion relationship between the i-th limb coordinate system and the head-mounted coordinate system at time t2. The posture conversion relationship is a fixed value when the user's limb makes a preset calibration posture, and the fixed value is usually a unit matrix. G R HMD represents the posture information of the head-mounted coordinate system relative to the world coordinate system at time t2, G R ref It represents the attitude transformation relationship between the world coordinate system and the reference coordinate system after correction at the t2th moment. Indicates the initial posture information of the i-th device coordinate system relative to the reference coordinate system at time t2, B i represents the coordinate system of the ith limb, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, HMD represents the head-mounted coordinate system, ref represents the reference coordinate system, R represents the posture information, and -1 represents the inversion.

[0179] It should be noted that in this application, the limbs can be regarded as fixed in the limb coordinate system, and the posture of the limb coordinate system relative to the world can be equivalent to the posture of the limb relative to the world coordinate system.

[0180] Furthermore, the motion capture device may be fixed to the motion capture device coordinate system, and the relationship between the motion capture device coordinate system and the reference coordinate system or the world coordinate system is equivalent to the relationship between the motion capture device and the reference coordinate system or the world coordinate system.

[0181] The calibration method of the motion capture device provided in the embodiment of the present application uses a camera to collect the spot image of the light-emitting unit on the motion capture device worn on the human limb, so as to determine the target posture information of the device coordinate system relative to the world coordinate system based on the spot image, and then calibrate the motion capture device based on the target posture information. The present application uses the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human posture restoration. In addition, the present application can also calibrate the limb coordinate system based on the posture conversion relationship between the limb coordinate system and the world coordinate system, and determine the posture of the limb in the world coordinate system based on the calibrated limb coordinate system, thereby improving the accuracy and correctness of determining the posture of the human limb in the world coordinate system and improving the capture effect of limb movement.

[0182] The following describes a calibration device for a motion capture device according to an embodiment of the present application with reference to FIG10 . FIG10 is a schematic block diagram of a calibration device for a motion capture device according to an embodiment of the present application.

[0183] The motion capture device is worn on a human limb and is configured on a head-mounted display device, which includes a camera. As shown in FIG10 , the motion capture device calibration device 500 includes: an image acquisition module 510 , a posture determination module 520 , and a device calibration module 530 .

[0184] The image acquisition module 510 is configured to acquire the image of the motion capture device through the camera;

[0185] a posture determination module 520 for determining target posture information of a device coordinate system relative to a world coordinate system based on the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system;

[0186] The device calibration module 530 is configured to calibrate the motion capture device according to the target posture information.

[0187] In an optional implementation of the embodiment of the present application, the posture determination module 520 is specifically configured to:

[0188] Extracting image feature points from the motion capture device image;

[0189] According to the image feature points, target posture information of the device coordinate system relative to the world coordinate system is determined.

[0190] In an optional implementation of the embodiment of the present application, the motion capture device includes a light-emitting unit and an image acquisition module 510, which is specifically configured to:

[0191] Collecting a light spot image of the light-emitting unit on the motion capture device by the camera;

[0192] Accordingly, the posture determination module 520 is specifically configured to:

[0193] According to the light spot image, target posture information of the device coordinate system relative to the world coordinate system is determined.

[0194] In an optional implementation of the embodiment of the present application, the posture determination module 520 is further configured to:

[0195] Determine the target pose information of the device coordinate system relative to the world coordinate system based on the light spot image using a pose estimation algorithm;

[0196] Posture information is extracted from the target pose information, and the posture information is determined as target posture information of the device coordinate system relative to the world coordinate system.

[0197] In an optional implementation of the embodiment of the present application, the apparatus 500 further includes:

[0198] A first determining module is configured to determine initial posture information of the device coordinate system relative to a reference coordinate system based on inertial measurement unit data sent by the motion capture device;

[0199] Accordingly, the device calibration module 530 is specifically configured to:

[0200] Correcting the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information;

[0201] The motion capture device is calibrated according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0202] In an optional implementation of the embodiment of the present application, the device calibration module 530 is further configured to:

[0203] Correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system.

[0204] In an optional implementation of the embodiment of the present application, the inertial measurement unit data includes: accelerometer data, gyroscope data, and magnetometer data, and the first determination module is specifically configured to:

[0205] The complementary filtering algorithm is used to determine initial posture information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, the gyroscope data, and the magnetometer data sent by the motion capture device.

[0206] In an optional implementation of the embodiment of the present application, the apparatus 500 further includes:

[0207] A second determining module is used to determine the posture information of the device coordinate system relative to the limb coordinate system;

[0208] The third determining module is used to determine the posture conversion relationship between the limb coordinate system and the world coordinate system according to the posture information of the device coordinate system relative to the limb coordinate system and the posture information of the device coordinate system relative to the world coordinate system.

[0209] In an optional implementation of the embodiment of the present application, the second determining module is specifically configured to:

[0210] Under a preset calibration posture, determining a posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, as well as posture information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is a head-mounted display device coordinate system;

[0211] The posture information of the device coordinate system relative to the limb coordinate system is determined based on the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

[0212] In an optional implementation of the embodiment of the present application, the second determining module is further configured to:

[0213] The posture information of the device coordinate system relative to the world coordinate system is determined according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0214] In an optional implementation of the embodiment of the present application, the number of light spots on the motion capture device is at least three.

[0215] An optional implementation of the embodiment of the present application further includes:

[0216] The display module is used to display an image perspective screen and display guidance information in the image perspective screen for guiding the user to place the motion capture device in the image perspective screen.

[0217] The motion capture device calibration device provided in an embodiment of the present application uses a camera to capture an image of the motion capture device worn on a human limb, determines target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device image, and then calibrates the motion capture device based on the target pose information. By utilizing the motion capture device image to calibrate the motion capture device, the present application can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, reduce calibration errors, and improve the calibration accuracy of the motion capture device. As a result, the calibrated motion capture device can improve the effect of capturing limb motion, providing conditions for improving the accuracy of human posture restoration.

[0218] It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device 500 shown in Figure 10 can execute the method embodiment corresponding to Figure 3, and the aforementioned and other operations and / or functions of each module in the device 500 are respectively for implementing the corresponding processes in each method in Figure 3. For the sake of brevity, no further description is given here.

[0219] The above describes the device 500 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the first aspect method disclosed in conjunction with the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned first aspect method embodiment in conjunction with its hardware.

[0220] FIG11 is a schematic block diagram of an electronic device provided in an embodiment of the present application. As shown in FIG11 , the electronic device 600 may include:

[0221] The memory 610 and the processor 620 are configured to store a computer program and transmit the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the motion capture device calibration method in the embodiment of the present application.

[0222] For example, the processor 620 may be configured to execute the above-mentioned motion capture device calibration method embodiment according to the instructions in the computer program.

[0223] In some embodiments of the present application, the processor 620 may include but is not limited to:

[0224] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0225] In some embodiments of the present application, the memory 610 includes but is not limited to:

[0226] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0227] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to implement the motion capture device calibration method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0228] As shown in FIG11 , the electronic device 600 may further include:

[0229] The transceiver 630 may be connected to the processor 620 or the memory 610.

[0230] The processor 620 may control the transceiver 630 to communicate with other devices. Specifically, the processor 620 may send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include one or more antennas.

[0231] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0232] The present application also provides a computer storage medium having a computer program stored thereon, which enables the computer to perform the method of the above method embodiment when the computer program is executed by the computer.

[0233] An embodiment of the present application further provides a computer program product comprising program instructions, which, when executed on an electronic device, enables the electronic device to execute the method of the above method embodiment.

[0234] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0235] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0237] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0238] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for calibrating a motion capture device, comprising: The motion capture device image is collected by a camera included in a head-mounted display device, the motion capture device is worn on a limb of a human body, and the method is applied to the head-mounted display device; Determine, based on the motion capture device image, target posture information of a device coordinate system relative to a world coordinate system, the device coordinate system being the motion capture device coordinate system; The motion capture device is calibrated according to the target posture information.

2. The method according to claim 1, wherein determining the target posture information of the device coordinate system relative to the world coordinate system according to the motion capture device image comprises: Extracting image feature points from the motion capture device image; According to the image feature points, target posture information of the device coordinate system relative to the world coordinate system is determined.

3. The method according to claim 1, wherein the motion capture device comprises a light emitting unit, and collecting the motion capture device image through the camera comprises: Collecting a light spot image of the light-emitting unit on the motion capture device by using the camera; and Wherein, determining the target posture information of the device coordinate system relative to the world coordinate system according to the motion capture device image comprises: According to the light spot image, target posture information of the device coordinate system relative to the world coordinate system is determined.

4. The method according to claim 3, wherein determining the target posture information of the device coordinate system relative to the world coordinate system according to the spot image comprises: Determine the target pose information of the device coordinate system relative to the world coordinate system according to the light spot image by using a pose estimation algorithm; Posture information is extracted from the target posture information, and the posture information is determined as target posture information of the device coordinate system relative to the world coordinate system.

5. The method according to claim 3, further comprising: Determining initial posture information of the device coordinate system relative to a reference coordinate system based on inertial measurement unit data sent by the motion capture device; and The calibrating of the motion capture device comprises: Correcting the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information; as well as The motion capture device is calibrated according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

6. The method according to claim 5, wherein the correcting the posture conversion relationship between the world coordinate system and the reference coordinate system comprises: Correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system.

7. The method of claim 5, wherein the inertial measurement unit data comprises: accelerometer data, gyroscope data, and magnetometer data, and The step of determining the initial posture information of the device coordinate system relative to the reference coordinate system according to the inertial measurement unit data sent by the motion capture device includes: The complementary filtering algorithm is used to determine the initial posture information of the device coordinate system relative to the reference coordinate system according to the accelerometer data, the gyroscope data and the magnetometer data sent by the motion capture device.

8. The method according to any one of claims 1 to 7, further comprising: Determining posture information of the device coordinate system relative to the limb coordinate system; A posture conversion relationship between the limb coordinate system and the world coordinate system is determined according to the posture information of the device coordinate system relative to the limb coordinate system and the posture information of the device coordinate system relative to the world coordinate system.

9. The method according to claim 8, wherein determining the posture information of the device coordinate system relative to the limb coordinate system comprises: Under a preset calibration posture, determining a posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, and posture information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is a head-mounted display device coordinate system; The posture information of the device coordinate system relative to the limb coordinate system is determined according to the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

10. The method according to claim 9, wherein determining the posture information of the device coordinate system relative to the world coordinate system comprises: The posture information of the device coordinate system relative to the world coordinate system is determined according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system. The method according to claim 3 , wherein the number of light spots on the motion capture device is at least three.

12. The method according to claim 1, further comprising: An image perspective screen is displayed, and guiding information for guiding a user to place the motion capture device in the image perspective screen is displayed in the image perspective screen.

13. A calibration device for a motion capture device, comprising: An image acquisition module, used to acquire the image of the motion capture device through a camera included in a head-mounted display device, wherein the motion capture device is worn on a limb of a human body, and the device is applied to the head-mounted display device; A posture determination module, used to determine target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system; The device calibration module is used to calibrate the motion capture device according to the target posture information.

14. An electronic device comprising: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the motion capture device calibration method as described in any one of claims 1 to 12. 15 . A computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the motion capture device calibration method according to claim 1 . 16 . A computer program product comprising program instructions, wherein when the program instructions are executed on an electronic device, the electronic device executes the motion capture device calibration method according to claim 1 .

Images