Calibration method and device of motion capture equipment, equipment and medium

Abstract

The invention provides a calibration method and device of motion capture equipment, equipment and a medium, the motion capture equipment is worn on a limb of a human body, the method is applied to head-mounted display equipment, the head-mounted display equipment comprises a camera, and the method comprises the following steps: collecting an image of the motion capture equipment through the camera; according to the motion capture equipment image, determining target attitude information of an equipment coordinate system relative to a world coordinate system; and calibrating the motion capture equipment according to the target attitude information. The motion capture equipment is calibrated by using the image of the motion capture equipment, so that the calibration difficulty of the motion capture equipment can be effectively reduced, the calibration time can be shortened, the calibration error can be reduced, and the calibration precision of the motion capture equipment can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of motion capture technology, and in particular to a calibration method, apparatus, device and medium for a motion capture device. Background Art

[0002] Human motion capture technology has great application prospects in the fields of virtual reality (VR), human-computer interaction, etc. At present, human motion capture solutions mainly use motion capture devices (such as inertial sensors or tracking devices with inertial sensors) to capture motion and collect human motion data in the real world.

[0003] Before using a motion capture device to collect human motion data, the motion capture device needs to be calibrated to ensure the accuracy and precision of the motion data collected by the motion capture device. In the related art, when calibrating a motion capture device, it is necessary to use a reference object and fix the wearing position of the motion capture device. This method will undoubtedly increase the difficulty of operation for users, bring trouble to users, and even introduce calibration errors due to inaccurate wearing position of the motion capture device, resulting in low calibration accuracy and poor effect of the motion capture device. Summary of the invention

[0004] The present application provides a motion capture device calibration method, apparatus, equipment and medium. By utilizing the motion capture device image to calibrate the motion capture device, the difficulty of calibrating the motion capture device can be effectively reduced, the calibration time can be shortened, and the calibration error can be reduced, thereby improving the calibration accuracy of the motion capture device.

[0005] In a first aspect, an embodiment of the present application provides a calibration method for a motion capture device, wherein the motion capture device is worn on a limb of a human body, and the method is applied to a head-mounted display device, wherein the head-mounted display device includes a camera, and the method includes:

[0006] Collecting the motion capture device image through the camera;

[0007] Determine, based on the motion capture device image, target posture information of a device coordinate system relative to a world coordinate system, the device coordinate system being the motion capture device coordinate system;

[0008] The motion capture device is calibrated according to the target posture information.

[0009] In a second aspect, an embodiment of the present application provides a calibration device for a motion capture device, wherein the motion capture device is worn on a limb of a human body, and the device is configured on a head-mounted display device, wherein the head-mounted display device includes a camera, including:

[0010] An image acquisition module, used for acquiring the image of the motion capture device through the camera;

[0011] A posture determination module, used to determine target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system;

[0012] The device calibration module is used to calibrate the motion capture device according to the target posture information.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0014] A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the motion capture device calibration method described in the first aspect embodiment or its various implementation methods.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the motion capture device calibration method as described in the embodiment of the first aspect or its various implementations.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product comprising program instructions, which, when executed on an electronic device, enables the electronic device to execute a motion capture device calibration method as described in the embodiment of the first aspect or its various implementations.

[0017] The technical solution disclosed in the embodiments of the present application has at least the following beneficial effects:

[0018] The image of the motion capture device worn on the human body is captured by a camera to determine the target posture information of the device coordinate system relative to the world coordinate system based on the motion capture device image, and then the motion capture device is calibrated based on the target posture information. This application uses the motion capture device image to calibrate the motion capture device, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human posture restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of wearing a head-mounted display device, a motion capture device and a peripheral device on different parts of a human body provided by an embodiment of the present application;

[0022] Figure 3 It is a flow chart of a calibration method of a motion capture device provided in an embodiment of the present application;

[0023] Figure 4 is a flow chart of another method for calibrating a motion capture device provided in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the relationship between the world coordinate system and the device coordinate system provided in an embodiment of the present application;

[0025] Figure 6 is a flow chart of another method for calibrating a motion capture device provided in an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of the relationship between the world coordinate system, the reference coordinate system, and the device coordinate system provided in an embodiment of the present application;

[0027] Figure 8 It is a flowchart of another method for calibrating a motion capture device provided in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of the relationship between the device coordinate system, the world coordinate system and the limb coordinate system provided in the embodiment of the present application;

[0029] Fig.10 is a schematic block diagram of a calibration device for a motion capture device provided in an embodiment of the present application;

[0030] Fig.11 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The present application is applicable to the human motion capture scene. At present, the human motion capture scheme mainly uses motion capture equipment to capture motion to collect human motion data in the real world. The motion capture equipment is basically an inertial sensor or a tracking device with an inertial sensor. In order to ensure the accuracy and precision of the motion data collected by the motion capture equipment, the motion capture equipment needs to be calibrated. Considering that the relevant technology requires the use of reference objects and the wearing position of the motion capture equipment to be fixed, it will increase the difficulty of operation for the user and cause trouble to the user. It may even introduce calibration errors due to the inaccurate wearing position of the motion capture equipment, resulting in the problem of low calibration accuracy and poor effect of the motion capture equipment. Therefore, the present application designs a method for calibrating a motion capture equipment, so that the motion capture equipment can be calibrated by this method, which can effectively reduce the difficulty of calibrating the motion capture equipment, shorten the calibration time, and also reduce the calibration error and improve the calibration accuracy of the motion capture equipment.

[0034] In order to facilitate understanding of the embodiments of the present application, before describing each embodiment of the present application, some concepts involved in all embodiments of the present application are first appropriately explained, as follows:

[0035] 1) Virtual Reality (VR for short), a technology for creating and experiencing a virtual world, determines the generation of a virtual environment, is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.), realizes the simulation of the fusion and interactive three-dimensional dynamic vision and entity behavior of the virtual environment, immerses the user in a simulated virtual reality environment, and realizes the application in various virtual environments such as maps, games, videos, education, medical treatment, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0036] 2) Virtual reality devices (VR devices), terminals for realizing virtual reality effects, can usually be provided in the form of glasses, head mount displays (Head Mount Display, referred to as HMD), contact lenses, for realizing visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged according to actual needs.

[0037] Optionally, the virtual reality devices described in the embodiments of the present application may include but are not limited to the following types:

[0038] 2.1) PC-based virtual reality (PCVR) devices use the PC to perform relevant calculations and data output for virtual reality functions. External PC-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

[0039] 2.2) Mobile virtual reality devices support the configuration of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0040] 2.3) All-in-one virtual reality devices have a processor for performing relevant calculations of virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom in use.

[0041] 3) Virtual field of view: the area in the virtual environment that the user can perceive through the lens in the virtual reality device. The field of view (FOV) of the virtual field of view is used to represent the perceived area.

[0042] 4) Augmented Reality (AR): A technology that calculates the camera's posture parameters in the real world (or three-dimensional world, real world) in real time during the process of the camera capturing images, and adds virtual elements to the images captured by the camera based on the camera posture parameters. Virtual elements include but are not limited to: images, videos, and three-dimensional models. The goal of AR technology is to integrate the virtual world with the real world on the screen for interaction.

[0043] 5) Mixed Reality (MR): A simulated setting that integrates computer-created sensory input (e.g., virtual objects) with sensory input from a physical setting or its representation. In some MR settings, computer-created sensory input can adapt to changes in sensory input from the physical setting. In addition, some electronic systems used to present MR settings can monitor the orientation and / or position relative to the physical setting so that virtual objects can interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0044] 6) Extended Reality (XR) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices, which include virtual reality (VR), augmented reality (AR) and mixed reality (MR) and other forms.

[0045] 7) A virtual scene is a virtual scene displayed (or provided) when an application is running on an electronic device. The virtual scene can be a simulation of the real world, a semi-simulated and semi-fictitious virtual scene, or a purely fictitious virtual scene. The virtual scene can be any of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene. The embodiments of the present application do not limit the dimensions of the virtual scene. For example, a virtual scene can include the sky, land, ocean, etc. The land can include environmental elements such as deserts and cities, and users can control virtual objects to move in the virtual scene.

[0046] 8) Virtual objects are objects that interact in virtual scenes, are controlled by users or robot programs (for example, robot programs based on artificial intelligence), and can be still, move, and perform various behaviors in virtual scenes, such as various characters in games.

[0047] In order to clearly explain the technical solution of the present application, the application scenarios of the technical solution of the present application are described below. It should be understood that the technical solution of the present application can be applied to the following scenarios, but is not limited thereto:

[0048] For example, Figure 1This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the application scenario 1000 may include: a head mounted display device 100 and a motion capture device 200. Furthermore, the head mounted display device 100 and the motion capture device 200 may communicate with each other.

[0049] In some possible implementations, the head-mounted display device 100 may be an HMD, such as a head-mounted display in a VR all-in-one machine, etc. This embodiment does not impose any restrictions on this.

[0050] In addition, the head mounted display device 100 is provided with a camera to collect the surrounding environment through the camera, and based on the collected surrounding environment data, a simultaneous localization and mapping (SLAM) algorithm in computer vision is used for tracking and positioning. The number of the above-mentioned cameras can be at least one, Figure 1 In the example, the number of cameras is 4. In addition, the type of the above-mentioned camera can be a fisheye camera, a common camera, or other types of cameras, and this application does not impose any restrictions on this.

[0051] In some implementations, the motion capture device 200 may be an inertial sensor, or any device having an inertial sensor, etc.

[0052] In some implementations, the motion capture device 200 can be worn on human limbs, such as limbs and torso. The limbs include upper limbs and lower limbs. The torso includes the waist, etc. This application does not impose any restrictions on human limbs. For example, Figure 2 As shown, the motion capture device 200 is worn on the lower limbs of the human body. The lower limbs of the human body include: thighs and calves. That is to say, by wearing four motion capture devices 200 on the thighs and calves of the human body respectively, the motion capture devices 200 can be used to collect the thigh motion data and calf motion data of the human body (collectively referred to as lower limb motion data), and the lower limb motion data is sent to the head mounted display device 100, so as to achieve the purpose of tracking the lower limb motion of the human body.

[0053] It should be noted that the limb data collected by the motion capture device 200 worn on the limbs of the human body may be 3-degree of freedom (Dof) data or 6-DoF data.

[0054] Considering that the upper limbs of the human body can also make different movements, in addition to wearing the motion capture device 200 on the upper limbs of the human body, the present application can also optionally wear the external device 300 on the upper limbs of the human body. For example, it can be worn on the hands and / or arms of the human body. Then, the upper limb motion data of the human body is collected by the external device 300, and the upper limb motion data is sent to the head-mounted display device 100, thereby realizing the tracking of the upper limb motion of the human body. For details on how to wear the external device 300, please refer to Figure 2 .

[0055] In some implementations, the peripheral device 300 may be, but is not limited to, a handle, a glove, a bracelet, a wristband, a finger ring, and other wearable devices.

[0056] In addition, the peripheral device 300 is provided with an inertial sensor, and the inertial sensor can provide 6-DOF data including the position and posture of the upper limbs of the human body.

[0057] It should be noted that the inertial sensor involved in this embodiment specifically refers to an inertial measurement unit (IMU). Moreover, the IMU may be a six-axis IMU or a nine-axis IMU, which is not specifically limited here.

[0058] It can be understood that the embodiments of the present application can achieve full-body tracking of the human body by wearing motion capture devices 200 on the human body's limbs, or wearing external devices 300 on the human body's upper limbs and wearing motion capture devices on the human body's lower limbs.

[0059] It should be understood that Figure 1 and Figure 2 The head mounted display device 100, motion capture device 200 and peripheral device 300 shown are for illustration only and are not intended to be specific limitations to the present application.

[0060] After introducing the application scenarios of the embodiments of the present application, a calibration method, apparatus, device and medium for a motion capture device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0061] Figure 3 1 is a flow chart of a motion capture device calibration method provided in an embodiment of the present application. The embodiment of the present application can be applied to the calibration scenario of the motion capture device in human motion capture, and the motion capture device calibration method can be performed by a motion capture device calibration device. The motion capture device calibration device can be composed of hardware and / or software, and can be integrated into a head-mounted display device.

[0062] In the embodiment of the present application, the head mounted display device may be any electronic device that can reproduce human body movements. Optionally, the electronic device may be an extended reality (XR) device. The XR device may be a VR device, an augmented reality (AR) device, or a mixed reality (MR) device, etc. The present application does not specifically limit the type of electronic device.

[0063] like Figure 3 As shown, the method may include the following steps:

[0064] S101, collecting motion capture device images through a camera.

[0065] Considering that the head mounted display device may be equipped with multiple cameras, for example Figure 1 Therefore, when there are multiple cameras on the head-mounted display device, the present application preferentially uses the downward-looking camera located on the head-mounted display device to collect the motion capture device image, thereby making it easier to collect the motion capture device image worn on the human limb by using the camera close to the side of the human limb.

[0066] Optionally, before executing S101, the user first wears the head-mounted display device and the motion capture device on different parts of the human body, such as wearing the head-mounted display device on the human head, wearing the motion capture device on the human lower limbs, and wearing the external device on the human upper limbs. For details, please refer to the aforementioned Figure 2 For another example, a head-mounted display device is worn on a human head, and a motion capture device is worn on a human upper limb and a human lower limb. For another example, a head-mounted display device is worn on a human head, and a motion capture device is worn on a human upper limb, a human waist, a human lower limb, and the like.

[0067] After wearing the head-mounted display device, motion capture device and external devices, the user can start the head-mounted display device, motion capture device and external devices to establish a communication connection between the head-mounted display device and each motion capture device, as well as a communication connection between the head-mounted display device and external devices, laying the foundation for subsequent data communication.

[0068] Among them, the specific implementation methods of establishing the communication connection between the head-mounted display device and each motion capture device, and the communication connection between the head-mounted display device and the external device can refer to the existing establishment method, and this application will not go into details about this.

[0069] Furthermore, the head-mounted display device may capture images of motion capture devices worn on human limbs through a camera. Considering that the captured motion capture device images include not only the motion capture devices but also the human limbs wearing the motion capture devices. Therefore, in the present application, a frame of motion capture device images captured by the camera on the head-mounted display device may include at least one motion capture device. Thus, the head-mounted display device may distinguish each motion capture device in the motion capture device image based on the motion capture devices in the motion capture device image and the human limbs to which each motion capture device belongs. Furthermore, based on each motion capture device in the motion capture device image, the motion capture device worn on the human limbs may be calibrated.

[0070] In some optional embodiments, when the present application collects the motion capture device image through the camera on the head mounted display device, it is optional to collect only one motion capture device image in a time period, and calibrate the motion capture device based on the motion capture device image. This can reduce the image processing complexity of the head mounted display device and improve the calibration efficiency for the motion capture device.

[0071] Exemplarily, in the first time period, a motion capture device image is collected on the right calf. In the second time period, a motion capture device image is collected on the right thigh. In the third time period, a motion capture device image is collected on the left calf. In the fourth time period, a motion capture device image is collected on the left thigh. The order of the first time period, the second time period, the third time period, and the fourth time period can be the first time period → the second time period → the third time period → the fourth time period, which can be flexibly adjusted according to actual needs, and this application does not impose any restrictions on this.

[0072] In some optional embodiments, considering that the head-mounted display device can support the video see-through (VST) function, based on the VST function, the user can see the virtualized real space and real objects in proportion to the real world through the head-mounted display device. Among them, the virtualized real objects include but are not limited to tables, chairs, etc. Therefore, before the present application collects the image of the motion capture device through the camera, the optional head-mounted display device can display the video perspective screen in proportion to the real world through the VST function. At the same time, in any area of ​​the video perspective screen, the guidance information guiding the user to place any motion capture device in the video perspective screen is displayed, so that the user can place the above motion capture device in the video perspective screen based on the guidance information. Thus, when the user places any motion capture device in the video perspective screen according to the guidance information, the camera on the head-mounted display device can collect the above motion capture device image. The advantage of this setting is that it uses the VST function to output guidance information to users, so as to instruct users to make corresponding actions according to the guidance information, and through the image perspective function, users can see their own actions, so that users can visualize their actions, and the interaction between head-mounted display devices is more intuitive, which improves the ease of motion capture device calibration.

[0073] It should be noted that, considering that the calibration principle and calibration process of each motion capture device worn on the human limbs are the same, in order to simplify the description, the present application will describe in detail the technical solution of the present application by taking the calibration of one of the motion capture devices as an example in the following embodiments.

[0074] S102, determining target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image, where the device coordinate system is a motion capture device coordinate system.

[0075] Among them, the device coordinate system can be obtained through L i Indicates that L stands for Local, i is the identification information of the motion capture device, and the identification information can be any information that can uniquely determine the identity of the motion capture device, such as a number or a name, etc., and there is no specific limitation on it here.

[0076] The world coordinate system refers to the global world coordinate system, which can be represented by G, and the full name of G is Global.

[0077] In some possible implementations, the relationship between the world coordinate system and the device coordinate system can be as follows: Figure 4 As shown. Among them, O G is the world coordinate system, O L1 is the coordinate system of a motion capture device worn on a thigh of the human lower limb, and O L2Represents the coordinate system of another motion capture device worn on the calf associated with the above thigh.

[0078] Optionally, after acquiring the motion capture device image captured by the camera, the head mounted display device may extract image feature points from the motion capture device image, and then determine the target posture information of the device coordinate system relative to the world coordinate system based on the extracted image feature points.

[0079] Considering that the motion capture device image captured by the camera may include at least one motion capture device, the head mounted display device may extract image feature points from the motion capture device image in different ways according to the number of motion capture devices in the motion capture device image, as follows:

[0080] In the first manner, when there is only one motion capture device in the motion capture device image, the head mounted display device extracts image feature points from the motion capture device image.

[0081] In a second manner, when the motion capture device image includes at least two motion capture devices, the head mounted display device extracts image feature points of each motion capture device from the motion capture device image.

[0082] In the present application, the above-mentioned image feature points can be relatively prominent points or representative points in the motion capture device image, such as corner points, contour points, dark points in brighter areas, and bright points in darker areas of the motion capture device in the motion capture device image. This application does not impose any restrictions on image feature points.

[0083] Furthermore, the image feature points extracted from the motion capture device image are 2-dimensional points (ie, 2D points).

[0084] As an optional implementation, the present application may use a feature point extraction algorithm to extract image feature points from the motion capture device image. Exemplary feature point extraction algorithms may include but are not limited to: SIFT (Scale Invariant Feature Transform) extraction algorithm, SURF (Speeded-Up Robust Features) extraction algorithm, and ORB (Oriented FAST and Rotated BRIEF) extraction algorithm, etc.

[0085] After extracting image feature points from the motion capture device image, the head mounted display device can determine the target posture information of the device coordinate system corresponding to the motion capture device relative to the world coordinate system based on the image feature points extracted from the motion capture device image on the motion capture device. That is, determine the posture conversion relationship between the device coordinate system and the world coordinate system.

[0086] Considering that the model, specification and shape of the motion capture device worn on the human body are known. Therefore, it can be determined that the 3D model (i.e., 3D model) of the motion capture device is also known. Therefore, the present application is based on extracting the image feature points of any motion capture device from the motion capture device image, and the 3D model of the motion capture device, and can determine the 3D points corresponding to the 2D image feature points of the motion capture device. Then, using the preset calculation rules, the posture conversion relationship between the device coordinate system and the world coordinate system is calculated according to the 2D points and 3D points of the motion capture device.

[0087] The above-mentioned preset calculation rule can be selected as any algorithm or strategy that can calculate the posture conversion relationship between the device coordinate system and the world coordinate system based on 2D points and 3D points, and this application does not impose any restrictions on this.

[0088] In the present application, the above-mentioned 2D point can be understood as the projection of the 3D point on the imaging plane, and the 3D point can be understood as the 3D coordinate point in the world coordinate system.

[0089] Because the pose information of the head-mounted coordinate system relative to the world coordinate system is known, when the above-mentioned preset calculation rule is the perspective-n-point (PnP) algorithm in computer vision, the head-mounted display device can use the above-mentioned PNP algorithm to calculate the target pose information of the device coordinate system relative to the world coordinate system based on n points (2D and 3D points of the motion capture device in the motion capture device image). Then, the target pose information of the device coordinate system relative to the world coordinate system is extracted from the target pose information.

[0090] Generally, the PnP algorithm may include: a P3P algorithm, a P5P algorithm and other algorithms. Moreover, the above-mentioned P3P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on three points. The above-mentioned P5P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on five points. Therefore, in the present application, the number of motion capture device feature points extracted from the motion capture device image is at least three, and correspondingly, at least three 3D points need to be determined based on the above-mentioned at least three 2D points. Therefore, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the motion capture device image captured by the camera, different PnP algorithms can be selected based on the number of image feature points of the motion capture device in the motion capture device image.

[0091] S103: calibrate the motion capture device according to the target posture information.

[0092] After determining the target posture information of the device coordinate system relative to the world coordinate system, the present application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the subsequent body movements captured by the calibrated motion capture device are more accurate.

[0093] In some optional embodiments, the posture of the motion capture device relative to the world is calibrated. Optionally, the initial posture of the motion capture device relative to the world is calibrated, or the real-time posture of the motion capture device relative to the world is calibrated. This application does not impose any restrictions on this.

[0094] The motion capture device calibration method provided in the embodiment of the present application uses a camera to collect the image of the motion capture device worn on the human body to determine the target posture information of the device coordinate system relative to the world coordinate system based on the motion capture device image, and then calibrates the motion capture device based on the target posture information. By using the motion capture device image to calibrate the motion capture device, the present application can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human body posture restoration.

[0095] In another optional implementation scenario, it is considered that the motion capture device may include a light-emitting unit. That is, the motion capture device may be an inertial sensor with a light-emitting unit, or any device with an inertial sensor and a light-emitting unit, such as an optical tracker. Therefore, the present application optionally collects a light spot image of the light-emitting unit on the motion capture device image through a camera to calibrate the motion capture device according to the light spot image.

[0096] It should be noted that the light-emitting unit on the above-mentioned motion capture device can be selected as an LED or other light-emitting device, and the light emitted can be visible light or invisible light, as long as it can be captured by the camera. In addition, the above-mentioned light-emitting unit can be distributed in the shell of the motion capture device body or in accessories rigidly connected to the motion capture device body, such as a base, according to a preset deployment method. In the present application, the above-mentioned preset deployment method can be flexibly set according to the shape of the motion capture device body or the accessories rigidly connected to the motion capture device body, and no restrictions are imposed on it here.

[0097] Furthermore, when the motion capture device worn on the limbs of the human body includes a light-emitting unit, the limb data collected by the motion capture device includes two categories:

[0098] The first category is that when the light-emitting unit on the motion capture device is in the lighted state and is observed by the camera on the head-mounted display device, the limb movement data collected by the motion capture device is 6-degree-of-freedom (Dof) data including the position and posture of the human limbs.

[0099] The second category is that when the light-emitting unit on the motion capture device is in the lighted state and cannot be observed by the camera on the head-mounted display device, the limb movement data collected by the motion capture device is 3-degree-of-freedom (Dof) data including the human limb posture.

[0100] As mentioned above Figure 2 In the example shown, the lower limb motion data collected by the motion capture device 200 includes two categories: the first category, when the light-emitting unit on the motion capture device 200 is in a lit state and is observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 at this time is 6 degrees of freedom (Dof) data including the position and posture of the lower limbs of the human body. The second category, when the light-emitting unit on the motion capture device 200 is in a lit state and cannot be observed by the camera on the head-mounted display device 100, the lower limb motion data collected by the motion capture device 200 at this time is 3 degrees of freedom (Dof) data including the posture of the lower limbs of the human body.

[0101] Combine the following Figure 5 , the process of calibrating the motion capture device based on the spot image described above is specifically described. Figure 5 As shown, the method comprises the following steps:

[0102] S201, collecting a light spot image of a light-emitting unit on a motion capture device through a camera.

[0103] Considering that each motion capture device worn on a human limb is provided with a light-emitting unit, if the light-emitting units on each motion capture device are lit at the same time, and the camera on the head-mounted display device collects the light spot images of the light-emitting units on all motion capture devices at the same time, the head-mounted display device may not be able to accurately distinguish which motion capture device each light spot corresponds to based on the light spot images collected by the camera, thereby causing confusion in the calibration of the motion capture device, or even calibration errors.

[0104] Based on the above reasons, when the present application calibrates all motion capture devices, the light-emitting units on the motion capture devices are lit up in time-sharing manner, that is, each motion capture device is lit up in turn in a certain order in each cycle, so that the camera on the head-mounted display device only collects the light spot image of the light-emitting unit on one motion capture device in a time period, and uses the light spot image to calibrate the motion capture device, thereby ensuring that the calibration of each motion capture device is more accurate.

[0105] For example, in time period 1, the light-emitting unit in the motion capture device worn on the left thigh is lit, and the light-emitting units in the motion capture devices worn on the right thigh, right calf and left calf are controlled to be in an off state. In time period 2, the light-emitting unit in the motion capture device worn on the left calf is lit, and the light-emitting units in the motion capture devices worn on the right thigh, right calf and left thigh are controlled to be in an off state. In time period 3, the light-emitting unit in the motion capture device worn on the right thigh is lit, and the light-emitting units in the motion capture devices worn on the right calf, left thigh and left calf are controlled to be in an off state. In time period 4, the light-emitting unit in the motion capture device worn on the right calf is lit, and the light-emitting units in the motion capture devices worn on the left thigh, left calf and right thigh are controlled to be in an off state. Among them, the order of time period 1, time period 2, time period 3 and time period 4 can be selected as time period 1→time period 2→time period 3→time period 4, of course, it can also be other orders, and there is no specific limitation on it here.

[0106] The above-mentioned lighting up of the light-emitting unit on the motion capture device can be triggered manually by the user, or the head-mounted display device can light up the light-emitting unit on the motion capture device by sending a lighting instruction to the motion capture device. This application does not impose any restrictions on this.

[0107] In some optional implementations, when a camera is used to capture a light spot image of a light-emitting unit on a motion capture device in a lit state, if the user's top is loose or other conditions occur, the camera on the head-mounted display device may not be able to capture the light spot image of the light-emitting unit on the motion capture device in a lit state.

[0108] To this end, after lighting up the light-emitting unit on any motion capture device, the present application guides the user to lower his head, kick his legs and / or raise his arms, so that the camera on the head-mounted display device can capture the light spot image of the light-emitting unit on the motion capture device. Among them, the lower limb that kicks or the lower limb that raises the arm should be the one wearing the motion capture device in the lit state. Exemplarily, assuming that the light-emitting unit in the motion capture device on the left calf is in the lit state, then guide the user's left calf to kick. For another example, assuming that the light-emitting unit in the motion capture device on the right forearm is in the lit state, then guide the user to raise his right forearm.

[0109] It can be understood that the above-mentioned head lowering action, leg kicking action and / or arm raising action can be a head lowering action; or, it can also be a leg kicking action; or, it can also be an arm raising action; or, it can also be a head lowering action and an arm raising action; or, it can also be a head lowering action and a leg kicking action, etc., and there is no limitation on it here.

[0110] In addition, in the present application, when guiding the user to lower the head, kick the legs and / or raise the arms, guiding voice, guiding text and / or guiding animation can be output to the user, and there is no limitation on this here.

[0111] In some optional embodiments, considering that the head-mounted display device can support the video see-through (VST) function, based on the VST function, the user can see the virtualized real space and real objects in proportion to the real world through the head-mounted display device. Among them, the virtualized real objects include but are not limited to tables, chairs, etc. Therefore, before the present application collects the spot image of the light-emitting unit on the motion capture device through the camera, the optional head-mounted display device can display the video perspective screen in proportion to the real world through the VST function. At the same time, in any area of ​​the video perspective screen, the guidance information guiding the user to place any motion capture device with the light-emitting unit lit in the video perspective screen is displayed, so that the user can place the motion capture device with the light-emitting unit in the light-emitting state in the video perspective screen based on the guidance information. Thus, when the user places any motion capture device with the light-emitting unit lit in the video perspective screen according to the guidance information, the camera on the head-mounted display device can collect the spot image of the light-emitting unit on the motion capture device. The advantage of this setting is that it uses the VST function to output guidance information to the user, so as to guide the user to make corresponding actions according to the guidance information, and the user can see his or her own actions through the image perspective function, so that the user's head-lowering, kicking and / or arm-raising actions are visualized, and the interaction with the head-mounted display device is more intuitive, thereby improving the ease of calibration of the motion capture device.

[0112] S202: Determine target posture information of the device coordinate system relative to the world coordinate system according to the light spot image.

[0113] Optionally, the head-mounted display device of the present application may adopt a preset calculation rule to calculate the target posture information of the device coordinate system relative to the world coordinate system based on the spot image.

[0114] Considering that the pose information of the head-mounted coordinate system relative to the world coordinate system is known, the present application can use the pose estimation (Perspective-n-Point, PnP for short) algorithm in computer vision to calculate the pose information of the device coordinate system relative to the world coordinate system based on n points (n points corresponding to the light spot in the light spot image). Therefore, the present application can use the pose estimation algorithm to calculate the target pose information of the device coordinate system relative to the world coordinate system based on the light spot image collected by the camera when the deployment position of the light-emitting unit on the motion capture device is known. Then, the posture information is extracted from the target pose information, and the extracted posture information is determined as the target posture information of the device coordinate system relative to the world coordinate system.

[0115] Generally, the PnP algorithm may include: P3P algorithm, P5P algorithm and other algorithms. Moreover, the above-mentioned P3P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on three points. The above-mentioned P5P algorithm calculates the pose information of the device coordinate system relative to the world coordinate system based on five points. Therefore, in the present application, the number of light spots on the motion capture device is at least three. Therefore, when calculating the target pose information of the device coordinate system relative to the world coordinate system based on the light spot image captured by the camera, different PnP algorithms can be selected based on the number of light spots in the light spot image.

[0116] The number of light spots on the motion capture device may be the same as the number of light-emitting units on the motion capture device, or may be different from the number of light-emitting units on the motion capture device. If the number of light spots on the motion capture device is the same as the number of light-emitting units, the number of light-emitting units on the motion capture device is at least three. If the number of light spots on the motion capture device is not equal to the number of light-emitting units, the number of light-emitting units on the motion capture device is at least one.

[0117] In some optional embodiments, it is considered that the motion capture device may include a motion capture device body or include a motion capture device body and accessories rigidly connected to the motion capture device body, such as a base. Then, when the number of light-emitting units on the motion capture device is at least three, as an optional implementation method, the at least three light-emitting units can be distributed in the shell of the motion capture device body according to a preset deployment method. Alternatively, one light-emitting unit is installed in the shell of the motion capture device body according to a preset deployment method, and the remaining light-emitting units are installed in the base according to a preset deployment method. Alternatively, one light-emitting unit is installed in the base according to a preset deployment method, and the remaining light-emitting units are installed in the shell of the motion capture device body according to a preset deployment method. The above-mentioned preset deployment method can be flexibly set according to the shape of the motion capture device body and the base, and there is no restriction on it here.

[0118] S203: calibrate the motion capture device according to the target posture information.

[0119] After determining the target posture information of the device coordinate system relative to the world coordinate system, the present application can calibrate the posture of the motion capture device relative to the world based on the target posture information, so that the subsequent body movements captured by the calibrated motion capture device are more accurate.

[0120] In some optional embodiments, the posture of the motion capture device relative to the world is calibrated. Optionally, the initial posture of the motion capture device relative to the world is calibrated, or the real-time posture of the motion capture device relative to the world is calibrated. This application does not impose any restrictions on this.

[0121] The calibration method of the motion capture device provided in the embodiment of the present application uses a camera to collect the spot image of the light-emitting unit on the motion capture device worn on the human body to determine the target posture information of the device coordinate system relative to the world coordinate system based on the spot image, and then calibrates the motion capture device based on the target posture information. By using the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, the present application can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the motion capture device after calibration can improve the limb motion capture effect, and provide conditions for improving the accuracy of human body posture restoration.

[0122] Combine the following Figure 6 , the calibration method of the motion capture device provided in the embodiment of the present application is further explained. Figure 6 As shown, the method comprises the following steps:

[0123] S301, determining initial posture information of a device coordinate system relative to a reference coordinate system according to inertial measurement unit data sent by a motion capture device.

[0124] The reference coordinate system may be a geographic coordinate system, or may be any user-defined coordinate system that is different from the world coordinate system. In the present application, the roll angle (also called the rolling angle) and the pitch angle in the reference coordinate system are the same as the roll angle and the pitch angle in the world coordinate system.

[0125] The reference coordinate system may be represented as ref. Furthermore, when the reference coordinate system is a geographic coordinate system, the geographic coordinate system may be a north-east geographic coordinate system.

[0126] In the embodiment of the present application, the relationship between the world coordinate system, the reference coordinate system and the device coordinate system can be as follows: Figure 7 As shown. Among them, O G is the world coordinate system, O ref is the reference coordinate system, O L1 is the coordinate system of a motion capture device worn on a thigh of the human lower limb, and O L2 Represents the coordinate system of another motion capture device worn on the calf associated with the above thigh.

[0127] In some optional embodiments, when the motion capture device is in working state, the inertial sensor (i.e., inertial measurement unit) on the motion capture device will send the inertial measurement unit data to the head mounted display device in real time, so that the head mounted display device calculates the initial posture information of the device coordinate system relative to the reference coordinate system based on the inertial measurement unit data, that is, the posture conversion relationship between the device coordinate system and the reference coordinate system.

[0128] Since the inertial measurement unit data sent by the motion capture device may include: accelerometer data, gyroscope data and magnetometer data. Therefore, the present application may use a preset attitude solution algorithm, such as a complementary filtering algorithm, to determine the initial attitude information of the device coordinate system relative to the reference coordinate system based on the accelerometer data, gyroscope data and magnetometer data sent by the motion capture device.

[0129] The inertial measurement unit on the above-mentioned motion capture device sends the inertial measurement unit data to the head-mounted display device. When the camera on the head-mounted display device captures the light spot image, the motion capture device starts to obtain the inertial measurement unit data through the inertial measurement unit and sends the inertial measurement unit data. Alternatively, it may be started at any time before the camera on the head-mounted display device captures the light spot image, and then the motion capture device continues to obtain the inertial measurement unit data through the inertial measurement unit and sends the inertial measurement unit data. This application does not impose any restrictions on this.

[0130] In the present application, based on the inertial measurement unit data sent by the motion capture device, the number of initial posture information of the device coordinate system relative to the reference coordinate system can be one or more. Specifically, when the inertial sensor on the motion capture device only sends the inertial measurement unit data once, then the calculated initial posture information of the device coordinate system relative to the reference coordinate system is one. When the inertial sensor on the motion capture device sends the inertial measurement unit data multiple times, then the calculated initial posture information of the device coordinate system relative to the reference coordinate system is multiple. Among them, multiple initial posture information can be understood as the initial posture information of a device coordinate system relative to the reference coordinate system can be calculated each time the inertial measurement unit data is received.

[0131] Furthermore, the above-mentioned multiple initial posture information may be the same or different, which is specifically determined based on the user's motion state.

[0132] In the present application, a plurality may be understood as two or more than two, that is, at least two.

[0133] Optionally, the initial posture information of the i-th device coordinate system relative to the reference coordinate system in this application can be expressed as: Where, ref represents the reference coordinate system, t1 represents the t1th moment, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, and R represents the posture information.

[0134] S302, collecting a light spot image of a light-emitting unit on a motion capture device through a camera.

[0135] S303, determining target posture information of a device coordinate system relative to a world coordinate system according to the spot image, where the device coordinate system is a motion capture device coordinate system.

[0136] S304, correcting the posture conversion relationship between the world coordinate system and the reference coordinate system according to the target posture information and the initial posture information.

[0137] In some optional embodiments, considering that the camera on the head-mounted display device collects the spot image, the motion capture device can synchronously send the inertial measurement unit data to the head-mounted display device. Therefore, the present application optionally corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system based on the target attitude information determined by the spot image at the same time and the initial attitude information determined by the inertial measurement unit data, which can be optionally achieved by the following formula (1):

[0138]

[0139] in, G R refRepresents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, Represents the target posture information of the i-th device coordinate system relative to the world coordinate system at time t1, Indicates the initial posture information of the i-th device coordinate system relative to the reference coordinate system at time t1, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, R represents the posture information, and -1 represents inversion.

[0140] Considering the attitude conversion relationship between the world coordinate system and the reference coordinate system, it can be expressed by Euler angles. In addition, Euler angles may include heading angle, pitch angle, and roll angle. Therefore, the present application corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information, and may optionally correct the heading angle, pitch angle, and / or roll angle between the world coordinate system and the reference coordinate system.

[0141] It can be understood that the above-mentioned heading angle, pitch angle and / or roll angle can be the heading angle; or, it can be the pitch angle; or, it can be the roll angle; or, it can be the heading angle and the pitch angle; or, it can be the heading angle and the roll angle; or, it can also be the pitch angle and the roll angle; or, it can also be the heading angle, the pitch angle and the roll angle, and the present application does not impose any specific limitation on this.

[0142] In this application, the heading angle can be understood as the angle of rotation of the motion capture device around the X-axis, and the X-axis is perpendicular to the ground. The pitch angle can be understood as the angle of rotation of the motion capture device around the Y-axis, and the roll angle can be understood as the angle of rotation of the motion capture device around the Z-axis.

[0143] In some optional embodiments, considering that the attitude conversion relationship between the world coordinate system and the reference coordinate system is expressed by Euler angles, the pitch angle and the roll angle in the world coordinate system and the reference coordinate system are exactly the same, and only the heading angle parameter item has a deviation. Therefore, the present application corrects the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information, specifically refers to correcting the heading angle parameter item in the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information.

[0144] In the present application, correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system specifically includes: correcting the heading angle conversion relationship in the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information.

[0145] S305: calibrate the motion capture device according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0146] Optionally, after the posture conversion relationship between the world coordinate system and the reference coordinate system is corrected, the present application can determine the new target posture information with high accuracy between the device and the world coordinate system based on the posture conversion relationship obtained after correction and the initial posture information of the device coordinate system relative to the reference coordinate system. Furthermore, based on the new target posture information, the initial posture or real-time posture of the motion capture device can be calibrated, thereby further improving the calibration accuracy of the motion capture device.

[0147] In some optional embodiments, based on the corrected posture conversion relationship combined with the initial posture information of the device coordinate system relative to the reference coordinate system, the new target posture information between the device and the world coordinate system is determined, which can be achieved by the following formula (2):

[0148]

[0149] in, Represents the new target posture information of the i-th device coordinate system relative to the world coordinate system at any time after the t1-th time. G R ref Represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, represents the initial posture information of the i-th device coordinate system relative to the reference coordinate system at any time after the t1-th time, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system and R represents the posture information.

[0150] The calibration method of the motion capture device provided in the embodiment of the present application uses a camera to collect the spot image of the light-emitting unit on the motion capture device worn on the human body to determine the target posture information of the device coordinate system relative to the world coordinate system according to the spot image, and then calibrates the motion capture device according to the target posture information. The present application uses the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and can also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human body posture restoration. In addition, by determining the initial posture information of the device coordinate system relative to the reference coordinate system, and correcting the posture conversion relationship between the world coordinate system and the reference coordinate system according to the target posture information and the initial posture information, and then based on the corrected posture conversion relationship between the world coordinate system and the reference coordinate system, when the motion capture device is calibrated relative to the world, the calibration error can be further reduced and the calibration accuracy of the motion capture device can be improved.

[0151] As an optional implementation scenario, after calibrating the motion capture device, the present application can also determine the posture conversion relationship between the limb coordinate system and the world coordinate system, and then determine the posture of the limb corresponding to the motion capture device in the world coordinate system based on the posture conversion relationship. Figure 8 , a specific description is given of the process of determining the posture conversion relationship between the limb coordinates and the world coordinate system provided in the embodiment of the present application, and determining the posture of the limb in the world coordinate system based on the posture conversion relationship.

[0152] like Figure 8 As shown, the method comprises the following steps:

[0153] S401, determining posture information of a device coordinate system relative to a limb coordinate system.

[0154] In the embodiment of the present application, the limb coordinate system specifically refers to the limb coordinate system of the human limb, which can be expressed as B i , where B stands for Body, i is the identification information of the limb corresponding to the limb coordinate system, and the identification information can be any information that can uniquely determine the identity of the limb, such as the number or name of an important bone in the human body, etc., and there is no specific restriction on it here.

[0155] As an optional implementation, the relationship between the device coordinate system, the world coordinate system, and the limb coordinate system can be exemplified as follows: Fig. 9 As shown. Among them, O G Represents the world coordinate system, O B1 Represents the limb coordinate system of a thigh, OB2 Represents the limb coordinate system of a lower leg associated with the thigh, O L1 represents the device coordinate system of a motion capture device worn on the thigh, and O L2 Represents the device coordinate system of another motion capture device worn on the above calf.

[0156] Optionally, before determining the posture information of the device coordinate system relative to the limb coordinate system, the present application first guides the user to perform a preset calibration action (or a preset calibration posture) and maintain it for a certain period of time, so as to ensure that the user's face and the limbs wearing the motion capture device have a certain relative posture through the preset calibration action. Among them, the preset calibration action can be selected as but not limited to: T-pose, N-pose, other actions that can ensure that the user's face, chest and lower limbs (i.e., legs) are facing the same direction, or other actions that ensure that the user's face and the limbs wearing the motion capture device have a certain relative posture. Afterwards, the posture information of the device coordinate system relative to the limb coordinate system is determined based on the preset action. In this application, a certain time can be flexibly set according to actual calculation requirements, such as 1 second, 2 seconds or 3 seconds, etc., and this application does not impose any restrictions on this.

[0157] It should be noted that the above T-pose can be understood as a posture with both arms stretched horizontally to both sides in a T shape. The above N-pose can be understood as a posture similar to standing at attention.

[0158] In some optional implementations, determining the posture information of the device coordinate system relative to the limb coordinate system may include the following steps:

[0159] Step 1: Under a preset calibration posture, determine the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, as well as the posture information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is the coordinate system of the head-mounted display device.

[0160] Among them, the preset calibration posture can be understood as the above-mentioned preset calibration actions, such as T-pose, N-pose, and other actions that can ensure that the user's face, chest and lower limbs (ie, legs) are facing the same direction, or other actions that ensure that the user's face and the limbs wearing the motion capture device have a certain relative posture.

[0161] In some optional embodiments, after the user makes a preset calibration gesture, the positions of the human limbs and the human head are fixed. Correspondingly, the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system is also fixed. That is, at the moment when the user makes a preset action, the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system is a known quantity, and the known quantity is usually a constant, and the constant is a unit matrix.

[0162] In some possible implementations, since the camera on the head-mounted display device can collect surrounding environment data in real time, and the head-mounted display device can determine the position information of the head-mounted display device in real time based on the surrounding environment data collected by the camera through a computer vision SLAM algorithm. Therefore, when the present application determines the posture information of the head-mounted coordinate system relative to the world coordinate system, it can be determined by a computer vision SLAM algorithm based on the surrounding environment data collected by the camera. The specific determination process is a conventional technology in the field, and it will not be described in detail here.

[0163] Step 2: Determine the posture information of the device coordinate system relative to the limb coordinate system based on the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

[0164] Optionally, determining the posture information of the device coordinate system relative to the limb coordinate system can be achieved by the following formula (3):

[0165]

[0166] in, Represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2, BiR HMD represents the posture transformation relationship between the i-th limb coordinate system and the head-mounted coordinate system at the t2th moment, G R HMD represents the posture information of the head-mounted coordinate system relative to the world coordinate system at time t2, represents the posture information of the i-th device coordinate system relative to the world coordinate system at time t2, B i represents the i-th limb coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, HMD represents the head-mounted coordinate system, R represents the posture information, and -1 represents the inversion.

[0167] Above in, Represents the posture information of the i-th device coordinate system relative to the world coordinate system at time t2. G R ref Represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system, Indicates the initial posture information of the device coordinate system relative to the reference coordinate system at time t2, ref represents the reference coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, and R represents the posture information.

[0168] It should be understood that the above t1 refers to the time of acquiring the motion capture device image or the spot image, or the time of acquiring the motion capture device image or the spot image and completing the calibration of the motion capture device, and t2 refers to the time of determining the posture of the device coordinate system relative to the limb coordinate system under the preset calibration posture, and t2 may be later than t1, that is, t1 <t2。

[0169] Considering that the posture conversion relationship between the corrected world coordinate system and the reference coordinate system has been determined in S304 in the aforementioned embodiment, the posture conversion relationship is fixed during calibration or after the last calibration and before the next calibration. In addition, the inertial measurement unit in the motion capture device will send inertial measurement unit data in real time, so the initial posture information of the device coordinate system relative to the reference coordinate system at time t2 can be calculated based on the received inertial measurement unit data. Furthermore, based on the initial posture information of the device coordinate system relative to the reference coordinate system at time t2 and the posture conversion relationship between the corrected world coordinate system and the reference coordinate system, the present application can determine the posture information of the i-th device coordinate system relative to the world coordinate system at time t2 when the user makes a preset calibration posture.

[0170] Among them, based on the initial posture information of the device coordinate system relative to the reference coordinate system at time t2, and the corrected posture conversion relationship between the world coordinate system and the reference coordinate system, the posture information of the device coordinate system relative to the world coordinate system is determined. For details, please refer to the S305 part of the aforementioned embodiment, which will not be elaborated here.

[0171] S402, determining a posture conversion relationship between the limb coordinate system and the world coordinate system according to posture information of the device coordinate system relative to the limb coordinate system and posture information of the device coordinate system relative to the world coordinate system.

[0172] In some optional implementations, the posture conversion relationship between the limb coordinate system and the world coordinate system can be determined by the following formula (4):

[0173]

[0174] in, represents the posture transformation relationship between the limb coordinate system and the world coordinate system at the t2th moment, Represents the posture information of the i-th device relative to the world coordinate system at time t2. represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at time t2, B i represents the i-th limb coordinate system, L irepresents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, R represents the posture information, and -1 represents inversion.

[0175] Furthermore, the limb coordinate system can be calibrated based on the posture conversion relationship between the limb coordinate system and the world coordinate system.

[0176] It should be understood that the above-mentioned posture conversion relationship between the limb coordinate system and the world coordinate system can be understood as the posture information of the limb coordinate system relative to the world coordinate system, and the posture information can represent the posture of the limb in the world coordinate system.

[0177] In some optional implementations, the present application determines the posture information of the device coordinate system relative to the limb coordinate system, and the execution order of calibrating the motion capture device may be to first perform the calibration operation on the motion capture device, and then perform the determination of the posture information of the device coordinate system relative to the limb coordinate system, and then obtain the posture information of the limb coordinate system relative to the world coordinate system, and realize the corresponding limb motion capture function; or, it may also first perform the determination of the posture information of the device coordinate system relative to the limb coordinate system, and then perform the calibration operation on the motion capture device, and then obtain the posture information of the limb coordinate system relative to the world coordinate system, and realize the corresponding limb motion capture function. The present application does not impose specific restrictions on this.

[0178] Among them, when first determining the posture information of the device coordinate system relative to the limb coordinate system and then performing the calibration operation on the motion capture device, the present application first caches the posture information of the device coordinate system relative to the limb coordinate system at the time t2, and then when performing the calibration operation, the motion capture device is calibrated based on the cached posture information of the device coordinate system relative to the limb coordinate system at the time t2.

[0179] The above first determines the posture information of the device coordinate system relative to the limb coordinate system, which can be achieved by the following formula (5):

[0180]

[0181] in, represents the posture information of the i-th device coordinate system relative to the i-th limb coordinate system at the time t2. represents the posture conversion relationship between the i-th limb coordinate system and the head-mounted coordinate system at the t2th moment. The posture conversion relationship is a fixed value when the user's limb makes a preset calibration posture, and the fixed value is usually a unit matrix. G R HMD represents the posture information of the head-mounted coordinate system relative to the world coordinate system at time t2, G R refIt represents the attitude transformation relationship between the corrected world coordinate system and the reference coordinate system at the time t2. represents the initial posture information of the i-th device coordinate system relative to the reference coordinate system at time t2, B i represents the i-th limb coordinate system, L i represents the i-th motion capture device coordinate system, that is, the i-th device coordinate system, G represents the world coordinate system, HMD represents the head-mounted coordinate system, ref represents the reference coordinate system, R represents the posture information, and -1 represents inversion.

[0182] It should be noted that in the present application, the limbs can be regarded as fixed in the limb coordinate system, and the position of the limb coordinate system relative to the world can be equivalent to the position of the limb relative to the world coordinate system.

[0183] Furthermore, the motion capture device may be fixed to the motion capture device coordinate system, and the relationship between the motion capture device coordinate system and the reference coordinate system or the world coordinate system is equivalent to the relationship between the motion capture device and the reference coordinate system or the world coordinate system.

[0184] The calibration method of the motion capture device provided in the embodiment of the present application uses a camera to collect the spot image of the light-emitting unit on the motion capture device worn on the human limbs, so as to determine the target posture information of the device coordinate system relative to the world coordinate system according to the spot image, and then calibrate the motion capture device according to the target posture information. The present application uses the spot image of the light-emitting unit on the motion capture device to calibrate the motion capture device, which can effectively reduce the difficulty of calibrating the motion capture device, shorten the calibration time, and can also reduce the calibration error, improve the calibration accuracy of the motion capture device, so that the calibrated motion capture device can improve the limb motion capture effect, and provide conditions for improving the accuracy of human posture restoration. In addition, the present application can also calibrate the limb coordinate system based on the posture conversion relationship between the limb coordinate system and the world coordinate system, and determine the posture of the limb in the world coordinate system based on the calibrated limb coordinate system, thereby improving the accuracy and correctness of determining the posture of the human limb in the world coordinate system, and improving the capture effect of limb movement.

[0185] Please refer to the attached Fig.10 , a calibration device for a motion capture device proposed in an embodiment of the present application is described. Fig.10 It is a schematic block diagram of a calibration device for a motion capture device provided in an embodiment of the present application.

[0186] The motion capture device is worn on a human limb, and the device is configured on a head-mounted display device, which includes a camera. Fig.10As shown, the motion capture device calibration apparatus 500 includes: an image acquisition module 510 , a posture determination module 520 and a device calibration module 530 .

[0187] The image acquisition module 510 is used to acquire the image of the motion capture device through the camera;

[0188] A posture determination module 520, for determining target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image, the device coordinate system being the motion capture device coordinate system;

[0189] The device calibration module 530 is used to calibrate the motion capture device according to the target posture information.

[0190] In an optional implementation of the embodiment of the present application, the posture determination module 520 is specifically used to:

[0191] Extracting image feature points from the motion capture device image;

[0192] According to the image feature points, target posture information of the device coordinate system relative to the world coordinate system is determined.

[0193] In an optional implementation of the embodiment of the present application, the motion capture device includes a light emitting unit and an image acquisition module 510, which is specifically used for:

[0194] Collecting a light spot image of the light-emitting unit on the motion capture device by using the camera;

[0195] Accordingly, the posture determination module 520 is specifically used for:

[0196] According to the light spot image, target posture information of the device coordinate system relative to the world coordinate system is determined.

[0197] In an optional implementation of the embodiment of the present application, the posture determination module 520 is further configured to:

[0198] Determine the target pose information of the device coordinate system relative to the world coordinate system according to the light spot image by using a pose estimation algorithm;

[0199] Posture information is extracted from the target posture information, and the posture information is determined as target posture information of the device coordinate system relative to the world coordinate system.

[0200] In an optional implementation of the embodiment of the present application, the apparatus 500 further includes:

[0201] A first determination module, configured to determine initial posture information of the device coordinate system relative to a reference coordinate system according to inertial measurement unit data sent by the motion capture device;

[0202] Accordingly, the device calibration module 530 is specifically used for:

[0203] Correcting the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information;

[0204] The motion capture device is calibrated according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0205] In an optional implementation of the embodiment of the present application, the device calibration module 530 is further used to:

[0206] Correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system.

[0207] In an optional implementation of the embodiment of the present application, the inertial measurement unit data includes: accelerometer data, gyroscope data and magnetometer data, and the first determination module is specifically used to:

[0208] The complementary filtering algorithm is used to determine the initial posture information of the device coordinate system relative to the reference coordinate system according to the accelerometer data, the gyroscope data and the magnetometer data sent by the motion capture device.

[0209] In an optional implementation of the embodiment of the present application, the apparatus 500 further includes:

[0210] A second determination module, used to determine the posture information of the device coordinate system relative to the limb coordinate system;

[0211] The third determination module is used to determine the posture conversion relationship between the limb coordinate system and the world coordinate system according to the posture information of the device coordinate system relative to the limb coordinate system and the posture information of the device coordinate system relative to the world coordinate system.

[0212] In an optional implementation of the embodiment of the present application, the second determining module is specifically configured to:

[0213] Under a preset calibration posture, determining a posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, and posture information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is a head-mounted display device coordinate system;

[0214] The posture information of the device coordinate system relative to the limb coordinate system is determined according to the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

[0215] In an optional implementation manner of the embodiment of the present application, the second determining module is further configured to:

[0216] The posture information of the device coordinate system relative to the world coordinate system is determined according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

[0217] In an optional implementation manner of the embodiment of the present application, the number of light spots on the motion capture device is at least three.

[0218] An optional implementation of the embodiment of the present application further includes:

[0219] The display module is used to display an image perspective screen and display in the image perspective screen guiding information for guiding the user to place the motion capture device in the image perspective screen.

[0220] The calibration device for motion capture equipment provided in the embodiment of the present application uses a camera to collect images of motion capture equipment worn on human limbs, so as to determine target posture information of the device coordinate system relative to the world coordinate system based on the motion capture equipment image, and then calibrate the motion capture equipment based on the target posture information. By using the motion capture equipment image to calibrate the motion capture equipment, the present application can effectively reduce the difficulty of calibrating the motion capture equipment, shorten the calibration time, and also reduce the calibration error, improve the calibration accuracy of the motion capture equipment, so that the calibrated motion capture equipment can improve the limb motion capture effect, and provide conditions for improving the accuracy of human posture restoration.

[0221] It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.10 The apparatus 500 shown may perform Figure 3 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the device 500 are respectively to implement Figure 3 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0222] In the above, the device 500 of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the first aspect method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned first aspect method embodiment in conjunction with its hardware.

[0223] Fig.11 is a schematic block diagram of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 600 may include:

[0224] The memory 610 and the processor 620, the memory 610 is used to store the computer program and transmit the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the motion capture device calibration method in the embodiment of the present application.

[0225] For example, the processor 620 may be used to execute the above-mentioned motion capture device calibration method embodiment according to the instructions in the computer program.

[0226] In some embodiments of the present application, the processor 620 may include but is not limited to:

[0227] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0228] In some embodiments of the present application, the memory 610 includes but is not limited to:

[0229] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0230] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the motion capture device calibration method provided by the present application. The one or more modules may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0231] like Fig.11 As shown, the electronic device 600 may further include:

[0232] The transceiver 630 may be connected to the processor 620 or the memory 610 .

[0233] The processor 620 may control the transceiver 630 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include an antenna, and the number of antennas may be one or more.

[0234] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0235] The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the method of the above method embodiment.

[0236] An embodiment of the present application also provides a computer program product including program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method of the above method embodiment.

[0237] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital videodisc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0238] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0239] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0240] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0241] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for calibrating a motion capture device, characterized in that: The motion capture device is worn on a limb of a human body, the method is applied to a head-mounted display device, the head-mounted display device includes a camera, and the method includes: Collecting the motion capture device image through the camera; Determine, based on the motion capture device image, target posture information of a device coordinate system relative to a world coordinate system, the device coordinate system being the motion capture device coordinate system; The motion capture device is calibrated according to the target posture information.

2. The method according to claim 1, characterized in that Determining target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image includes: Extracting image feature points from the motion capture device image; According to the image feature points, target posture information of the device coordinate system relative to the world coordinate system is determined.

3. The method according to claim 1, characterized in that The motion capture device includes a light emitting unit; the step of collecting an image of the motion capture device through the camera includes: Collecting a light spot image of the light-emitting unit on the motion capture device by using the camera; Accordingly, determining target posture information of the device coordinate system relative to the world coordinate system according to the motion capture device image includes: According to the light spot image, target posture information of the device coordinate system relative to the world coordinate system is determined.

4. The method according to claim 3, characterized in that Determining target posture information of the device coordinate system relative to the world coordinate system according to the light spot image includes: Determine the target pose information of the device coordinate system relative to the world coordinate system according to the light spot image by using a pose estimation algorithm; Posture information is extracted from the target posture information, and the posture information is determined as target posture information of the device coordinate system relative to the world coordinate system.

5. The method according to claim 3, characterized in that: Also includes: Determining initial posture information of the device coordinate system relative to a reference coordinate system based on inertial measurement unit data sent by the motion capture device; Accordingly, the calibrating the motion capture device includes: Correcting the attitude conversion relationship between the world coordinate system and the reference coordinate system according to the target attitude information and the initial attitude information; The motion capture device is calibrated according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

6. The method according to claim 5, characterized in that The correcting the posture conversion relationship between the world coordinate system and the reference coordinate system comprises: Correcting the heading angle in the attitude conversion relationship between the world coordinate system and the reference coordinate system.

7. The method according to claim 5, characterized in that The inertial measurement unit data includes: accelerometer data, gyroscope data and magnetometer data, and the initial posture information of the device coordinate system relative to the reference coordinate system is determined according to the inertial measurement unit data sent by the motion capture device, including: The complementary filtering algorithm is used to determine the initial posture information of the device coordinate system relative to the reference coordinate system according to the accelerometer data, the gyroscope data and the magnetometer data sent by the motion capture device.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: Determining posture information of the device coordinate system relative to the limb coordinate system; A posture conversion relationship between the limb coordinate system and the world coordinate system is determined according to the posture information of the device coordinate system relative to the limb coordinate system and the posture information of the device coordinate system relative to the world coordinate system.

9. The method according to claim 8, characterized in that The determining of the posture information of the device coordinate system relative to the limb coordinate system includes: Under a preset calibration posture, determining a posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, and posture information of the head-mounted coordinate system relative to the world coordinate system, wherein the head-mounted coordinate system is a head-mounted display device coordinate system; The posture information of the device coordinate system relative to the limb coordinate system is determined according to the posture conversion relationship between the limb coordinate system and the head-mounted coordinate system, the posture information of the head-mounted coordinate system relative to the world coordinate system, and the posture information of the device coordinate system relative to the world coordinate system.

10. The method according to claim 9, characterized in that The posture information of the device coordinate system relative to the world coordinate system includes: The posture information of the device coordinate system relative to the world coordinate system is determined according to the corrected posture conversion relationship between the world coordinate system and the reference coordinate system.

11. The method according to claim 3, characterized in that The number of light spots on the motion capture device is at least three.

12. The method according to claim 1, characterized in that Also includes: An image perspective screen is displayed, and guiding information for guiding a user to place the motion capture device in the image perspective screen is displayed in the image perspective screen.

13. A calibration device for a motion capture device, characterized in that: The motion capture device is worn on a limb of a human body, and the device is configured on a head-mounted display device, and the head-mounted display device includes a camera, including: An image acquisition module, used for acquiring the image of the motion capture device through the camera; A posture determination module, used to determine target posture information of a device coordinate system relative to a world coordinate system according to the motion capture device image, wherein the device coordinate system is the motion capture device coordinate system; The device calibration module is used to calibrate the motion capture device according to the target posture information.

14. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the motion capture device calibration method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the motion capture device calibration method according to any one of claims 1 to 12.

16. A computer program product comprising program instructions, characterized in that When the program instructions are executed on an electronic device, the electronic device is enabled to execute the motion capture device calibration method according to any one of claims 1 to 12.

Images