Virtual data set generation method and device, equipment, storage medium and test system

Abstract

The embodiment of the invention provides a virtual data set generation method and device, equipment, a storage medium and a test system. The method comprises the following steps: acquiring a real motion track, a synchronous localization and mapping (SLAM) estimation track and inertial measurement unit (IMU) data when the mixed reality equipment moves. And according to the real motion track, the SLAM estimation track and the IMU data, a virtual data set of a virtual scene of the mixed reality equipment is generated, and the virtual data set is used for testing an SLAM algorithm. The method is used for achieving the effect of improving the adaptability and the natural degree of the generated virtual data set and the virtual scene, and the accuracy of testing the SLAM algorithm by using the virtual data set is improved.

Description

Technical Field

[0001] This application relates to the field of extended reality, and more particularly to a method, apparatus, device, storage medium, and testing system for generating virtual datasets. Background Technology

[0002] In the context of the data-driven metaverse development, Extended Reality (XR) devices are increasingly demanding powerful spatial awareness and interactive capabilities. This demand relies on large-scale Simultaneous Localization and Mapping (SLAM) datasets for algorithm training and validation. Currently, traditional data acquisition and simulation methods face physical and logical limitations of real-world environments, which often lack diversity and fail to meet the needs of different research and applications. Furthermore, the cost of setting up environments and acquiring data is very high. Therefore, virtual datasets have emerged as a solution. However, currently generated virtual datasets suffer from poor adaptability to virtual scenes and appear unnatural.

[0003] Therefore, improving the adaptability and naturalness of the generated virtual datasets to the virtual scenes is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and testing system for generating virtual datasets, which aims to improve the adaptability and naturalness of the generated virtual datasets to virtual scenes.

[0005] In a first aspect, embodiments of this application provide a method for generating a virtual dataset, including:

[0006] Acquire the real-time motion trajectory of the mixed reality device, the trajectory estimated by Simultaneous Localization and Mapping (SLAM), and the IMU data;

[0007] Based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data, a virtual dataset of the virtual scene of the mixed reality device is generated, and the virtual dataset is used for testing the SLAM algorithm.

[0008] Optionally, generating a virtual dataset of the virtual scene of the mixed reality device based on the real motion trajectory, the SLAM-estimated trajectory, and the IMU data includes:

[0009] Based on the actual motion trajectory and the IMU data, the IMU trajectory of the mixed reality device is generated;

[0010] Based on the IMU trajectory and the SLAM estimated trajectory, a first trajectory ground truth value of the mixed reality device is generated, wherein the first trajectory ground truth value is the actual motion trajectory of the mixed reality device derived from the IMU trajectory;

[0011] The first trajectory true value is transformed into the first coordinate system used by the virtual scene to generate the second trajectory true value of the mixed reality device;

[0012] Render the second trajectory ground truth value to generate a sequence of rendered images in the virtual scene;

[0013] The virtual dataset is generated based on the first trajectory ground truth, the rendered image sequence, and the IMU data.

[0014] Optionally, rendering the second trajectory ground truth value to generate the rendered image sequence in the virtual scene includes:

[0015] Obtain the virtual camera extrinsic parameters of the mixed reality device;

[0016] The rendered image sequence is generated based on the virtual camera extrinsic parameters and the second trajectory ground truth.

[0017] Optionally, the step of converting the first trajectory truth value to the first coordinate system used by the virtual scene to generate the second trajectory truth value includes:

[0018] Based on the true value of the first trajectory and the SLAM-estimated trajectory, the coordinate system transformation relationship between the first coordinate system and the second coordinate system used by the actual motion trajectory is obtained;

[0019] Based on the coordinate system transformation relationship, the true value of the first trajectory is transformed into the first coordinate system to generate the true value of the second trajectory.

[0020] Optionally, generating the first ground truth value of the mixed reality device's trajectory based on the IMU trajectory and the SLAM-estimated trajectory includes:

[0021] Based on the IMU trajectory and the SLAM estimated trajectory, obtain the extrinsic parameters of the IMU corresponding to the IMU trajectory and the mixed reality device;

[0022] The true value of the first trajectory is generated based on the IMU trajectory and the extrinsic parameters.

[0023] Optionally, obtaining the extrinsic parameters of the IMU corresponding to the IMU trajectory and the mixed reality device based on the IMU trajectory and the SLAM estimated trajectory includes:

[0024] Based on the IMU trajectory, the first relative pose transformation of the IMU between the first and second moments during the motion of the mixed reality device is obtained;

[0025] The second relative pose transformation of the mixed reality device between the first time moment and the second time moment is obtained based on the SLAM estimated trajectory;

[0026] The extrinsic parameters are obtained based on the first relative pose transformation and the second relative pose transformation.

[0027] Optionally, before obtaining the first relative pose transformation of the IMU between the first and second moments during the motion of the mixed reality device based on the IMU trajectory, the method further includes:

[0028] Obtain the operating time information of the IMU and the operating time information of the mixed reality device;

[0029] Based on the IMU's operating time information and the mixed reality device's operating time information, determine whether the IMU and the mixed reality device's operating times are synchronized;

[0030] If the IMU and the mixed reality device are not synchronized in their operating times, then the operating times of the IMU and the mixed reality device shall be synchronized.

[0031] Optionally, the method further includes:

[0032] Obtain the coordinate system type of the first coordinate system used by the virtual scene, and the coordinate system type of the second coordinate system used by the real motion trajectory, wherein the coordinate system type includes a left-handed coordinate system and a right-handed coordinate system;

[0033] If the coordinate system types of the first coordinate system and the second coordinate system are different, then the coordinate system types of the first coordinate system and the second coordinate system shall be unified.

[0034] Secondly, embodiments of this application provide a testing system, the system comprising: an IMU, a mixed reality device, a real-world target, and a motion capture device, wherein the positions of the IMU, the mixed reality device, and the real-world target remain relatively fixed when the mixed reality device moves;

[0035] The IMU is used to collect IMU data from the mixed reality device;

[0036] The mixed reality device is used to acquire SLAM-estimated trajectories;

[0037] The motion capture device is used to capture the real-world motion trajectory of the mixed reality device through the real-world target.

[0038] Thirdly, embodiments of this application provide a virtual dataset generation apparatus, characterized in that the apparatus includes:

[0039] The acquisition module is used to acquire the actual motion trajectory, SLAM estimated trajectory, and inertial measurement unit (IMU) data of the mixed reality device during its movement.

[0040] The processing module is used to generate a virtual dataset of the virtual scene of the mixed reality device based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data. The virtual dataset is used for testing the SLAM algorithm.

[0041] Fourthly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0042] The memory stores computer-executed instructions;

[0043] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0044] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first aspect.

[0045] In a sixth aspect, this application provides a computer program product that, when executed by a processor, is used to implement the method described in the first aspect.

[0046] The virtual dataset generation method, apparatus, device, storage medium, and testing system provided in this application simultaneously acquire the motion trajectory of the MR device in reality, the SLAM estimated trajectory estimated by the SLAM algorithm, and the IMU data during the motion of the MR device in reality while it is moving. Then, data calibration and alignment are performed based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data to generate a virtual dataset for the virtual scene. This method integrates the SLAM estimated trajectory of the MR device, the real motion trajectory, and the IMU data to generate a virtual dataset that is more adaptable to the virtual scene and closer to the motion patterns in the real world. This improves the adaptability of the generated virtual dataset to the virtual scene, the naturalness of the virtual dataset, and ultimately enhances the accuracy of SLAM algorithm testing using this virtual dataset. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 This is a schematic diagram of the structure of a testing system provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating a virtual dataset generation method provided in an embodiment of this application;

[0050] Figure 3 A flowchart illustrating another virtual dataset generation method provided in this application embodiment;

[0051] Figure 4 A flowchart illustrating another virtual dataset generation method provided in this application embodiment;

[0052] Figure 5 A flowchart illustrating another virtual dataset generation method provided in this application embodiment;

[0053] Figure 6 A flowchart illustrating another virtual dataset generation method provided in this application embodiment;

[0054] Figure 7 A flowchart illustrating another virtual dataset generation method provided in this application embodiment;

[0055] Figure 8 This is a schematic diagram of the structure of a virtual dataset generation device provided in an embodiment of this application;

[0056] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0059] First, let me explain the terms used in this application:

[0060] Mixed Reality (MR) refers to the technology that merges the real and virtual worlds to create new visual environments. It combines the characteristics of augmented reality and virtual reality, enabling deep integration and interaction between virtual and real objects.

[0061] Motion capture (MoCap) is a technology that uses motion capture systems or devices to record the movement trajectory of people or objects. For example, reflective balls can be attached to people or objects to capture the light emitted by an infrared high-speed camera of the motion capture system by reflecting the light off the reflective balls.

[0062] An Inertial Measurement Unit (IMU) is an electronic device used to measure the acceleration, angular velocity, and / or magnetic field of an object. It consists of accelerometers, gyroscopes, etc., and is used for attitude estimation, navigation, etc.

[0063] Simultaneous Localization and Mapping (SLAM) dataset: This is a dataset used to study and evaluate SLAM algorithms, containing environmental and location information, etc.

[0064] A Software Development Kit (SDK) is a collection of resources that provides the tools, documentation, sample code, and other resources needed to develop software, helping to improve development efficiency. Developers can use the resources in the SDK to quickly build the framework of software and call predefined functions, thereby reducing development time and workload, improving development efficiency, and ensuring software compatibility with specific platforms or technologies.

[0065] The following section introduces the current methods for generating virtual datasets:

[0066] Method 1: Generate simulated trajectories based on a virtual scene, and obtain the corresponding virtual dataset by rendering the simulated trajectories. This method can generate simulated trajectories in a virtual scene using specific algorithms, such as kinematic equations or stochastic process models. By defining parameters such as initial conditions, boundary conditions, and motion laws, the motion trajectory of objects (i.e., simulated trajectories) is generated within the spatial framework of the virtual scene. Then, based on existing object models, material properties, lighting models, and other information in the virtual scene, the scene state at various moments along the simulated trajectory is visually presented to output the corresponding virtual dataset.

[0067] However, the motion trajectories of people or objects in the real world and / or their associated IMU data are complex and unpredictable, while virtual simulation trajectories are often constructed based on predetermined models and assumptions. Although this method can simulate motion trajectories in the real world to some extent, it still cannot fully match the complexity and diversity of motion trajectories in the real world.

[0068] Method 2: Adapt the pre-collected real-world trajectories to the virtual scene, then render and synthesize them to obtain the corresponding virtual dataset. This method processes the pre-collected real-world trajectories, directly converting them into the virtual scene so that the real-world trajectories can be adapted to the virtual scene. Then, rendering is performed based on the real-world trajectories in the virtual scene to generate a virtual dataset based on the rendering results of the real-world trajectories.

[0069] However, when directly transferring pre-collected real-world trajectories into a virtual scene, unnatural interaction phenomena often occur, such as clipping or abnormal object collisions. These anomalies in the generated virtual dataset negatively impact subsequent applications. For instance, when the virtual dataset generated in this way is used to develop or test spatial perception and interaction algorithms for MR devices, these problems may prevent the algorithms from accurately understanding or responding to the physical rules of the virtual scene, thus affecting user immersion and experience quality in practical applications. Furthermore, these adaptation issues reduce the consistency between the virtual scene and the expected real-world scene, leading to decreased accuracy and reliability of simulation results when relying on this virtual dataset for simulations.

[0070] In view of this, this application provides a method for generating a virtual dataset. This method simultaneously acquires the MR device's real-world motion trajectory, SLAM-estimated trajectory, and IMU data during the MR device's motion. Then, data calibration and alignment are performed based on the real-world motion trajectory, SLAM-estimated trajectory, and IMU data to generate a virtual dataset for the virtual scene. This method integrates the MR device's SLAM-estimated trajectory, real-world motion trajectory, and IMU data to generate a virtual dataset that is more adaptable to the virtual scene and more closely resembles the motion patterns in the real world. This improves the adaptability and naturalness of the generated virtual dataset to the virtual scene, thereby enhancing the accuracy of SLAM algorithm testing using this virtual dataset.

[0071] The execution entity of the virtual dataset generation method provided in this application can be a terminal device with data processing capabilities, or the processing chip of the terminal device, or software or program code that implements the virtual dataset generation method. When the execution entity is a terminal device with data processing capabilities, the terminal device can be, for example, a computing device such as a computer or mobile phone with computing capabilities. The computing device can be equipped with software or program code that runs the virtual dataset generation method. The software or program code processes the actual motion trajectory of the MR device, the SLAM estimated trajectory, and the IMU data to generate a virtual dataset corresponding to the virtual scene.

[0072] Figure 1 This is a schematic diagram of the structure of a testing system provided in an embodiment of this application. Figure 1 As shown, the test system includes: an IMU, a real-world target, and motion capture equipment. This test system is used for data collection from MR devices.

[0073] The positions of the IMU, MR device, and real target remain relatively fixed when the MR device moves.

[0074] One possible implementation involves a rigid connection between the IMU, the MR device, and the real-world target in the testing system. Optionally, the IMU and the real-world target can be directly connected to the MR device separately, or the IMU can be directly connected to the real-world target, and then either the IMU or the real-world target can be directly connected to the MR device. Alternatively, the IMU can be built into the MR device. Alternatively, the IMU, the real-world target, and the MR device can be rigidly connected via connectors, such as connecting rods. Figure 1 The diagram illustrates the structure of the test system, using the example of directly connecting the IMU and the real-world target to the MR device.

[0075] Another possible implementation is that the test system can be connected to the IMU, MR device, and real target through different brackets. The device connection ends of these three different brackets (i.e., the ends that connect to the IMU, MR device, and real target) remain relatively fixed in position when moving, so that the IMU, MR device, and real target can remain relatively fixed in position when the MR device moves.

[0076] The IMU is used to collect IMU data from the MR device, which may include gyroscope data, accelerometer data, etc. When the MR device moves, the IMU remains relatively fixed in position to the MR device; therefore, the IMU data of the MR device can be obtained based on the IMU data collected by the IMU.

[0077] MR devices are used to acquire SLAM-estimated trajectories. The virtual scene of the MR device can be built based on an application development platform, or it can be directly obtained from existing public datasets. When the virtual scene is built on a virtual scene development platform, this platform can include, for example, 3D modeling and simulation platforms such as Blender, Unity 3D, or Nvidia Omniverse. The virtual scene is built using the SDK provided by the MR device within the supported virtual scene development platform. The virtual scene can be imported into the development platform, and the tools and scripting languages ​​provided by the platform can be used to create and generate an MR application. Running this MR application through the MR device allows for real-time acquisition of the SLAM-estimated trajectory using SLAM algorithms during application execution.

[0078] Optionally, the desired range of motion in the virtual scene can be completely contained within the activity space of the real space by setting an activity space range that matches the spatial range of the virtual scene, thereby ensuring the correctness and integrity of the subsequently generated virtual dataset.

[0079] Motion capture equipment is used to acquire the real-world motion trajectory of a mixed reality (MR) device using a real-world target. The motion capture equipment can capture the motion of the MR device by using a real-world target whose relative position to the MR device remains fixed during movement. This real-world target can be, for example, the aforementioned reflective sphere. Taking a reflective sphere as an example, the motion capture equipment can capture the real-world motion trajectory of the MR device by capturing the light emitted by the motion capture equipment's infrared high-speed camera reflected from the reflective sphere.

[0080] This application does not limit the specific implementation method of motion capture equipment and real target, as long as the motion capture equipment and real target can realize the function of acquiring the real motion trajectory of MR equipment through the above method.

[0081] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0082] Figure 2 This is a flowchart illustrating a method for generating a virtual dataset according to an embodiment of this application. Figure 2 As shown, the method includes:

[0083] S201. Acquire the actual motion trajectory, SLAM estimated trajectory, and IMU data of the MR device during its movement.

[0084] The MR device used in this step can be, for example, a head-mounted MR device, a handheld MR device, etc., as long as it can merge the real and virtual worlds and achieve deep integration and interaction between virtual and real objects. This application does not impose any restrictions on this. For example, the MR device can be an MR head-mounted display device.

[0085] One possible implementation is that the actual motion trajectory, SLAM-estimated trajectory, and IMU data of the MR device during movement are obtained from data acquired through pre-testing, such as from a database storing pre-testing data. This pre-testing can be performed as described above. Figure 1 The test system shown obtained this data.

[0086] Another possible implementation is that the MR device's actual motion trajectory, SLAM-estimated trajectory, and IMU data are obtained in real time via, for example... Figure 1 The test system shown was used to obtain the data.

[0087] S202. Based on the actual motion trajectory, SLAM estimated trajectory, and IMU data, generate a virtual dataset of the virtual scene of the MR device.

[0088] This virtual dataset is used to test the SLAM algorithm. For example, a portion of the data in the virtual dataset can be used as input to the SLAM algorithm to obtain the output data. This output data can then be compared with data of the same type in the virtual dataset to test the effectiveness of the SLAM algorithm.

[0089] Since the frequency and accuracy of real-world motion trajectories acquired through motion capture are typically higher than those estimated by SLAM, to improve the frequency and accuracy of SLAM-estimated trajectories used in virtual datasets and enhance their adaptability and realism to the virtual scene, data calibration and alignment can be performed by combining real-world motion trajectories, SLAM-estimated trajectories, and IMU data. This transforms the real-world motion trajectories (i.e., ground truth trajectories) from the second coordinate system of the real-world scene to a first coordinate system that conforms to the virtual scene (i.e., another type of ground truth trajectories). This second coordinate system is different from the first coordinate system.

[0090] Based on the trajectory ground truth and IMU data, a virtual dataset of the virtual scene of the MR device can be generated. This virtual dataset can include the trajectory ground truth, IMU data, and a sequence of rendered images generated in the virtual scene based on the trajectory ground truth. Optionally, the trajectory ground truth can be, for example, the trajectory ground truth in a second coordinate system or the trajectory ground truth in a first coordinate system, etc., and this application does not limit this.

[0091] The virtual dataset generation method provided in this application acquires the motion trajectory of the MR device in reality, the SLAM estimated trajectory estimated by the SLAM algorithm, and the IMU data during the motion of the MR device in reality while it is moving. Then, data calibration and alignment are performed based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data to generate a virtual dataset for the virtual scene. This method integrates the SLAM estimated trajectory of the MR device, the real motion trajectory, and the IMU data to generate a virtual dataset that is more adaptable to the virtual scene and closer to the motion patterns in the real world. This improves the adaptability of the generated virtual dataset to the virtual scene, the naturalness of the virtual dataset, and ultimately enhances the accuracy of SLAM algorithm testing using this virtual dataset.

[0092] The following section details how step S202 above generates a virtual dataset of the virtual scene for the MR device based on the real motion trajectory, the SLAM-estimated trajectory, and the IMU data. Figure 3 This is a flowchart illustrating another virtual dataset generation method provided in an embodiment of this application. Figure 3 As shown, the aforementioned step S202 may specifically include:

[0093] S301. Generate the IMU trajectory of the MR device based on the actual motion trajectory and IMU data.

[0094] Here, the actual motion trajectory is essentially the motion trajectory of the real target on the MR device within a second coordinate system of the real scene. For example, denoted by B for the second coordinate system and M for the real target, the actual motion trajectory of the real target can be represented as follows: .pass By performing global nonlinear optimization processing on the IMU data, a high-precision six-degrees-of-freedom (6-DoF) trajectory of the IMU in the second coordinate system can be obtained. Where I represents IMU. It contains complete motion information of the IMU in the three-dimensional space (coordinate system B) of the real scene, including six degrees of freedom of motion such as translation along the three coordinate axes and rotation around the three coordinate axes. For details on how to perform global nonlinear optimization processing, please refer to existing technologies, which will not be elaborated here.

[0095] S302. Based on the IMU trajectory and the SLAM estimated trajectory, generate the first trajectory ground truth value of the MR device.

[0096] The first trajectory true value is the actual motion trajectory of the MR device derived from the IMU trajectory. The difference between the actual motion trajectory derived here and the actual motion trajectory of the MR device directly acquired above is that the moving subject is different. The actual motion trajectory derived here is the actual motion trajectory of the MR device (or a preset reference object in the MR device); while the actual motion trajectory directly acquired is actually the actual motion trajectory of the actual target captured by the motion capture device.

[0097] Among them, the accuracy of the true value of the first trajectory is higher than that of the trajectory estimated by SLAM, and the frequency of the trajectory data of the true value of the first trajectory is also higher than that of the trajectory data of the trajectory estimated by SLAM.

[0098] The SLAM estimated trajectory of the MR device can be either the estimated trajectory of the MR device itself, or it can be the motion trajectory of a reference object within the MR device. When the SLAM estimated trajectory of the MR device is the motion trajectory of a reference object within the MR device, the SLAM estimated trajectory is obtained based on the motion trajectory of the reference object during the movement of the MR device. For example, taking a virtual environment built on the Unity development platform, the SLAM estimated trajectory exported through the MR device's SDK refers to the motion trajectory of a reference object in the MR device set in the Unity development platform within the first coordinate system of the virtual environment. It should be understood that the above is merely a specific example using the Unity development platform; this application does not limit how the reference object is determined, as it is related to the development platform used to build the virtual environment.

[0099] Furthermore, this first trajectory ground truth is essentially generated based on the reference object corresponding to the IMU trajectory and the SLAM estimated trajectory, or by the MR device. By using the extrinsic parameters between the reference object (or MR device) and the IMU, the relationship between the position and orientation of the reference object (or MR device) and the IMU data during motion is determined. Based on these extrinsic parameters and the IMU trajectory, the data points in the actual motion trajectory of the reference object (or MR device) corresponding to each data point in the IMU trajectory can be obtained. The actual motion trajectory of the reference object (or MR device) is then the first trajectory ground truth of the MR device (i.e., the motion trajectory of the MR device in the second coordinate system of the real scene). Since the data points included in the first trajectory ground truth correspond one-to-one with the data points included in the IMU trajectory, the accuracy of the first trajectory ground truth is the same as the accuracy of the IMU trajectory, both being higher than the accuracy of the SLAM estimated trajectory.

[0100] S303. Transform the true value of the first trajectory into the first coordinate system used by the virtual scene to generate the true value of the second trajectory.

[0101] Since the true value of the first trajectory is essentially the actual motion trajectory of the MR device or reference object, it is based on data in the second coordinate system of the real scene. However, the second coordinate system used in the real scene is different from the first coordinate system used in the virtual scene, and the accuracy of SLAM-estimated trajectory is lower than that of the real motion trajectory and IMU data. Therefore, to obtain a high-precision motion trajectory for use in a virtual scene, the true value of the first trajectory can be transformed to the first coordinate system used in the virtual scene to generate the true value of the second trajectory of the reference object. This second true value is the motion trajectory based on the first coordinate system of the virtual scene, and since it is obtained by coordinate system transformation based on the true value of the first trajectory, its accuracy is the same as that of the true value of the first trajectory.

[0102] Specifically, by obtaining the transformation relationship between the first coordinate system and the second coordinate system, the true value of the first trajectory can be transformed from the second coordinate system to the first coordinate system according to the transformation relationship, thereby generating the true value of the second trajectory of the reference object.

[0103] Optionally, the transformation relationship can be a preset transformation relationship obtained directly; or it can be calculated based on the parameters or definitions of the first coordinate system and the second coordinate system; or it can be obtained by matching the true value of the first trajectory with the estimated trajectory using a spatial matching algorithm.

[0104] S304. Render the true value of the second trajectory to generate a sequence of rendered images in the virtual scene.

[0105] The virtual camera in the MR device is used to obtain the observation results of the virtual camera on the true value of the second trajectory, and the observation results are used to call the rendering engine to render and generate an image sequence from the perspective of the virtual camera, so as to generate a rendered image sequence.

[0106] The rendering engine can be, for example, a rendering simulation engine such as Unity, UE, or Blender. This rendering simulation engine can provide a large number of basic tools, including modeling, rendering, animation, visual effects, compositing, texturing, and simulation functions, to achieve image rendering based on the observation results of a virtual camera using the second trajectory ground truth. This application does not limit the choice of the rendering engine, as long as it can achieve the above functions.

[0107] Specifically, the detailed steps for rendering the second trajectory ground truth value and generating the rendered image sequence in the virtual scene can be referred to the following sub-steps:

[0108] S3041. Obtain the virtual camera extrinsic parameters of the MR device.

[0109] The virtual camera has specific parameter settings in the virtual scene, such as position, orientation, and field of view. The virtual camera is used to determine from which perspective to observe elements in the virtual scene (such as trajectory ground truth), thereby generating a corresponding sequence of rendered images.

[0110] Optionally, the virtual camera extrinsic parameters can be predetermined when setting up the virtual camera. For example, when setting up the virtual camera, the extrinsic parameters of the virtual camera and the MR device are determined based on the virtual camera's parameter settings. Alternatively, the virtual camera extrinsic parameters can be determined based on the pre-set parameter settings of the virtual camera. For example, taking the trajectory of the reference object of the MR device as an example where the second trajectory truth value is true, the extrinsic parameters of the virtual camera are the calibration extrinsic parameters of the virtual camera and the reference object.

[0111] S3042. Based on the virtual camera extrinsic parameters and the true value of the second trajectory, generate a sequence of rendered images.

[0112] Based on the virtual camera's extrinsic parameters and the true value of the second trajectory, the observation results of the virtual camera on the true value of the second trajectory are obtained, and the rendering engine is used to render and generate an image sequence from the virtual camera's perspective, so as to generate a rendered image sequence.

[0113] For example, taking the trajectory of a reference object (let's say D) in the MR device as the true value of the second trajectory, then the extrinsic parameter of the virtual camera is the extrinsic parameter from the virtual camera to the reference object D (let's say D is the reference object). The observation result of the virtual camera on the true value of the second trajectory can be obtained by the following formula (1):

[0114] (1)

[0115] in, The virtual camera's observation of the true value of the second trajectory. This represents the true value of the second trajectory. In the formula... The first coordinate system used to represent the virtual scene is C, which represents the virtual camera.

[0116] In acquiring After that, it can be used. In a virtual scene, a rendering engine renders a sequence of images from the perspective of the virtual camera.

[0117] S305. Generate a virtual dataset based on the true value of the first trajectory, the rendered image sequence, and the IMU data.

[0118] Optionally, the virtual dataset can be a dataset that includes the ground truth values ​​of the first trajectory, the rendered image sequence, and the IMU data. Alternatively, the virtual dataset can be a dataset that includes the IMU trajectory, the rendered image sequence, and the IMU data. Or, the virtual dataset can also be a dataset that includes the ground truth values ​​of the second trajectory, the rendered image sequence, and the IMU data.

[0119] The method provided in this application generates an IMU trajectory for an MR device using real-world motion trajectories and IMU data. Based on the IMU trajectory and the SLAM-estimated trajectory, a first true trajectory value for the MR device is generated. This first true trajectory value is then transformed into a first coordinate system used by the virtual scene to generate a second true trajectory value. Based on the second true trajectory value, the IMU data, and the first true trajectory value, a virtual dataset for the virtual scene is generated. By integrating the MR device's SLAM-estimated trajectory, the real-world motion trajectories, and the IMU data, a virtual dataset with higher adaptability to the virtual scene and closer to the motion patterns in the real world is generated. This improves the adaptability of the generated virtual dataset to the virtual scene and the naturalness of the virtual dataset, thereby enhancing the user experience or the accuracy of the simulation.

[0120] The following section provides a detailed explanation of how the true value of the first trajectory is converted to the first coordinate system used by the virtual scene in the aforementioned step S303 to generate the true value of the second trajectory. Figure 4 This is a flowchart illustrating another virtual dataset generation method provided in an embodiment of this application. Figure 4 As shown, the aforementioned step S303 may specifically include:

[0121] S401. Based on the true value of the first trajectory and the trajectory estimated by SLAM, obtain the coordinate system transformation relationship between the first coordinate system and the second coordinate system used by the actual motion trajectory.

[0122] A spatial matching algorithm is used to match the ground truth of the first trajectory with the SLAM-estimated trajectory to obtain the coordinate system transformation relationship between the first coordinate system and the second coordinate system used by the actual motion trajectory. This spatial matching algorithm can refer to existing technologies, as long as it can achieve the function of matching two motion trajectories of the same object (in this application, the reference object in the MR device or the MR device itself) under different coordinate systems in different spaces (i.e., the first coordinate system corresponding to the virtual environment and the second coordinate system corresponding to the real environment) to obtain the coordinate system transformation relationship between the first and second coordinate systems.

[0123] S402. Based on the coordinate system transformation relationship, transform the true value of the first trajectory to the first coordinate system to generate the true value of the second trajectory.

[0124] Suppose that the coordinate system transformation relationship can be expressed as Then, the true value of the first trajectory is transformed into the first coordinate system, and the true value of the second trajectory can be calculated using the following formula (2):

[0125] (2)

[0126] in, This is the true value of the first trajectory, that is, the true value of the trajectory of the reference object D in the second coordinate system B; The coordinate transformation relationship between the first coordinate system W and the second coordinate system B; This is the true value of the second trajectory, that is, the true value of the trajectory of the reference object D in the first coordinate system W.

[0127] The following section provides a detailed explanation of how the first true value of the MR device's trajectory is generated based on the IMU trajectory and the SLAM-estimated trajectory in step S302. Figure 5 This is a flowchart illustrating another method for generating a virtual dataset provided in an embodiment of this application. Figure 5 As shown, the aforementioned step S302 may specifically include:

[0128] S501. Based on the IMU trajectory and the SLAM estimated trajectory, obtain the external parameters of the IMU and MR devices corresponding to the IMU trajectory.

[0129] The IMU corresponding to this IMU trajectory is the aforementioned IMU in the MR device that maintains a fixed relative position during movement. This extrinsic parameter is calculated based on this IMU trajectory and the SLAM-estimated trajectory using a hand-eye extrinsic parameter calibration method.

[0130] When the extrinsic parameter is calculated using the hand-eye extrinsic parameter calibration method based on the SLAM estimated trajectory of the IMU trajectory and the MR device, taking the SLAM estimated trajectory as the estimated trajectory obtained based on the reference object in the MR device as an example, then the extrinsic parameter is the extrinsic parameter between the reference object and the IMU.

[0131] Specifically, the steps for calculating and obtaining extrinsic parameters based on the IMU trajectory and the SLAM-estimated trajectory of the MR device using the hand-eye extrinsic parameter calibration method can be referred to the following sub-steps:

[0132] S5011. Obtain the first relative pose transformation of the IMU between the first and second moments during the movement of the MR device based on the IMU trajectory.

[0133] First, the corresponding IMU is determined based on the IMU trajectory. Then, IMU data is acquired from this IMU; the IMU data may include, for example, accelerometer data and gyroscope data. The accelerometer measures the acceleration of the MR device along three axes, and the gyroscope measures the angular velocity of the MR device around three axes. Between the first and second time points, integrating the accelerometer data yields the velocity change, and further integration yields the position change. Integrating the gyroscope data yields the angular change.

[0134] By combining these position and angle changes, the first relative pose transformation of the IMU between the first and second time points can be constructed, for example, represented by a first relative pose transformation matrix. This first relative pose transformation matrix can describe the relative changes in the IMU's position and attitude (such as rotation and translation) from the first time point to the second time point, i.e., the first relative pose transformation.

[0135] S5012. Obtain the second relative pose transformation of the MR device between the first and second time points based on the SLAM-estimated trajectory.

[0136] The SLAM estimation trajectory of the MR device contains the pose information of the MR device at different times.

[0137] By estimating the trajectory using SLAM, the poses of the MR device in the virtual scene at the first and second moments can be extracted. By calculating the transformation relationship between the pose at the first moment and the pose at the second moment, the second relative pose transformation of the MR device between the first and second moments can be obtained.

[0138] S5013. Obtain extrinsic parameters based on the first relative pose transformation and the second relative pose transformation.

[0139] Since the relative positions of the MR device (or reference object) and the IMU remain fixed during movement, their relative pose transformations between the first and second moments are essentially the same. Therefore, extrinsic parameters can be obtained based on the first and second relative pose transformations through hand-eye extrinsic parameter calibration and nonlinear optimization.

[0140] For example, taking the pose transformation with the second relative pose transformation as the reference object, where P represents the first relative pose transformation and Q represents the second relative pose transformation, based on the above principle, the relationship between the external part and the first relative pose transformation and the second relative pose transformation can be shown in the following formula (3):

[0141] (3)

[0142] in, , is an external parameter, representing the external parameter relationship between the IMU (i.e., I) and the reference object (i.e., D).

[0143] S502. Based on the IMU trajectory and the extrinsic parameters, generate the true value of the first trajectory.

[0144] Since the IMU trajectory is known, as well as the extrinsic parameter representing the extrinsic relationship between the IMU and the MR device, which represents the pose relationship between the IMU and the MR device, the trajectory data points of the MR device in the second coordinate system of the real scene (i.e., the first trajectory true value) corresponding to the trajectory data points of the IMU can be calculated one by one based on the trajectory data points included in the IMU trajectory and the extrinsic parameter.

[0145] The method provided in this application embodiment obtains the extrinsic parameters of the IMU and MR device corresponding to the IMU trajectory by transforming their respective relative poses between the first and second time points. Based on the IMU trajectory and the extrinsic parameters, a first trajectory true value corresponding to the IMU trajectory is generated. This first trajectory true value is the motion trajectory of the MR device in the second coordinate system of the real scene, thereby obtaining a first trajectory true value with higher accuracy than the SLAM estimated trajectory. This improves the accuracy of the virtual dataset when generating the virtual dataset in the future, and also improves the realism of the motion trajectory represented by the virtual dataset.

[0146] In one possible implementation, Figure 5 Prior to the method shown, this application can also process whether the working times of the IMU and MR devices are aligned, so as to improve the accuracy of obtaining the second extrinsic parameter through hand-eye extrinsic parameter calibration, and further improve the accuracy of generating virtual datasets based on the second extrinsic parameter.

[0147] Figure 6 This is a flowchart illustrating another method for generating a virtual dataset provided in an embodiment of this application. Figure 6 As shown, the method may further include:

[0148] S601. Obtain the IMU's operating time information and the MR device's operating time information.

[0149] In an MR system, both the IMU and the MR device have their own internal time management mechanisms.

[0150] For an IMU, its operating time information can include the IMU's startup time, which is defined as time 0, from which the IMU begins timing. For an MR device, the operating time information is similar to that of an IMU; it can include the MR device's startup time, which is defined as time 0, from which the MR device begins timing.

[0151] The operating time information of the IMU and the MR device can be obtained through the driver interface of the IMU and the MR device or through specific system query commands, which is not limited in this application.

[0152] S602. Based on the IMU's working time information and the MR device's working time information, determine whether the working times of the IMU and the MR device are synchronized.

[0153] Whether the working time of the IMU and MR devices is synchronized can be determined by whether their startup times are the same.

[0154] Consider the difference in the start-up times of the IMU and MR devices. If the IMU starts timing from its own start-up time, while the MR device starts timing from an initialization time different from the IMU's start-up time, then these two timing start points are inconsistent on the timeline, indicating that the working times of the IMU and MR are not synchronized.

[0155] S603. If the working times of the IMU and MR devices are not synchronized, then synchronize the working times of the IMU and MR devices.

[0156] Optionally, the IMU's operating time can be adjusted to synchronize with the MR device's operating time. Alternatively, the MR device's operating time can be adjusted to synchronize with the IMU's operating time. Or, the MR device's operating time and the IMU's operating time can be synchronized with a preset time.

[0157] When the startup time of the IMU differs from that of the MR device, an offset can be introduced into the time calculation. For example, if the MR device starts up 10,000 nanoseconds earlier than the IMU, this 10,000 nanosecond offset can be added to each timestamp when processing time-related data from the IMU, thus synchronizing the two in time calculation.

[0158] The method provided in this application embodiment synchronizes the working time of the IMU and MR device to ensure the aforementioned Figure 5 The first and second moments for determining the relative pose transformation mentioned in the document are the same moments for both the IMU and the MR device. That is, the first moment used by the IMU to determine the first relative pose transformation is the same moment used by the MR device to determine the second relative pose transformation, and the second moment used by the IMU to determine the first relative pose transformation is the same moment used by the MR device to determine the second relative pose transformation. This improves the accuracy of the acquired extrinsic parameters and enhances the accuracy of the subsequently generated virtual dataset.

[0159] The method provided in this application addresses the simultaneous existence of virtual and real-world scenes, requiring data processing based on data from both scenarios. Different virtual scene development platforms and motion capture devices / systems use coordinate systems determined according to their specific needs. Therefore, this application can further reduce computational load and improve the efficiency of generating virtual datasets by pre-unifying the coordinate system type of the first coordinate system used in the virtual scene and the second coordinate system used in the real-world scene.

[0160] Figure 7 This is a flowchart illustrating another method for generating a virtual dataset provided in an embodiment of this application. Figure 7 As shown, the method may further include:

[0161] S701. Obtain the coordinate system type of the first coordinate system used in the virtual scene, and the coordinate system type of the second coordinate system used in the real motion trajectory.

[0162] The coordinate system types include left-handed coordinate system and right-handed coordinate system.

[0163] As mentioned earlier, the coordinate system type of the first coordinate system in a virtual scene may differ from the coordinate system type of the second coordinate system in a real-world scene. For the first coordinate system used by a virtual scene, its coordinate system type information is typically determined and stored during the virtual scene's construction process. For example, in a virtual scene built using the Unity engine, the coordinate system type is set during scene initialization. Assuming there is a specific identifier representing the coordinate system type in the virtual scene's configuration file or in the Unity engine's internal data structure, the type of the first coordinate system can be obtained by reading this identifier.

[0164] The type of the second coordinate system is determined by the motion capture device. The coordinate system type can be obtained from the configuration parameters of the motion capture device.

[0165] S702. If the coordinate system types of the first coordinate system and the second coordinate system are different, then the coordinate system types of the first coordinate system and the second coordinate system shall be unified.

[0166] When the types of the first and second coordinate systems are determined to be different, for example, the first coordinate system is a left-handed coordinate system and the second coordinate system is a right-handed coordinate system, the coordinate system types of the first and second coordinate systems can be unified by performing coordinate system transformation.

[0167] For example, taking the Unity platform as an example, the Unity platform uses a left-handed coordinate system (where the positive x-axis points to the right, the positive y-axis points upward, and the positive z-axis points forward). In this case, the left-handed coordinate system can be transformed into a right-handed coordinate system by keeping the positive directions of the x and z axes unchanged and reversing the y-axis downward.

[0168] Specifically, if the displacement of a point in the Unity coordinate system is represented as (Tx, Ty, Tz), and the rotation is represented by quaternions as (Qw, Qx, Qy, Qz), for example, the left-handed coordinate system can be transformed to the right-handed coordinate system through position transformation and rotation transformation. The position transformation of this point can be represented by the following formula (4):

[0169] (4)

[0170] in, This represents the position parameter of the point in the right-handed coordinate system.

[0171] The rotational transformation of this point can be expressed by the following formula (5):

[0172] (5)

[0173] in, This represents the rotation parameter of the point in the right-handed coordinate system.

[0174] The method provided in this application embodiment obtains the coordinate system type of the first coordinate system used in the virtual scene and the coordinate system type of the second coordinate system used in the real motion trajectory. When the coordinate system types of the first coordinate system and the second coordinate system are different, the coordinate system types of the first coordinate system and the second coordinate system are unified through coordinate system transformation, thereby improving the accuracy and generation efficiency of the generated virtual dataset.

[0175] Figure 8 This is a schematic diagram of a virtual dataset generation device provided in an embodiment of this application. Figure 8 As shown, the virtual dataset generation device may include: an acquisition module 11 and a processing module 12.

[0176] The acquisition module 11 is used to acquire the actual motion trajectory, SLAM estimated trajectory, and IMU data of the MR device during movement.

[0177] The processing module 12 is used to generate a virtual dataset of the virtual scene of the MR device based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data. The virtual dataset is used for testing the SLAM algorithm.

[0178] Optionally, processing module 12 is specifically used to generate the IMU trajectory of the MR device based on the actual motion trajectory and the IMU data. Based on the IMU trajectory and the SLAM-estimated trajectory, a first true trajectory value of the MR device is generated. The first true trajectory value is transformed into a first coordinate system used by the virtual scene to generate a second true trajectory value of the MR device. The second true trajectory value is rendered to generate a sequence of rendered images in the virtual scene, and the virtual dataset is generated based on the first true trajectory value, the rendered image sequence, and the IMU data. The first true trajectory value is the actual motion trajectory of the MR device derived from the IMU trajectory.

[0179] Optionally, the processing module 12 is specifically used to obtain the virtual camera extrinsic parameters of the MR device. Based on the virtual camera extrinsic parameters and the second trajectory ground truth value, the rendered image sequence is generated.

[0180] Optionally, the processing module 12 is specifically used to obtain the coordinate system transformation relationship between the first coordinate system and the second coordinate system used by the actual motion trajectory based on the first trajectory ground truth value and the SLAM estimated trajectory. Based on the coordinate system transformation relationship, the first trajectory ground truth value is transformed into the first coordinate system to generate the second trajectory ground truth value.

[0181] Optionally, the processing module 12 is specifically used to obtain the extrinsic parameters of the IMU and the MR device corresponding to the IMU trajectory based on the IMU trajectory and the SLAM estimated trajectory. Based on the IMU trajectory and the extrinsic parameters, the true value of the first trajectory is generated.

[0182] Optionally, the processing module 12 is specifically configured to obtain, based on the IMU trajectory, a first relative pose transformation of the IMU between a first moment and a second moment during the movement of the MR device. It also obtains, based on the SLAM-estimated trajectory, a second relative pose transformation of the MR device between the first moment and the second moment. Finally, it obtains the extrinsic parameters based on the first relative pose transformation and the second relative pose transformation.

[0183] Optionally, before acquiring the first relative pose change of the IMU between the first and second moments during the movement of the MR device, the processing module 12 is further configured to acquire the operating time information of the IMU and the operating time information of the MR device. Based on the operating time information of the IMU and the operating time information of the MR device, it is determined whether the operating time of the IMU and the MR device is synchronized. If the operating time of the IMU and the MR device is not synchronized, the operating time of the IMU and the MR device is synchronized.

[0184] Optionally, the acquisition module 11 is further configured to acquire the coordinate system type of the first coordinate system used by the virtual scene, and the coordinate system type of the second coordinate system used by the real motion trajectory, wherein the coordinate system type includes a left-handed coordinate system and a right-handed coordinate system. The processing module 12, in the case that the coordinate system types of the first coordinate system and the second coordinate system are different, is further configured to unify the coordinate system types of the first coordinate system and the second coordinate system.

[0185] The virtual dataset generation apparatus provided in this application embodiment can execute the virtual dataset generation method in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0186] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device is used to execute the aforementioned virtual dataset generation method, and may be, for example, a terminal device with data processing capabilities as described above. Figure 9 As shown, the electronic device 900 may include at least one processor 901, a memory 902, and a communication interface 903.

[0187] The memory 902 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0188] The memory 902 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0189] The processor 901 is used to execute computer execution instructions stored in the memory 902 to implement the method described in the foregoing method embodiments. The processor 901 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0190] The processor 901 can communicate and interact with external devices through the communication interface 903. These external devices can be, for example, the aforementioned motion capture device or motion capture system. In specific implementations, if the communication interface 903, memory 902, and processor 901 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0191] Optionally, in a specific implementation, if the communication interface 903, memory 902, and processor 901 are integrated on a single chip, then the communication interface 903, memory 902, and processor 901 can communicate through an internal interface.

[0192] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the methods described in the above embodiments.

[0193] This application also provides a program product including executable instructions stored in a readable storage medium. At least one processor of a computing device can read the executable instructions from the readable storage medium, and the execution of the executable instructions by the at least one processor causes the computing device to implement the described virtual dataset generation method.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a virtual dataset, characterized in that, include: Acquire the real-time motion trajectory of the mixed reality device, the trajectory estimated by Simultaneous Localization and Mapping (SLAM), and the IMU data; Based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data, a virtual dataset of the virtual scene of the mixed reality device is generated, and the virtual dataset is used for testing the SLAM algorithm.

2. The method according to claim 1, characterized in that, The step of generating a virtual dataset of the virtual scene of the mixed reality device based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data includes: Based on the actual motion trajectory and the IMU data, the IMU trajectory of the mixed reality device is generated; Based on the IMU trajectory and the SLAM estimated trajectory, a first trajectory ground truth value of the mixed reality device is generated, wherein the first trajectory ground truth value is the actual motion trajectory of the mixed reality device derived from the IMU trajectory; The first trajectory true value is transformed into the first coordinate system used by the virtual scene to generate the second trajectory true value of the mixed reality device; Render the second trajectory ground truth value to generate a sequence of rendered images in the virtual scene; The virtual dataset is generated based on the first trajectory ground truth, the rendered image sequence, and the IMU data.

3. The method according to claim 2, characterized in that, The step of rendering the second trajectory ground truth value and generating the rendered image sequence in the virtual scene includes: Obtain the virtual camera extrinsic parameters of the mixed reality device; The rendered image sequence is generated based on the virtual camera extrinsic parameters and the second trajectory ground truth.

4. The method according to claim 2, characterized in that, The step of converting the first trajectory truth value to the first coordinate system used by the virtual scene to generate the second trajectory truth value includes: Based on the true value of the first trajectory and the SLAM-estimated trajectory, the coordinate system transformation relationship between the first coordinate system and the second coordinate system used by the actual motion trajectory is obtained; Based on the coordinate system transformation relationship, the true value of the first trajectory is transformed into the first coordinate system to generate the true value of the second trajectory.

5. The method according to claim 2, characterized in that, The step of generating the first trajectory ground truth value of the mixed reality device based on the IMU trajectory and the SLAM estimated trajectory includes: Based on the IMU trajectory and the SLAM estimated trajectory, obtain the extrinsic parameters of the IMU corresponding to the IMU trajectory and the mixed reality device; The true value of the first trajectory is generated based on the IMU trajectory and the extrinsic parameters.

6. The method according to claim 5, characterized in that, The step of obtaining the extrinsic parameters of the IMU corresponding to the IMU trajectory and the mixed reality device based on the IMU trajectory and the SLAM estimated trajectory includes: Based on the IMU trajectory, the first relative pose transformation of the IMU between the first and second moments during the motion of the mixed reality device is obtained; The second relative pose transformation of the mixed reality device between the first time moment and the second time moment is obtained based on the SLAM estimated trajectory; The extrinsic parameters are obtained based on the first relative pose transformation and the second relative pose transformation.

7. The method according to claim 6, characterized in that, Before obtaining the first relative pose transformation of the IMU between the first and second moments during the motion of the mixed reality device based on the IMU trajectory, the method further includes: Obtain the operating time information of the IMU and the operating time information of the mixed reality device; Based on the IMU's operating time information and the mixed reality device's operating time information, determine whether the IMU and the mixed reality device's operating times are synchronized; If the IMU and the mixed reality device are not synchronized in their operating times, then the operating times of the IMU and the mixed reality device shall be synchronized.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Obtain the coordinate system type of the first coordinate system used by the virtual scene, and the coordinate system type of the second coordinate system used by the real motion trajectory, wherein the coordinate system type includes a left-handed coordinate system and a right-handed coordinate system; If the coordinate system types of the first coordinate system and the second coordinate system are different, then the coordinate system types of the first coordinate system and the second coordinate system shall be unified.

9. A testing system for data collection from mixed reality devices, characterized in that, The system includes: an IMU, a real-world target, and a motion capture device. The positions of the IMU, the mixed-reality device, and the real-world target remain relatively fixed when the mixed-reality device moves. The IMU is used to collect IMU data from the mixed reality device; The mixed reality device is used to acquire SLAM-estimated trajectories; The motion capture device is used to capture the real-world motion trajectory of the mixed reality device through the real-world target.

10. The testing system according to claim 9, characterized in that, The IMU, the mixed reality device, and the real-world target in the test system are rigidly connected.

11. A virtual dataset generation device, characterized in that, The device includes: The acquisition module is used to acquire the actual motion trajectory, SLAM estimated trajectory, and inertial measurement unit (IMU) data of the mixed reality device during its movement. The processing module is used to generate a virtual dataset of the virtual scene of the mixed reality device based on the real motion trajectory, the SLAM estimated trajectory, and the IMU data. The virtual dataset is used for testing the SLAM algorithm.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

14. A computer program product, characterized in that, When the computer program product is executed by a processor, it is used to implement the method as described in any one of claims 1-8.

Images