Method and apparatus for determining relative pose, extended reality system, device and medium

Abstract

Embodiments of the present disclosure disclose a method and apparatus for determining a relative pose, an extended reality system, a device, and a medium. Among them, an auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle, and the auxiliary positioning device is fixedly connected to the vehicle. The method includes: obtaining an image collected by an image sensor when a user wears the target head-mounted display device, first inertial data collected by a first inertial measurement unit, and second inertial data collected by a second inertial measurement unit; determining pose data of the target head-mounted display device relative to the auxiliary positioning device based on the image; performing data fusion on the first inertial data, the second inertial data, and the pose data and performing a fusion detection; when the fusion detection passes, determining a relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data.

Description

Technical Field

[0001] The present disclosure relates to extended reality technology, and in particular, to a method and apparatus for determining a relative pose, an extended reality system, a device, and a medium. Background Art

[0002] Currently, extended reality technology has been widely applied to various fields, such as healthcare, retail, education, social media, entertainment, etc., to enhance the user experience.

[0003] Extended reality may include augmented reality, virtual reality, mixed reality, etc. It can provide an extended reality experience to users by integrating rendered virtual content or integrating the rendered virtual content with the physical world, and allows users to interact with the real or physical environment enhanced or augmented by the virtual content.

[0004] When a user wears a head-mounted display device in a moving vehicle (such as a car), due to the different motion states of the user and the vehicle, there is relative motion between the head-mounted display device and the vehicle.

[0005] In related technologies, in order to distinguish the head movement of the user from the movement of the vehicle, generally, a vision sensor and an inertial measurement unit provided on the head-mounted display device are used to collect image data and inertial data respectively, and calculate the relative pose of the head-mounted display device relative to the vehicle, so as to characterize the relative motion between the head-mounted display device and the vehicle. Summary of the Invention

[0006] Embodiments of the present disclosure provide a method and apparatus for determining a relative pose, an extended reality system, a device, and a medium.

[0007] In one aspect of the embodiments of the present disclosure, a method for determining a relative pose is provided. The method is applied to an auxiliary positioning device. The auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle. The auxiliary positioning device is fixedly connected to the vehicle and is provided with an image sensor and a first inertial measurement unit. The target head-mounted display device is provided with a second inertial measurement unit. The method includes: obtaining an image collected by the image sensor when the user wears the target head-mounted display device, first inertial data collected by the first inertial measurement unit, and second inertial data collected by the second inertial measurement unit; determining pose data of the target head-mounted display device relative to the auxiliary positioning device based on the image; performing data fusion on the first inertial data, the second inertial data, and the pose data and performing a fusion detection; when the fusion detection passes, determining the relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data.

[0008] In another aspect of the embodiments of the present disclosure, a device for determining a relative pose is provided. The device is applied to an auxiliary positioning device. The auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle. The auxiliary positioning device is fixedly connected to the vehicle and is provided with an image sensor and a first inertial measurement unit. The target head-mounted display device is provided with a second inertial measurement unit. The device includes: a data acquisition unit configured to acquire an image collected by the image sensor when a user wears the target head-mounted display device, first inertial data collected by the first inertial measurement unit, and second inertial data collected by the second inertial measurement unit; a pose prediction unit configured to determine pose data of the target head-mounted display device relative to the auxiliary positioning device based on the image; a data fusion unit configured to perform data fusion on the first inertial data, the second inertial data, and the pose data and perform fusion detection; and a pose determination unit configured to, when the fusion detection passes, determine the relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data.

[0009] In yet another aspect of the embodiments of the present disclosure, an extended reality system is provided, including an auxiliary positioning device, a head-mounted display device, and a device for determining a relative pose. The auxiliary positioning device is fixedly connected to a movable vehicle and is provided with an image sensor and a first inertial measurement unit. The head-mounted display device is provided with a second inertial measurement unit. When the head-mounted display device and the auxiliary positioning device are located inside a moving vehicle, the device for determining a relative pose determines the relative pose of the head-mounted display device relative to the vehicle by the method in the above embodiments, so that the head-mounted display device generates a projection image based on the relative pose.

[0010] In still another aspect of the embodiments of the present disclosure, an electronic device is provided, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, and when the computer program is executed, implementing the method in any of the above embodiments.

[0011] In still another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implementing the method described in any of the above embodiments.

[0012] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0013] The drawings constituting a part of the specification depict embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0014] Referring to the accompanying drawings, the present disclosure can be more clearly understood according to the following detailed description, where:

[0015] Figure 1 Schematic diagram of the architecture of an extended reality system according to some embodiments of the present disclosure;

[0016] Figure 2 Schematic diagram of the scenario applicable to the extended reality system according to some embodiments of the present disclosure;

[0017] Figure 3 Schematic flowchart of some embodiments of the method for determining relative pose according to the present disclosure;

[0018] Figure 4 Schematic flowchart of the fusion detection in some embodiments of the method for determining relative pose according to the present disclosure;

[0019] Figure 5 Schematic flowchart of some embodiments of the method for determining relative pose according to the present disclosure;

[0020] Figure 6 Schematic flowchart of some other embodiments of the method for determining relative pose according to the present disclosure;

[0021] Figure 7 Schematic diagram of the structure of some embodiments of the device for determining relative pose according to the present disclosure;

[0022] Figure 8 Schematic diagram of the structure of some application embodiments of the electronic device according to the present disclosure. Detailed implementation manners

[0023] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0024] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.

[0025] It should also be understood that in the embodiments of the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two or more.

[0026] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, unless otherwise clearly defined or given a contrary indication in the context, it can generally be understood as one or more.

[0027] In addition, the term "and / or" in the present disclosure merely describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, both A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0028] It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be elaborated one by one.

[0029] Meanwhile, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn in actual proportional relationships.

[0030] The following description of at least one exemplary embodiment is actually only illustrative and in no way a limitation on the present disclosure and its application or use.

[0031] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said techniques, methods, and devices should be regarded as part of the specification.

[0032] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0033] The embodiments of the present disclosure can be applied to electronic devices for extended implementation, such as AR glasses and VR glasses, and can also be terminal devices loaded with extended implementation application software and / or virtual reality application software, such as smartphones and tablets. The method disclosed in this application can run in a head-mounted display device, a target-assisted positioning device, or other electronic devices such as a car computer, and there is no unique limitation here. It can be understood that the device disclosed in this application can also be correspondingly arranged in a head-mounted display device, a target-assisted positioning device, or other electronic devices such as a car computer.

[0034] The present disclosure provides methods, devices, systems, electronic devices, and computer-readable storage media that can be used in scenarios for providing an immersive extended reality experience on a moving vehicle, so that the extended reality system can separate the movement of the user from the movement of the moving vehicle where the user is located, thereby making the virtual image content match the movement perceived by the user.

[0035] In practice, when the same vehicle includes multiple sets of head-mounted display devices and auxiliary positioning devices, when the visual sensor of the head-mounted display device is used to capture the image of the auxiliary positioning device, it is easy to be interfered by the auxiliary positioning devices corresponding to other head-mounted display devices, resulting in the problem of identification conflict between the multiple sets of head-mounted display devices and the auxiliary positioning devices, affecting the calculation accuracy of the posture data. The method for determining relative posture disclosed in the present invention uses an image sensor to capture an image of a user wearing a head-mounted display device, and predicts the posture data of the head-mounted display device relative to the auxiliary positioning device. Through fusion detection, it is determined whether there is a target head-mounted display device corresponding to the auxiliary positioning device in the posture data, thereby solving the problem of identification conflict and helping to improve the calculation accuracy of the posture data.

[0036] Exemplary system

[0037] Figure 1 An extended reality system of some embodiments of the present disclosure is shown, including: a head-mounted display device, an auxiliary positioning device, and a device for determining a relative posture, wherein the head-mounted display device is provided with a second inertial measurement unit, the auxiliary positioning device includes an image sensor and a first inertial measurement unit, and the first inertial measurement unit can be directly set on the auxiliary positioning device, or can be set outside the main body of the auxiliary positioning device in a split manner, for example, it can be fixed on a movable carrier, or the inertial measurement unit of the movable carrier can be directly used as the second inertial measurement unit of the auxiliary positioning device. The device for determining the relative posture can be set in the head-mounted display device, or can be set on the auxiliary positioning device, or can be set on other electronic devices, such as smart phones. The device for determining the relative posture can estimate the posture of the user relative to the vehicle based on the received data, so that the head-mounted display device generates a projection image based on the provided posture data, thereby providing an immersive extended reality experience.

[0038] In the present application, the auxiliary positioning device can be a split device or an integrated device, and there is no unique limitation here. In the case where the auxiliary positioning device is a split device, the first inertial measurement unit and the auxiliary positioning device, such as an image sensor, can be a separate structure. In this case, the first inertial measurement unit can be fixed separately to the mobile vehicle as a separate structure. For example, the first inertial measurement unit can be an inertial sensing device, or the first inertial measurement unit can also be an IMU that comes with the mobile vehicle.

[0039] The above-mentioned extended reality system can also be applied to a mobile vehicle, in which case the auxiliary positioning device can be removably fixed to the mobile vehicle, and the mobile vehicle can include any type of transportation tool or mobile environment, such as a vehicle, airplane, train, ship, elevator or park amusement facility.

[0040] In some embodiments, the apparatus for determining the relative pose may include: a data acquisition unit, a pose prediction unit, a data fusion unit, and a pose determination unit.

[0041] The data acquisition unit may receive data from one or more of an image sensor, a motion sensor, and an external sensor. Optionally, the image sensor may include any suitable type and number of cameras, such as a depth camera, an RGB camera, and combinations thereof, for collecting image data. The motion sensor may be one or more inertial measurement units (IMUs) for obtaining inertial data of the auxiliary positioning device; the external sensor may include a motion sensor provided on the head-mounted display device for obtaining inertial data of the head-mounted display device.

[0042] The pose prediction unit may be configured to predict the pose data of the head-mounted display device relative to the auxiliary positioning device based on the image data obtained by the image sensor.

[0043] The data fusion unit may be configured to fuse the data obtained by the motion sensor and the external sensor with the pose data and perform a fusion detection.

[0044] The pose determination unit may be configured to, when the fusion detection passes, fuse the data obtained by the motion sensor and the external sensor with the pose data to determine the relative pose of the head-mounted display device relative to the vehicle. Optionally, the fusion calculation of the pose data may be performed using the EKF algorithm (Extended Kalman Filter) or the Gauss-Newton optimization method, or may be modeled as a non-linear least squares problem.

[0045] Such embodiments propose a new method for determining the relative pose. By taking an image of a user wearing a head-mounted display device through an image sensor provided on the auxiliary positioning device, predicting the pose data of the head-mounted display device relative to the auxiliary positioning device, and then performing a fusion detection on the first inertial data of the auxiliary positioning device, the second inertial data of the head-mounted display device, and the pose data. When the detection passes, the relative pose of the head-mounted display device relative to the vehicle is determined by a multi-sensor fusion method. It can solve the recognition conflict problem generated when multiple users use the auxiliary positioning device and the head-mounted display device simultaneously, improve the accuracy of the determined relative pose, and help improve the display effect of subsequent image rendering based on the relative pose.

[0046] Figure 2 Shows a scene diagram applicable to the extended reality system of some embodiments of the present disclosure. As Figure 2As shown, in this application scenario, the vehicle 210 can be a vehicle. The auxiliary positioning device 220 and the auxiliary positioning device 230 are fixed inside the vehicle 210. Both the auxiliary positioning device 220 and the auxiliary positioning device 230 are provided with a first inertial measurement unit (not shown in the figure). The image sensor 221 can be the image sensor of the auxiliary positioning device 220, and the image sensor of the auxiliary positioning device 230 is not shown. Both the head-mounted display device 240 and the head-mounted display device 250 are provided with a second inertial measurement unit (not shown in the figure). In Figure 2 The image sensor 221 and the auxiliary positioning device 220 are separate devices. It can be understood that the image sensor 221 and the auxiliary positioning device 220 can also be an integrated device.

[0047] Taking the auxiliary positioning device 220 as an example for illustration. When the wearers of the head-mounted display device 240 and the head-mounted display device 250 are inside the vehicle 210, the image sensor 221 can capture images of the wearers of the head-mounted display device 240 and the head-mounted display device 250. At the same time, the first inertial measurement unit of the auxiliary positioning device 220 can obtain the first inertial data of the auxiliary positioning device, and the second inertial measurement unit of the head-mounted display device 240 can obtain the second inertial data. After the device for determining the relative pose acquires the images, it can predict the poses of the head-mounted display device 240 and the head-mounted display device 250 relative to the auxiliary positioning device 220 based on the images respectively, and use the obtained two poses as the pose data of the target head-mounted display device. The first inertial data and the second inertial data are respectively fused with the poses of the head-mounted display device 240 and the head-mounted display device 250 for detection to determine whether the pose data includes available poses (i.e., the pose of the head-mounted display device 240 that matches the auxiliary positioning device 220). When the fusion detection passes, the pose data can be fused with the first inertial data and the second inertial data to determine the relative pose of the target head-mounted display device relative to the vehicle.

[0048] Exemplary method

[0049] Figure 3 Some embodiments of the present disclosure show a method for determining the relative pose, such as Figure 3 As shown, the method includes the following steps:

[0050] Step 310: Obtain the images collected by the image sensor when the user wears the target head-mounted display device, the first inertial data collected by the first inertial measurement unit, and the second inertial data collected by the second inertial measurement unit.

[0051] In such embodiments, the auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle. Among them, the auxiliary positioning device can be fixed on the vehicle, and the auxiliary positioning device can be provided with an image sensor and a first inertial measurement unit; the target head-mounted display device can be provided with a second inertial measurement unit.

[0052] It should be noted that the image sensor can be arranged on the body of the auxiliary positioning device. At this time, the auxiliary positioning device and the image sensor can be an integrated device. Or, the image sensor can also be arranged outside the body of the auxiliary positioning device. At this time, the auxiliary positioning device and the image sensor can be a split device. The image sensor and the auxiliary positioning device can be fixed inside the vehicle, and the image sensor is communicatively connected to the auxiliary positioning device.

[0053] When a user wearing the target head-mounted display device is inside the movable vehicle, the image sensor of the auxiliary positioning device can capture an image of the user. At the same time, the first inertial measurement unit and the second inertial measurement unit can respectively obtain the first inertial data of the auxiliary positioning device and the second inertial data of the head-mounted display device, such as three-axis attitude angular velocity and acceleration.

[0054] As an example, the first inertial measurement unit and the second inertial measurement unit can adopt an inertial measurement unit (IMU).

[0055] It should be noted that the image captured by the image sensor when the user wears the target head-mounted display device should at least include an image of the user's head area.

[0056] Step 320: Based on the image, determine the pose data of the target head-mounted display device relative to the auxiliary positioning device.

[0057] In such embodiments, the pose data of the target head-mounted display device relative to the auxiliary positioning device can be combined with the inertial data obtained in the subsequent steps to determine the relative pose of the target head-mounted display device relative to the vehicle.

[0058] As an example, a pre-trained pose recognition model (such as a convolutional neural network, a residual network, etc.) can be used to recognize the image to determine the pose of at least one head-mounted display device in the image relative to the auxiliary positioning device, and the obtained at least one pose is used as the pose data of the target head-mounted display device relative to the auxiliary positioning device.

[0059] Step 330: Perform data fusion on the first inertial data, the second inertial data, and the pose data and perform fusion detection.

[0060] In such embodiments, the fusion detection indicates the presence of an available pose in the pose data, that is, the pose of the target head-mounted display device that matches the auxiliary positioning device. The failure of the fusion detection indicates the absence of an available pose in the pose data, that is, none of the head-mounted display devices in the image match the auxiliary positioning device.

[0061] Here, the fusion detection can be performed in various ways. As an example, the difference between the first inertial data and the second inertial data can be determined first, and then the confidence level can be calculated based on the residual between the pose data and the difference, and then it can be determined whether the fusion detection is passed through the confidence level. Alternatively, for the optical marker positions displayed in the detected image and the predicted optical marker positions of the target auxiliary positioning device, the residual of the pixel difference between the two can be calculated to determine the confidence level, and then it can be determined whether the fusion detection is passed through the confidence level.

[0062] Step 340, when the fusion detection is passed, determine the relative pose of the head-mounted display device with respect to the vehicle based on the first inertial data, the second inertial data, and the pose data.

[0063] As an example, the pose data, the first inertial data, and the second inertial data can be fused by the EKF (Extended Kalman Filter) method, the Gauss-Newton optimization method, or by constructing a non-linear least squares problem to determine the relative pose of the head-mounted display device with respect to the vehicle.

[0064] Such embodiments propose a new method for determining the relative pose. An image of the user wearing the head-mounted display device is captured by an image sensor provided on the auxiliary positioning device, and the pose data of the head-mounted display device with respect to the auxiliary positioning device is predicted. Then, the first inertial data of the auxiliary positioning device, the second inertial data of the head-mounted display device, and the pose data are subjected to fusion detection. When the detection is passed, the relative pose of the head-mounted display device with respect to the vehicle can be determined by the multi-sensor data fusion method. Thus, the recognition conflict problem generated when multiple users use the auxiliary positioning device and the head-mounted display device simultaneously can be solved, the accuracy of the determined relative pose can be improved, and the processing effect of subsequent image rendering based on the relative pose can be improved.

[0065] Next, refer to Figure 4 , Figure 4 which shows a flowchart of the fusion detection in some embodiments of the method for determining the relative pose of the present disclosure. As shown in Figure 4 , the process includes the following steps:

[0066] Step 410, determine the difference between the first inertial data and the second inertial data.

[0067] Step 420: Determine the confidence level based on the residual between the pose data from the vision sensor and the difference value.

[0068] Step 430: When the determined confidence level is greater than or equal to a preset confidence level threshold, determine that the fusion detection passes.

[0069] In a specific example, the image collected by the image sensor may include 3 head-mounted display devices, and the pose data obtained through Step 320 may include 3 poses. Then, the differences between the first inertial data and the second inertial data and the residuals of the 3 poses are respectively determined, and the confidence levels corresponding to the 3 poses are determined. When there is one or more confidence levels not less than the confidence level threshold among the 3 confidence levels, it indicates that there are available poses among the 3 poses (i.e., the pose of the head-mounted display device relative to the target auxiliary positioning device), and then it can be determined that the fusion detection passes. When there is no confidence level greater than or equal to the confidence level threshold among the 3 confidence levels, it indicates that there are no available poses among the 3 poses, that is, none of the 3 poses is the pose of the target head-mounted display device relative to the auxiliary positioning device, and it is determined that the fusion detection fails.

[0070] In such embodiments, the confidence level can be determined through the difference of inertial data and the residual of pose data, and it can be determined whether the fusion detection passes according to the confidence level, and the inertial data is used to detect the image data collected by the image sensor to ensure the accuracy and reliability of the pose data.

[0071] In some alternative embodiments, when there are multiple head-mounted display devices in the image, the poses of the multiple head-mounted display devices relative to the auxiliary positioning device can be respectively determined to obtain multiple candidate poses.

[0072] When there are multiple head-mounted display devices in the vehicle interior at the same time, the image collected by the image sensor usually includes multiple head-mounted display devices. To prevent missing the pose of the target head-mounted display device, the poses of the multiple head-mounted display devices relative to the auxiliary positioning device can be respectively determined to obtain multiple candidate poses as the pose data of the target head-mounted display device.

[0073] Furthermore, the multiple candidate poses can be determined through the following steps: use a pre-trained recognition model to recognize the image to determine at least one user included in the image; for the user among the determined at least one user, in response to the user wearing a head-mounted display device, determine the head pose of the user as the candidate pose corresponding to the user to obtain multiple candidate poses.

[0074] In such embodiments, an identification model can be used to identify the head images of multiple users from an image, and the head pose of the user wearing the head-mounted display device is identified as the candidate pose of the user, so as to obtain multiple candidate poses, which can improve the operation efficiency and accuracy of calculating the candidate poses and avoid wasting operation resources.

[0075] Optionally, in response to the user not wearing the head-mounted display device, the tracking of the head pose of the user can be abandoned to avoid wasting operation resources.

[0076] The following refers to Figure 5 , Figure 5 shows a flowchart of some embodiments for determining the relative pose of the present disclosure. As Figure 5 shown, the process includes the following steps:

[0077] Step 510: Obtain the image collected by the image sensor when the user wears the target head-mounted display device, the first inertial data collected by the first inertial measurement unit, and the second inertial data collected by the second inertial measurement unit.

[0078] In such embodiments, the auxiliary positioning device and the target head-mounted display device can be in one-to-one correspondence, that is, each auxiliary positioning device can only perform data interaction with its corresponding target head-mounted display device. For example, the auxiliary positioning device can only obtain the data of the target head-mounted display device and / or send data to the target head-mounted display device, and cannot perform data interaction with other head-mounted display devices.

[0079] Step 520: When the image includes multiple head-mounted display devices, respectively determine the poses of the multiple head-mounted display devices relative to the auxiliary positioning device to obtain multiple candidate poses.

[0080] Step 530: Perform data fusion on the multiple candidate poses with the first inertial data and the second inertial data respectively to determine the confidence levels of the multiple candidate poses.

[0081] Step 540: When there is a confidence level greater than or equal to a preset first confidence threshold, determine that the fusion detection passes.

[0082] Step 550: When the fusion detection passes, determine the relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data.

[0083] In a specific example, there can be a total of 4 sets of auxiliary positioning devices and head-mounted display devices inside the vehicle, and there is a one-to-one correspondence between the auxiliary positioning devices and the head-mounted display devices. For any one set of auxiliary positioning device and head-mounted display device, the auxiliary positioning device can only receive the second inertial data of the head-mounted display device corresponding to it. In this example, there may be at most 4 head-mounted display devices in the image captured by the image sensor. Through step 520, 4 candidate poses can be obtained, and then the 4 candidate poses are respectively fused with the first inertial data and the second inertial data to obtain 4 confidence levels. When one or more of the 4 confidence levels are greater than or equal to the first confidence threshold, it can indicate that the 4 candidate poses include the pose of the head-mounted display device corresponding to the auxiliary positioning device. At this time, it can be determined that the fusion detection passes. Subsequently, the candidate pose with the highest confidence level can be selected from them and fused with the first inertial data and the second inertial data to determine the relative pose of the head-mounted display device corresponding to the auxiliary positioning device relative to the vehicle.

[0084] Figure 5 In the embodiment shown, when there are multiple sets of one-to-one corresponding auxiliary positioning devices and head-mounted display devices in the vehicle, multiple candidate poses can be respectively fused and detected with the first inertial data and the second inertial data to determine whether the candidate poses include the pose of the head-mounted display device corresponding to the auxiliary positioning device, determine the pose of the head-mounted display device corresponding to each auxiliary positioning device, and further determine the relative pose of the head-mounted display device corresponding to each auxiliary positioning device relative to the vehicle through data fusion, which can solve the identification conflict of the head-mounted display device when multiple sets of one-to-one corresponding auxiliary positioning devices and head-mounted display devices are used simultaneously, and help improve the accuracy of calculating the relative pose.

[0085] In Figure 5 In some optional implementation manners of the embodiment shown, before step 550, the method further includes: determining the candidate pose with the highest confidence level as the pose data of the target head-mounted display device relative to the auxiliary positioning device.

[0086] In such embodiments, the confidence level can represent the matching degree between the candidate pose and the auxiliary positioning device. The head-mounted display device corresponding to the candidate pose with the highest confidence level has the highest matching degree with the auxiliary positioning device, indicating that the probability of the corresponding relationship between the head-mounted display device and the auxiliary positioning device is the greatest. Determining the candidate pose with the highest confidence level as the pose data of the target head-mounted display device relative to the auxiliary positioning device can more accurately identify the pose data of the target head-mounted display device corresponding to the auxiliary positioning device from the image. Subsequently, fusing the pose data with the first inertial data and the second inertial data to determine the relative pose of the target head-mounted display device relative to the vehicle further improves the calculation accuracy of the relative pose.

[0087] In Figure 5 In some alternative embodiments of the illustrated embodiment, the method further includes: if the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold, abandoning the tracking of the head pose of the user.

[0088] In such embodiments, the smaller the confidence of the candidate pose corresponding to the user, the smaller the probability that there is a corresponding relationship between the head-mounted display device worn by the user and the auxiliary positioning device. When the confidence of the candidate pose corresponding to the user is less than the preset second confidence threshold, it means that the value of the candidate pose corresponding to the user in the subsequent steps can be ignored. At this time, the tracking of the head pose of the user can be abandoned to avoid wasting computing resources.

[0089] The following refers to Figure 6 , Figure 6 which shows a flowchart of some embodiments of the method for determining the relative pose of the present disclosure. As Figure 6 shown, the process includes the following steps:

[0090] Step 610, obtain the image collected by the image sensor when the user wears the target head-mounted display device, the first inertial data collected by the first inertial measurement unit, and the second inertial data collected by the second inertial measurement unit.

[0091] In such embodiments, the auxiliary positioning device corresponds to multiple target head-mounted devices; at this time, the second inertial data should include the inertial data collected by the second inertial measurement units of the multiple target head-mounted devices respectively.

[0092] Optionally, the second inertial data can be obtained in the following manner: obtain the inertial data collected by the second inertial measurement units corresponding to the multiple head-mounted display devices respectively to obtain multiple candidate inertial data as the second inertial data.

[0093] The auxiliary positioning device corresponding to multiple target head-mounted devices means that the auxiliary positioning device can interact with multiple head-mounted display devices simultaneously. For example, it can obtain the candidate inertial data of multiple head-mounted display devices and determine the relative poses of multiple head-mounted display devices with respect to the vehicle respectively.

[0094] It can be understood that after the auxiliary positioning device receives the candidate inertial data of multiple head-mounted display devices, it is still unable to determine the corresponding relationship between the candidate inertial data and the poses of the multiple head-mounted display devices, and thus cannot determine the relative poses of the multiple head-mounted display devices with respect to the vehicle through data fusion.

[0095] Step 620, when multiple head-mounted display devices are included in the image, determine the poses of the multiple head-mounted display devices with respect to the auxiliary positioning device respectively to obtain multiple candidate poses.

[0096] Step 630: Fuse the first inertial data with each of the multiple candidate inertial data and each of the multiple candidate poses respectively, and determine multiple candidate confidence levels corresponding to each candidate pose among the multiple candidate poses.

[0097] In such embodiments, permutations and combinations can be performed on the multiple candidate poses and the multiple candidate inertial data, and then each combination is fused and detected with the first inertial data respectively to obtain the candidate confidence level corresponding to each combination. For example, assume that the candidate inertial data obtained in step 610 includes candidate inertial data a and candidate inertial data b, and the candidate poses obtained in step 620 include candidate pose 1 and candidate pose 2. Then the following combination methods can be obtained: (a, 1), (a, 2), (b, 1), and (b, 2). After fusing each combination with the first inertial data respectively, 4 corresponding candidate confidence levels can be obtained: A, B, C, and D. Among them, the candidate confidence levels corresponding to candidate pose 1 are A and C, and the candidate confidence levels corresponding to candidate pose 2 are B and D.

[0098] In such embodiments, the candidate confidence level can characterize the matching degree between the candidate pose and the second inertial data. For any group of candidate pose and candidate inertial data, if the candidate inertial data and the candidate pose belong to the same head-mounted display device, then the matching degree between the candidate pose and the candidate inertial data is higher, and correspondingly, the candidate confidence level corresponding to this combination is greater. Conversely, if the candidate inertial data and the candidate pose belong to different head-mounted display devices, then the matching degree between the candidate pose and the candidate inertial data is lower, and correspondingly, the candidate confidence level corresponding to this combination is smaller.

[0099] Step 640: When there is a candidate confidence level greater than or equal to a preset third confidence threshold, determine that the fusion detection passes.

[0100] In such embodiments, when there is a candidate confidence level greater than or equal to a preset third confidence threshold, it indicates that the effective combinations of the candidate pose and the candidate inertial data (i.e., the combinations composed of the candidate pose and the candidate inertial data of the same head-mounted display device) are covered among the multiple combination methods obtained during the fusion detection in step 630.

[0101] Step 650: When the fusion detection passes, determine the relative pose of the target head-mounted display device with respect to the vehicle based on the first inertial data, the second inertial data, and the pose data.

[0102] In a specific example, the auxiliary positioning device corresponds to 5 head-mounted display devices simultaneously. Correspondingly, the second inertial data includes 5 candidate inertial data, and at the same time, 5 candidate poses can be obtained in step 620. After that, the 5 candidate poses and the 5 candidate inertial data can be combined in permutations and combinations to obtain 25 combinations, and then the 25 combinations are respectively fused and detected with the first inertial data to obtain 25 candidate confidence levels. If there is a candidate confidence level greater than or equal to a preset third confidence threshold among the 25 candidate confidence levels, the fusion detection passes. At this time, based on the 25 candidate confidence levels, the highest candidate confidence level corresponding to each candidate pose can be selected, and the candidate pose and candidate inertial data corresponding to this candidate confidence level are determined as the corresponding relationship. In this way, the candidate poses corresponding to multiple candidate inertial data can be determined respectively, and then the first inertial data is fused with the candidate inertial data and candidate poses with corresponding relationships, and the relative pose of the head-mounted display device corresponding to the candidate inertial data relative to the vehicle can be determined.

[0103] Figure 6 In the embodiment shown, the first inertial data is respectively fused with each candidate inertial data among the multiple candidate inertial data and each candidate pose among the multiple candidate poses to determine multiple candidate confidence levels corresponding to each candidate pose among the multiple candidate poses. The candidate confidence level is used to characterize the matching degree between the candidate pose and the candidate inertial data, so as to determine the candidate pose and candidate inertial data corresponding to each head-mounted display device, which can solve the recognition conflict problem when the auxiliary positioning device corresponds to multiple head-mounted display devices simultaneously and helps to improve the calculation accuracy of the relative pose.

[0104] In Figure 6 In some optional implementation manners of the embodiment shown, the above step 650 includes: determining the head-mounted display device corresponding to each candidate pose among the multiple candidate poses based on the candidate inertial data corresponding to the highest candidate confidence level of each candidate pose among the multiple candidate poses; respectively determining the relative pose of each head-mounted display device among the multiple head-mounted display devices relative to the vehicle based on the first inertial data, the candidate pose corresponding to each head-mounted display device among the multiple head-mounted display devices, and the candidate inertial data corresponding to the candidate pose. In this case, each head-mounted display device among the above multiple head-mounted display devices can be a target head-mounted display device.

[0105] Continuing with the example in step 630, assume that the values of the four candidate confidence levels A, B, C, and D are 0.7, 0.8, 0.9, and 0.6 respectively. Then the highest candidate confidence level corresponding to candidate pose 1 is C, and the corresponding combination is (b, 1); the highest candidate confidence level corresponding to candidate pose 2 is B, and the corresponding combination is (a, 2). Correspondingly, the head-mounted display device corresponding to candidate pose 1 is the head-mounted display device corresponding to candidate inertial data b, and the head-mounted display device corresponding to candidate pose 2 is the head-mounted display device corresponding to candidate inertial data a.

[0106] In such an embodiment, the correspondence between the candidate pose and the candidate inertial data can be determined through the candidate confidence level, ensuring the correspondence between the pose data and the inertial data during data fusion, which helps to improve the calculation accuracy of the relative pose.

[0107] In Figure 6 some alternative embodiments of the illustrated embodiment, the method further includes: if multiple candidate confidence levels corresponding to the user are all less than a preset fourth confidence threshold, giving up tracking the head pose of the user.

[0108] In such an embodiment, if multiple candidate confidence levels corresponding to the user are all less than a preset fourth confidence threshold, it indicates that the user is an invalid user, that is, the head-mounted display device worn by the user does not belong to the head-mounted display device corresponding to the auxiliary positioning device. At this time, the head pose of the user can be given up tracking to avoid introducing interference data and wasting computing resources.

[0109] In some embodiments, after obtaining the relative pose, the method further includes: sending the relative pose to the target head-mounted display device so that the target head-mounted display device performs rendering processing based on the relative pose.

[0110] In such an embodiment, the auxiliary positioning device can send the calculated relative pose to the corresponding head-mounted display device so that the target head-mounted display device performs rendering processing based on the relative pose, and the quality of the rendering processing can be improved through the high-precision relative pose.

[0111] Exemplary device

[0112] Next, refer to Figure 7 , Figure 7FIG. 0 shows a schematic structural diagram of an embodiment of the apparatus for determining relative pose according to the present disclosure. The auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle. The auxiliary positioning device is fixedly connected to the vehicle and is provided with an image sensor and a first inertial measurement unit. The target head-mounted display device is provided with a second inertial measurement unit. The apparatus includes: a data acquisition unit 710 configured to acquire an image collected by the image sensor when a user wears the target head-mounted display device, first inertial data collected by the first inertial measurement unit, and second inertial data collected by the second inertial measurement unit; a pose prediction unit 720 configured to determine pose data of the target head-mounted display device relative to the auxiliary positioning device based on the image; a data fusion unit 730 configured to perform data fusion on the first inertial data, the second inertial data, and the pose data and perform fusion detection; and a pose determination unit 740 configured to determine the relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data when the fusion detection passes.

[0113] In one embodiment, the fusion detection unit 730 includes: a first calculation module configured to determine a difference between the first inertial data and the second inertial data; a second calculation module configured to determine a confidence level based on a residual between the pose data and the difference; and a detection module configured to determine that the fusion detection passes when the determined confidence level is greater than or equal to a preset confidence level threshold.

[0114] In one embodiment, the pose determination unit 740 is further configured to: when multiple head-mounted display devices are included in the image, respectively determine the poses of the multiple head-mounted display devices relative to the auxiliary positioning device to obtain multiple candidate poses.

[0115] In one embodiment, the pose determination unit 740 is further configured to include: identifying the image by using a pre-trained recognition model to determine at least one user included in the image; for the user among the determined at least one user, in response to the user wearing a head-mounted display device, determining the head pose of the user as the candidate pose corresponding to the user to obtain multiple candidate poses.

[0116] In one embodiment, the auxiliary positioning device and the target head-mounted display device are in one-to-one correspondence; the data fusion unit further includes: a first fusion module configured to perform data fusion on the multiple candidate poses respectively with the first inertial data and the second inertial data to determine the confidence levels of the multiple candidate poses; and a first judgment module configured to determine that the fusion detection passes when there is a confidence level greater than or equal to a preset first confidence level threshold.

[0117] In one embodiment, the device further includes a data determination unit configured to determine the candidate pose with the highest confidence as the pose data of the target head-mounted display device relative to the auxiliary positioning device.

[0118] In one embodiment, the device further includes a first tracking module configured to abandon tracking the head pose of the user if the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold.

[0119] In one embodiment, the auxiliary positioning device corresponds to a plurality of target head-mounted devices; the second inertial data is obtained by: acquiring the inertial data respectively collected by a plurality of second inertial measurement units corresponding to the plurality of head-mounted display devices to obtain a plurality of candidate inertial data as the second inertial data.

[0120] In one embodiment, the data fusion unit 730 includes: a second fusion module configured to fuse the first inertial data with each of the plurality of candidate inertial data and each of the plurality of candidate poses among the plurality of candidate inertial data to determine a plurality of candidate confidences corresponding to each candidate pose among the plurality of candidate poses; a second judgment module configured to determine that the fusion detection passes when there is a candidate confidence greater than or equal to a preset third confidence threshold.

[0121] In one embodiment, the pose determination unit 740 further includes: a determination module configured to determine the head-mounted display device corresponding to each candidate pose among the plurality of candidate poses based on the candidate inertial data corresponding to the highest candidate confidence of each candidate pose among the plurality of candidate poses; a pose calculation module configured to respectively determine the relative pose of each head-mounted display device among the plurality of head-mounted display devices relative to the vehicle based on the first inertial data, the candidate pose corresponding to each head-mounted display device among the plurality of head-mounted display devices, and the candidate inertial data corresponding to the candidate pose.

[0122] In one embodiment, the device further includes a second tracking unit configured to abandon tracking the head pose of the user if the plurality of candidate confidences corresponding to the user are all less than a preset fourth confidence threshold.

[0123] In one embodiment, the device further includes a sending unit configured to send the relative pose to the target head-mounted display device so that the target head-mounted display device performs rendering processing based on the relative pose.

[0124] Exemplary electronic device

[0125] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0126] A memory for storing a computer program;

[0127] A processor for executing the computer program stored in the memory, and when the computer program is executed, implementing the method for determining a relative pose according to any one of the above embodiments of the present disclosure.

[0128] Figure 8 This is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. Below, reference Figure 8 is made to describe the electronic device according to an embodiment of the present disclosure. As Figure 8 shown, the electronic device includes one or more processors and a memory.

[0129] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0130] The memory may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement the method for determining a relative pose according to various embodiments of the present disclosure described above and / or other desired functions.

[0131] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0132] In addition, the input device may further include, for example, a keyboard, a mouse, and so on.

[0133] The output device may output various information to the outside, including the determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0134] Of course, for simplicity, Figure 8 only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device may further include any other appropriate components.

[0135] In addition to the above methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps in the method for determining a relative pose according to various embodiments of the present disclosure described in the foregoing part of this specification.

[0136] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on a user computing device, partially on a user device, executed as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0137] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored, and the computer program instructions, when run by a processor, cause the processor to execute the steps in the method for determining a relative pose according to various embodiments of the present disclosure described in the foregoing part of this specification.

[0138] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0139] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program may be stored in a computer-readable storage medium, and when the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other various media that can store program code.

[0140] The basic principles of the present disclosure have been described in connection with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. Additionally, the specific details disclosed above are only for illustrative and facilitating understanding purposes, rather than limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0141] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple. For related parts, reference can be made to the partial description of the method embodiments.

[0142] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with each other, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with each other.

[0143] The methods and apparatuses of the present disclosure can be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is only for illustration purposes. The steps of the method of the present disclosure are not limited to the above specifically described order, unless otherwise specifically stated in other ways. Additionally, in some embodiments, the present disclosure can also be implemented as a program recorded on a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers the recording medium storing the program for executing the method according to the present disclosure.

[0144] It should also be noted that in the apparatuses, equipment, and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0145] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0146] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. A method for determining a relative pose, an auxiliary positioning device and a corresponding target head-mounted display device are located inside a movable vehicle. The auxiliary positioning device is fixedly connected to the vehicle and is provided with an image sensor and a first inertial measurement unit; The target head-mounted display device is provided with a second inertial measurement unit, and the method includes: Obtaining an image collected by the image sensor when a user wears the target head-mounted display device, first inertial data collected by the first inertial measurement unit, and second inertial data collected by the second inertial measurement unit; Based on the image, determining pose data of the target head-mounted display device relative to the auxiliary positioning device, where the pose data includes the pose of at least one head-mounted display device in the image relative to the auxiliary positioning device; Performing data fusion on the first inertial data, the second inertial data, and the pose data and performing fusion detection, where the fusion detection indicates the presence of the pose of the target head-mounted display device matching the auxiliary positioning device in the pose data corresponding to the at least one head-mounted display device, and determining that the fusion detection passes when it is determined through the fusion result of the data fusion that the confidence level of the presence of the pose data is greater than or equal to a preset confidence threshold; When the fusion detection passes, determining the relative pose of the target head-mounted display device relative to the vehicle based on the first inertial data, the second inertial data, and the pose data.

2. The method according to claim 1, wherein, The performing data fusion on the first inertial data, the second inertial data, and the pose data and performing fusion detection includes: Determining the difference between the first inertial data and the second inertial data; Determining a confidence level based on the residual between the pose data and the difference; Judging that the fusion detection passes when the determined confidence level is greater than or equal to a preset confidence threshold.

3. The method according to claim 1, wherein, The determining the pose data of the target head-mounted display device relative to the auxiliary positioning device based on the image includes: When the image includes multiple head-mounted display devices, respectively determining the poses of the multiple head-mounted display devices relative to the auxiliary positioning device to obtain multiple candidate poses.

4. The method according to claim 3, wherein The when the image includes multiple head-mounted display devices, respectively determining the poses of the multiple head-mounted display devices relative to the auxiliary positioning device to obtain multiple candidate poses includes: Using a pre-trained recognition model to recognize the image to determine at least one user included in the image; For the user among the determined at least one user, in response to the user wearing a head-mounted display device, determining the head pose of the user as the candidate pose corresponding to the user to obtain the multiple candidate poses.

5. The method according to claim 3, wherein, The auxiliary positioning device corresponds to the target head-mounted display device one by one; The performing data fusion on the first inertial data, the second inertial data, and the pose data and performing fusion detection includes: Performing data fusion on the multiple candidate poses with the first inertial data and the second inertial data respectively to determine the confidence levels of the multiple candidate poses; Determining that the fusion detection passes when there is a confidence level greater than or equal to a preset first confidence threshold.

6. The method according to claim 5, wherein Before determining the relative pose of the target head-mounted display device with respect to the vehicle based on the first inertial data, the second inertial data, and the pose data, the method further includes: Determine the pose data of the target head-mounted display device with respect to the auxiliary positioning device by taking the candidate pose with the highest confidence.

7. The method according to claim 5, wherein It further includes: If the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold, give up tracking the head pose of the user.

8. The method according to claim 4, wherein The auxiliary positioning device corresponds to a plurality of the target head-mounted display devices; The second inertial data is obtained by the following method: Obtain inertial data respectively collected by a plurality of the second inertial measurement units corresponding to the plurality of head-mounted display devices to obtain a plurality of candidate inertial data, which are used as the second inertial data.

9. The method according to claim 8, wherein, Performing data fusion and fusion detection on the first inertial data, the second inertial data, and the pose data includes: Fuse the first inertial data with each of the plurality of candidate inertial data and each of the plurality of candidate poses among the plurality of candidate inertial data to determine a plurality of candidate confidences corresponding to each candidate pose among the plurality of candidate poses; When there is a candidate confidence greater than or equal to a preset third confidence threshold, determine that the fusion detection passes.

10. The method according to claim 9, wherein, Determining the relative pose of the target head-mounted display device with respect to the vehicle based on the first inertial data, the second inertial data, and the pose data includes: Based on the candidate inertial data corresponding to the highest candidate confidence of each candidate pose among the plurality of candidate poses, determine the head-mounted display device corresponding to each candidate pose among the plurality of candidate poses; Based on the first inertial data, the candidate poses corresponding to each head-mounted display device among the plurality of head-mounted display devices, and the candidate inertial data corresponding to the candidate poses, respectively determine the relative pose of each head-mounted display device among the plurality of head-mounted display devices with respect to the vehicle.

11. The method according to claim 10, wherein It further includes: If all the plurality of candidate confidences corresponding to the user are less than a preset fourth confidence threshold, give up tracking the head pose of the user.

12. The method according to any one of claims 1 to 11, wherein, It further includes: Send the relative pose to the target head-mounted display device so that the target head-mounted display device performs rendering processing based on the relative pose.

13. A device for determining relative pose, an auxiliary positioning device and its corresponding target head-mounted display device are located inside a movable vehicle. The auxiliary positioning device is fixedly connected to the vehicle and is provided with an image sensor and a first inertial measurement unit; The target head-mounted display device is provided with a second inertial measurement unit, and the device includes: A data acquisition unit configured to acquire an image of the user wearing the target head-mounted display device collected by the image sensor, first inertial data collected by the first inertial measurement unit, and second inertial data collected by the second inertial measurement unit; A pose prediction unit configured to determine the pose data of the target head-mounted display device with respect to the auxiliary positioning device based on the image, where the pose data includes the pose of at least one head-mounted display device in the image with respect to the auxiliary positioning device; A data fusion unit, configured to perform data fusion on the first inertial data, the second inertial data, and the pose data and perform fusion detection, wherein the fusion detection indicates the presence of the pose of the target head-mounted display device that matches the auxiliary positioning device in the pose data corresponding to the at least one head-mounted display device, and determines that the fusion detection passes when it is determined through the fusion result of the data fusion that the confidence level of the presence of the pose data is greater than or equal to a preset confidence level threshold; A pose determination unit, configured to, when the fusion detection passes, determine the relative pose of the target head-mounted display device with respect to the vehicle based on the first inertial data, the second inertial data, and the pose data.

14. An extended reality system, comprising an auxiliary positioning device, a head-mounted display device, and a device for determining a relative pose, wherein the auxiliary positioning device is fixedly connected to a movable vehicle and is provided with an image sensor and a first inertial measurement unit, and the head-mounted display device is provided with a second inertial measurement unit; When the head-mounted display device and the auxiliary positioning device are inside the moving vehicle, the device for determining a relative pose determines the relative pose of the head-mounted display device with respect to the vehicle by the method according to any one of claims 1 to 12, so that the head-mounted display device generates a projection image based on the relative pose.

15. An electronic device, comprising: A memory, configured to store a computer program; A processor, configured to execute the computer program stored in the memory, and when the computer program is executed, implement the method according to any one of claims 1 to 12 above.

16. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implement the method according to any one of claims 1 to 12 above.

Images