Method and device for determining a real object pose and storage medium

Abstract

The application discloses a real object pose determination method and device and a storage medium, relates to the virtual reality technical field, discloses a real object pose determination method, and is applied to a virtual reality device, and comprises the following steps: acquiring a positioning image collected by an image collection device, and determining pixel coordinates of a two-dimensional Marker point in the positioning image; determining a relative pose between a three-dimensional Marker and the image collection device according to the pixel coordinates; and determining a target pose of an entity corresponding to the three-dimensional Marker point in a world coordinate system according to the relative pose and calibration data of the image collection device. According to the application, the relative pose between the three-dimensional Marker and the image collection device is determined, the target pose of the three-dimensional Marker point in the world coordinate system is determined by using the calibration data, the accuracy of pose coordination between a virtual object and a real object in the virtual reality device is improved, and the alignment cost is reduced without relying on external base station construction.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to methods, apparatus and storage media for determining the pose of physical objects. Background Technology

[0002] When a Virtual Reality (VR) device is powered on, it creates a world coordinate system with the center of the Inertial Measurement Unit (IMU) as the origin to describe the pose of the output content. In related technologies, to ensure the pose coordination between virtual objects output by the VR device and real objects, the pose description information of the virtual and real objects is generally aligned using pre-built external base stations. However, this method suffers from high alignment costs due to the need for pre-built external base stations.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus and storage medium for determining the pose of a physical object, which aims to solve the technical problem of fusion and alignment of the poses of virtual objects and real objects.

[0005] To achieve the above objectives, this application proposes a method for determining the pose of a physical object, applied to a virtual reality device, wherein the virtual reality device is equipped with an image acquisition device, and the method includes:

[0006] The positioning image acquired by the image acquisition device is obtained, and the pixel coordinates of the two-dimensional marker points in the positioning image are determined.

[0007] The relative pose between the 3D Marker and the image acquisition device is determined based on the pixel coordinates.

[0008] Based on the relative pose and the calibration data of the image acquisition device, the target pose of the entity corresponding to the 3D Marker point in the world coordinate system is determined.

[0009] In one embodiment, the step of determining the relative pose between the 3D Marker and the image acquisition device based on the pixel coordinates includes:

[0010] The order of selectable spots corresponding to the two-dimensional marker points in the positioning image is determined based on the pixel coordinates.

[0011] Set the optional spot order with the smallest projection error as the target spot order;

[0012] Two-dimensional-three-dimensional point pairs are constructed according to the order of the target spots, and the relative pose is determined based on the two-dimensional-three-dimensional point pairs.

[0013] In one embodiment, before the step of setting the optional spot order with the smallest projection error as the target spot order, the method further includes:

[0014] Determine the rotation vector and translation vector corresponding to each of the optional spot sequences;

[0015] The projection error corresponding to the optional spot order is determined based on the rotation vector and the translation vector.

[0016] In one embodiment, the step of constructing two-dimensional-three-dimensional point pairs according to the target spot order and determining the relative pose based on the two-dimensional-three-dimensional point pairs includes:

[0017] Construct the two-dimensional-three-dimensional point pairs according to the order of the target spots;

[0018] Determine the control points and construct a local coordinate system based on them;

[0019] A system of equations is constructed based on the two-dimensional-three-dimensional point pairs and the local coordinate system, and the relative pose is solved based on the system of equations.

[0020] In one embodiment, the image acquisition device is a binocular camera, and the step of determining the relative pose between the 3D marker and the image acquisition device based on the pixel coordinates includes:

[0021] Obtain the binocular baseline and binocular extrinsic parameters corresponding to the binocular camera, and determine the initial relative pose based on the positioning image acquired by the sub-camera of the binocular camera;

[0022] Based on the binocular baseline and the binocular extrinsic parameters, establish the relationship equations for the projection positions of the three-dimensional marker points in the binocular camera image;

[0023] The relative pose is determined based on the relationship equation and the initial relative pose.

[0024] In one embodiment, the step of determining the target pose of the entity corresponding to the 3D Marker point in the world coordinate system based on the relative pose and the calibration data of the image acquisition device further includes:

[0025] The device pose of the virtual reality device is determined based on the inertial data collected by the inertial measurement unit.

[0026] The image acquisition device pose is determined based on the calibration data and the device pose.

[0027] The target pose is determined based on the pose of the acquisition device and the relative pose.

[0028] In one embodiment, the method further includes:

[0029] Obtain the display parameters of the virtual object to be output, and render the virtual object according to the display parameters;

[0030] The display pose of the virtual object is determined based on the target pose.

[0031] In one embodiment, the step of acquiring the positioning image captured by the image acquisition device and determining the pixel coordinates of the two-dimensional marker points in the positioning image includes:

[0032] The two-dimensional marker points of the positioning image are identified according to the image recognition algorithm, and the edge contours of the two-dimensional marker points are determined.

[0033] Calculate the center point of the edge contour and determine the pixel coordinates of the corresponding two-dimensional marker point.

[0034] In addition, to achieve the above objectives, this application also proposes a virtual reality device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for determining the pose of a physical object as described in any one of claims 1 to 8.

[0035] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method for determining the physical pose as described above.

[0036] This application provides a method, apparatus, and storage medium for determining the pose of a physical object. It involves acquiring a positioning image captured by an image acquisition device and determining the pixel coordinates of a two-dimensional marker point in the positioning image; determining the relative pose between a three-dimensional marker and the image acquisition device based on the pixel coordinates; and determining the target pose of the entity corresponding to the three-dimensional marker point in the world coordinate system based on the relative pose and the calibration data of the image acquisition device. This application determines the target pose of the three-dimensional marker point in the world coordinate system based on the calibration data of the three-dimensional marker point and the camera, improving the accuracy of pose coordination between virtual and real objects in virtual reality devices. Furthermore, it eliminates the need for external base station construction, reducing the alignment cost between virtual and real scenes. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for determining the physical pose in the embodiments of this application;

[0040] Figure 2 This is a flowchart illustrating the first embodiment of the method for determining the physical pose of the object according to this application;

[0041] Figure 3 This is a first flowchart illustrating the second embodiment of the method for determining the physical pose of the object in this application;

[0042] Figure 4 This is a second flowchart illustrating the second embodiment of the method for determining the physical pose of the object in this application;

[0043] Figure 5 This is a schematic diagram illustrating a specific application scenario of the second embodiment of the method for determining the physical pose of the object in this application;

[0044] Figure 6 This is a schematic diagram of the algorithm flow of the second embodiment of the method for determining the pose of a physical object in this application.

[0045] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0047] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0048] In Virtual Reality (VR) systems, achieving alignment between the virtual and real worlds relies primarily on precise tracking and positioning devices, namely head-mounted displays (HMDs) and external trackers (often called lighthouses). The working principle of a VR system is to use infrared light signals emitted by the lighthouses to create a known light pattern in space. These signals are then captured by photosensitive sensors on the HMD. By measuring the angles and intensities of signals from multiple lighthouses, the VR device can continuously calculate its precise position and orientation in three-dimensional space.

[0049] Once objects and scenes in the virtual world are rendered and displayed on the HMD by a computer, the position and posture information of the VR device are captured in real time and transmitted back to the computer system when the user moves in the real world. Based on this data, the computer will adjust the perspective and scene of the virtual world accordingly, so that the movement of virtual objects and the real world are highly consistent. This real-time alignment of virtual and real worlds achieves seamless matching between the virtual environment and the user's actions, greatly enhancing the user's sense of immersion.

[0050] To maximize system performance, lighthouses should be strategically positioned around user activity areas to avoid signal blind spots. Similarly, avoid exposing lighthouses to direct sunlight or other strong light sources, as this may cause signal interference. While using the VR device, users should ensure that their own or others' limbs or other objects do not obstruct the infrared light signals emitted by the lighthouses, so as not to affect tracking quality.

[0051] When a virtual reality (VR) device is powered on, it creates a world coordinate system with the center of the Inertial Measurement Unit (IMU) as the origin to describe the pose of the output content. In related technologies, to ensure pose coordination between virtual objects output by the VR device and real objects, the pose description information of the virtual and real objects is typically aligned using pre-built external base stations. However, this method suffers from high alignment costs due to the need for pre-built external base stations.

[0052] In view of the above problems, this application proposes a method for determining the pose of a physical object. This method determines the target pose of the 3D marker point in the world coordinate system by determining the relative pose between the 3D marker and the image acquisition device and using calibration data. This improves the accuracy of pose coordination between virtual objects and real objects in virtual reality devices. Furthermore, this method does not require the construction of external base stations, thus reducing alignment costs.

[0053] To more clearly illustrate this technical solution, this application uses a VR headset as the implementation subject. Please refer to [link / reference]. Figure 1 , Figure 1This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0054] like Figure 1 As shown, a virtual reality device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the virtual reality device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the virtual reality device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show virtual reality devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0055] Those skilled in the art will understand that, although Figure 1 The diagram shows various structural devices, but this does not constitute a limitation on electronic devices; alternatively, systems with more or fewer components may be implemented.

[0056] This application provides a method for determining the pose of a physical object. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the method for determining the physical pose of the object according to this application.

[0057] In this embodiment, the method for determining the pose of the physical object includes the following steps:

[0058] Step S10: Obtain the positioning image acquired by the image acquisition device, and determine the pixel coordinates of the two-dimensional marker points in the positioning image.

[0059] It should be noted that marker points are typically deployed on one or more objects or scenes that require precise positioning and measurement. These marker points form rigid bodies in a non-linear and asymmetrical manner, and are arranged and placed on the real object. There are many reasonable arrangements of these rigid bodies.

[0060] It should be explained that since the three points are not collinear in three-dimensional space, they can determine a unique plane. Therefore, at least three marker points are needed to define the start point, end point, and midpoint of the rigid body.

[0061] Understandably, in order to ensure that the image acquisition device can clearly identify each marker point and reduce interference between adjacent marker points, the spacing between marker points needs to be set to at least twice the marker diameter.

[0062] It should be noted that marker points are typically circular or square and made of infrared reflective material. This material reflects infrared light, creating prominent bright spots in the acquired image for easy identification and localization. When the user puts on the HMD and powers it on, the world coordinate system is reconstructed with the center of the IMU as the origin. The system then prompts the user to enter the calibration process, triggering the image acquisition device to acquire images.

[0063] Optionally, marker points can be high-contrast patterns, such as black and white checks or specific shapes, to be displayed in a specific color in the image captured by the camera. Alternatively, irregular arrays of LEDs or other light sources can be used as marker points. These light sources can be programmed to operate at specific flashing frequencies and / or color patterns, making them easily identifiable in the image captured by the camera, and the pixel coordinates of each light source can be determined through image analysis techniques.

[0064] Optionally, five objects with marker points are arranged into a rigid body according to a specific shape and placed on a real-world object. Then, the user enters the correction process, triggering the image acquisition device to acquire images of the rigid body.

[0065] It is important to clarify that as the number of marker points increases, the system can obtain more positional constraint information, which helps to more accurately determine the position and orientation of the rigid body. Furthermore, more marker points can provide redundancy information; when some marker points cannot be captured by the system due to occlusion or recognition problems, other marker points can still maintain the system's positioning accuracy. However, too many marker points may increase the system's computational load. Therefore, in practical applications, an appropriate number of marker points can be selected according to the specific situation and requirements to achieve the best balance. This embodiment does not impose any limitation on the number of marker points.

[0066] In one optional implementation, after acquiring the positioning image, two-dimensional marker points in the positioning image can be identified first using an image recognition algorithm, and the edge contours of the two-dimensional marker points can be determined. Then, the center point of the edge contours can be calculated to determine the pixel coordinates of the corresponding two-dimensional marker points. Here, the edge contour refers to the specific shape boundary of the marker point in the image. By identifying the edge contours, the boundary of the marker point can be located.

[0067] Optionally, after acquiring the localization image, it is first converted to a grayscale image and noise reduction is performed simultaneously. Then, the horizontal and vertical gradients of each pixel are calculated using two 3x3 kernels to obtain the gradient information for each pixel. The total gradient magnitude and direction are calculated by combining the gradient information to identify potential edge points. Then, through non-maximum suppression edge refinement and double thresholding, pixels with the largest gradient magnitude are retained, and pixels with gradient magnitudes higher than the high threshold and lower than the low threshold are suppressed. Pixels between the high and low thresholds are considered edges if they are connected to strong edge points. After obtaining a clear edge contour through the above steps, the edge contour of the marker point is determined according to the contour tracking algorithm, and the center point of the edge contour is calculated to determine the pixel coordinates of the marker point in the image.

[0068] To better understand the solution presented in this example, further explanation will be provided in conjunction with specific application scenarios.

[0069] In this example, the augmented reality (AR) game application requires placing virtual objects at specific locations in the real world. Therefore, multiple marker points need to be placed in the real environment. These marker points can be effectively identified in the positioning images captured by the camera and used as references to determine the position of the virtual objects. Five objects with marker points can be arranged into a rigid body according to a specific shape and placed on objects in the real environment. When the user moves within the game area while wearing the HMD (Handheld Device), the camera on the AR headset will capture the corresponding marker points and collect positioning images.

[0070] The acquired localization image is converted to grayscale and noise reduction is performed. Then, two 3x3 kernels are used to calculate the horizontal and vertical gradients of each pixel in the image, thus obtaining the gradient information for each pixel. Combining this gradient information, the magnitude and direction of the total gradient are further calculated. A non-maximum suppression algorithm is then used to retain the pixels with the largest gradient magnitude, and a double thresholding process is employed to filter out true edge points. Pixels with gradient magnitudes higher than the high threshold are accepted as strong edges, while those lower than the low threshold are suppressed. Through these steps, clear edge contours of five marker points in the localization image are obtained. The center points of the corresponding marker points are then calculated to determine the pixel coordinates of the five markers in the localization image.

[0071] Optionally, a template image corresponding to the marker points of the rigid body can be created first. This template image contains the precise shape and size of the marker points. Then, the template image is matched against the positioning image acquired by the image acquisition device. Starting from the top left corner of the positioning image, the template image is compared with regions of the same size in the positioning image. This comparison is performed by calculating the sum of the squares of the differences in pixel values ​​between the two regions. The template image is then moved and compared with the next region in the positioning image, continuing until the entire positioning image has been traversed. The coordinates of the region with the highest matching degree are obtained, which are the pixel coordinates of the marker in the positioning image.

[0072] Optionally, a large image dataset containing various types of marker points is collected and manually labeled to indicate the pixel coordinates of each marker point. Then, a convolutional neural network model is trained using the collected image dataset. During training, the model learns and recognizes the features of each type of marker point, including shape and pattern. By adjusting parameters, the model's accuracy in recognizing marker point pixel coordinates is gradually improved. The trained model is then integrated into the AR system. When an AR device equipped with a camera captures a location image of the marker points, the system uses the trained convolutional neural network model in real time to identify the marker points in the location image and output the corresponding pixel coordinates of the marker points.

[0073] Step S20: Determine the relative pose between the 3D Marker and the image acquisition device based on the pixel coordinates.

[0074] It should be noted that pixel coordinates are the horizontal and vertical positions of each pixel relative to the top left corner of a digital image, and are usually represented using a two-dimensional coordinate system.

[0075] It should be noted that relative pose includes translation vectors (position) and rotation matrices (attitude) in three-dimensional space, describing the transformation relationship from the coordinate system of the image acquisition device to the coordinate system of the three-dimensional marker point.

[0076] Understandably, determining the relative pose between the 3D marker and the image acquisition device requires converting pixel coordinates to 3D coordinates. Therefore, camera calibration is necessary to determine its intrinsic and extrinsic parameters. Intrinsic parameters include focal length, optical center position, radial and tangential distortion coefficients, while extrinsic parameters refer to the camera's position and orientation in the world coordinate system.

[0077] In one alternative implementation, camera calibration can use a calibration board of known size and shape, such as a black and white checkerboard. Images of the calibration board are then taken from different angles. Image processing algorithms are used to detect the corner points of the checkerboard in the images. By comparing the positions of these corner points in the images with their positions in the real world, the intrinsic and extrinsic parameters of the camera are calculated.

[0078] In one alternative implementation, a self-calibration method can be used to capture images containing straight lines or vanishing points, such as corners of walls or square objects; then, using these straight lines or vanishing points, such as the side length of a square object, the camera's intrinsic parameters can be calculated.

[0079] In one alternative implementation, after the camera is calibrated, points on the two-dimensional image plane are mapped onto rays in three-dimensional space through inverse projection transformation. The inverse projection transformation requires using camera intrinsic parameters to correct image distortion, then converting pixel coordinates into normalized coordinates, and then using camera extrinsic parameters to convert the normalized coordinates into three-dimensional coordinates in the camera coordinate system.

[0080] In one alternative implementation, after obtaining the camera's intrinsic and extrinsic parameters, a specific light pattern is projected onto the scene using structured light, and the camera captures the deformation of the light pattern to calculate the depth; alternatively, a ToF sensor is used to measure the time it takes for a light wave to travel from emission to reflection and directly calculate the depth, obtaining the depth information of each pixel in the localization image. Combining the pixel coordinates of the 2D marker points and the corresponding depth information, the spatial coordinates of the 3D marker points are calculated using the principle of triangulation. Based on the spatial coordinates, a point cloud representing the scene is constructed, where each marker point is a data point in the point cloud, and its position corresponds to its position in the real world. Then, the Iterative ClosestPoint (ICP) algorithm is used to register the 3D point cloud in the camera coordinate system with the known 3D marker point cloud, minimizing the distance between corresponding points in the two point clouds, thereby finding the optimal coordinate transformation and obtaining the relative pose of the camera coordinate system relative to the 3D marker.

[0081] Step S30: Determine the target pose of the entity corresponding to the 3D Marker point in the world coordinate system based on the relative pose and the calibration data of the image acquisition device.

[0082] It should be noted that calibration data is a set of known data points used in the camera calibration process, and these data points have specific coordinates in the real world.

[0083] It should be noted that target pose refers to the position and orientation of an object or feature point in three-dimensional space that you want to determine. Target pose typically consists of a rotation matrix and a translation vector.

[0084] It needs to be explained that calculating the target pose is to map the two-dimensional image information captured by the image acquisition device to the actual coordinate system of the three-dimensional world, so as to achieve accurate positioning and attitude determination of the real object.

[0085] It should be explained that transforming the local coordinates of a 3D marker point to the global world coordinate system requires taking into account the viewpoint and position of the image acquisition device. The calibration data provides the reference for the transformation, while the relative pose determines the specific position and orientation of the 3D marker point as observed from the camera's viewpoint.

[0086] In an optional implementation, the relative pose calculated in step S20 is converted into a pose in the world coordinate system using the intrinsic and extrinsic parameters obtained from camera calibration; for each 3D marker point, the coordinates of the corresponding marker point in the world coordinate system are calculated through coordinate transformation based on its translation vector and rotation matrix relative to the camera, as well as the calibration data.

[0087] In one alternative implementation, a transformation matrix is ​​constructed using the intrinsic and extrinsic parameters obtained from camera calibration, as well as the relative pose; the coordinates of the point in the world coordinate system are obtained by multiplying the local coordinates of the 3D marker point with the transformation matrix.

[0088] This embodiment provides a method for determining the pose of a physical object. By capturing a positioning image and extracting the pixel coordinates of a two-dimensional marker point, the relative pose between the three-dimensional marker point and the image acquisition device is determined using camera calibration and pose estimation algorithms. Then, the target pose of the three-dimensional marker point in the world coordinate system is calculated by combining the calibration data, thereby achieving accurate positioning and pose determination of each element in the virtual reality scene.

[0089] This application provides a method for determining the pose of a physical object. Please refer to... Figure 3 , Figure 3This is a first flowchart illustrating a second embodiment of the method for determining the physical pose of an object according to this application. Based on the first embodiment, step S20 further includes:

[0090] Step S201: Determine the order of selectable spots corresponding to the two-dimensional marker points in the positioning image based on the pixel coordinates.

[0091] Step S202: Set the optional spot order with the smallest projection error as the target spot order.

[0092] Step S203: Construct two-dimensional-three-dimensional point pairs according to the order of the target spots, and determine the relative pose based on the two-dimensional-three-dimensional point pairs.

[0093] It should be noted that after determining multiple possible speckle sequences, the system will evaluate the projection error of each speckle sequence. The projection error refers to the error generated when projecting points in three-dimensional space onto a two-dimensional image plane, reflecting the degree of matching between the two-dimensional image and the three-dimensional model.

[0094] It is understandable that the blob sequence with the smallest projection error indicates the highest degree of matching between the corresponding 2D image and 3D model, and is therefore most likely to reflect the actual pose of the object, so it is set as the target blob sequence.

[0095] Understandably, after determining the order of the target spots, the system will establish a correspondence between the two-dimensional image and the three-dimensional model, that is, construct two-dimensional-three-dimensional point pairs.

[0096] Optionally, before step S202, the following steps are also included:

[0097] Determine the rotation vector and translation vector corresponding to each of the optional spot sequences;

[0098] The projection error corresponding to the optional spot order is determined based on the rotation vector and the translation vector.

[0099] It should be noted that each optional blob order corresponds to a unique set of rotation and translation vectors. This set of rotation and translation vectors depends not only on the actual position of the corresponding marker point but also on its order within the image. Therefore, different blob orders may result in different combinations of rotation and translation.

[0100] Optionally, step S203 further includes:

[0101] Construct the two-dimensional-three-dimensional point pairs according to the order of the target spots;

[0102] Determine the control points and construct a local coordinate system based on them;

[0103] A system of equations is constructed based on the two-dimensional-three-dimensional point pairs and the local coordinate system, and the relative pose is solved based on the system of equations.

[0104] It should be noted that control points are specific points in a real-world scene whose locations are known and fixed. They can be man-made markers or natural feature points.

[0105] It is understandable that the positions of control points in the real scene are fixed and known, meaning that the corresponding control points can be accurately located in the image, and their coordinates in the world coordinate system can be determined. Through this correspondence, the position and orientation of the camera in three-dimensional space can be determined, thereby enabling camera calibration and pose estimation.

[0106] Optionally, the image acquisition device is a binocular camera; please refer to [reference needed]. Figure 4 , Figure 4 This is a second flowchart illustrating a second embodiment of the method for determining the physical pose of an object according to this application. Based on the first embodiment, step S20 further includes:

[0107] Step S211: Obtain the binocular baseline and binocular extrinsic parameters corresponding to the binocular camera, and determine the initial relative pose based on the positioning image acquired by the sub-camera of the binocular camera;

[0108] Step S212: Based on the binocular baseline and the binocular extrinsic parameters, establish the relationship equations for the projection positions of the three-dimensional marker points in the binocular camera image.

[0109] Step S213: Determine the relative pose based on the relational equation and the initial relative pose.

[0110] It should be noted that the binocular baseline refers to the distance between the two cameras, which can also be understood as the difference in position between the two cameras.

[0111] It should be noted that binocular extrinsic parameters refer to the offset and rotation of the camera relative to external coordinates (also known as world coordinates); in a binocular vision system, each camera has its own extrinsic parameters, which represent its relationship with the world coordinate system.

[0112] Understandably, based on the definition of binocular baseline (the distance between the two cameras is fixed), a relational equation can be established regarding the projected position of the marker point in the images of the two cameras. Solving this relational equation can determine the position of the marker point in three-dimensional space.

[0113] Understandably, by using binocular extrinsic parameters for constraints, the position of a determined marker point in three-dimensional space can be transformed into the world coordinate system.

[0114] Optionally, step S30 further includes:

[0115] The device pose of the virtual reality device is determined based on the inertial data collected by the inertial measurement unit.

[0116] The image acquisition device pose is determined based on the calibration data and the device pose.

[0117] The target pose is determined based on the pose of the acquisition device and the relative pose.

[0118] It should be noted that an inertial measurement unit (IMU) is a sensor that can be used to measure and report information such as the acceleration, angular velocity, and magnetic field of a device, providing detailed information about the device's movement and rotation in three-dimensional space.

[0119] It should be noted that the pose of the image acquisition device refers to the position and orientation of the camera in space. Since the fixed position of the image acquisition device relative to the device on a virtual reality device is known, the pose of the image acquisition device can be determined by combining calibration data and the device pose.

[0120] Optionally, a method for determining the pose of a physical object further includes:

[0121] Obtain the display parameters of the virtual object to be output, and render the virtual object according to the display parameters;

[0122] The display pose of the virtual object is determined based on the target pose.

[0123] It should be noted that in Augmented Reality (AR) technology, in order to achieve accurate fusion of virtual objects with the real world, it is necessary to determine the pose of the real object. In this process, firstly, based on the target pose of the entity corresponding to the 3D marker point in the world coordinate system, the actual scene where the target object is located is subjected to perspective rendering, and the virtual object is rendered to ensure that the target pose of the virtual object in the world coordinate system is consistent with the target pose of the entity corresponding to the 3D marker point in the world coordinate system. Then, the rendered virtual object and the 3D marker point are fused to form the real view. The fusion process is specifically implemented by superimposing the image of the virtual object onto the corresponding part of the real scene.

[0124] To better understand the solution presented in this example, please refer to... Figure 5 This example will be further explained in conjunction with specific application scenarios.

[0125] In this example, the virtual-real fusion scene is based on a driving training cabin. The "virtual" in the VR headset is a rendered environment similar to high-speed driving, with the scenery outside the car window moving quickly backward; the "real" in the actual environment is the driver's cabin in front of you, with display screens and various dashboards.

[0126] The perspective function in VR headsets allows users to see the external environment. All external environments seen through the VR headset except for the dashboard are replaced with "virtual" scenes. Therefore, it is necessary to align the virtual scenery seen from the headset with the real external scene seen through perspective, except for the dashboard.

[0127] The headset is powered on at point O. Using O_World at the moment of VR headset power-on as the reference coordinate system, the VR headset ("virtual") knows its relationship T_HMD_O with O at every moment. By calculating the relationship T_HMD_W between the VR headset and the real cockpit at every moment, the relationship between the real cockpit ("real") and O_World can be calculated. The formula is as follows:

[0128] T_W_O = T_HMD_O * T_HMD_W

[0129] The problem then becomes solving for T_HMD_W. Therefore, we introduce marker points, fix the 3D marker points onto the actual cockpit, and use the camera on the VR headset to identify the marker points and solve for the pose to obtain T_HMD_W.

[0130] Specifically, the VR headset is powered on at point O, the camera is aimed at the 3D marker rigid body to start taking pictures, and the system provides the positioning image to the image processing algorithm according to the LoadImage interface;

[0131] After the image processing algorithm performs the calculation, the system obtains the relative pose T_Marker_Camera of the marker and camera using the GetMarkerPose interface.

[0132] Parsing the HMD calibration file yields the pose transformation relationship T_IMU_Camera between the camera and IMU. Based on T_Marker_Camera, matrix operations are performed to obtain the pose T_Marker_IMU of the marker relative to the IMU. This is equivalent to... Figure 5 The formula for T_HMD_W in this context is as follows:

[0133] T_Marker_IMU=T_IMU_Camera*T_Marker_Camera

[0134] Then, the pose T_HMD_O of the system spatial positioning is obtained, and matrix operation is performed with T_HMD_W. According to the formula: T_W_O=T_HMD_O*T_HMD_W, the target pose T_W_O of the entity corresponding to the 3D marker point in the world coordinate system is obtained.

[0135] In one alternative implementation, please refer to Figure 6 The algorithm flow of this embodiment may include the following steps:

[0136] First, the system is initialized, including preprocessing the acquired images and loading the camera calibration file and algorithm configuration file.

[0137] Then, the algorithm matching step is performed, including using the OpenCV library for blob detection and calling the Blob::detect function to identify marker points in the image;

[0138] Based on the known order of the 3D Marker points, traverse all possible order of the dots, and use the projection error as a constraint to determine the correct 2D point order by minimizing the projection error.

[0139] By establishing the correspondence between two-dimensional and three-dimensional points, the system will perform spatial pose calculation to obtain the pose of the marker relative to the camera.

[0140] Finally, by utilizing the dual constraints of the binocular baseline and binocular extrinsic parameters, a unique and correct pose value is obtained.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method for determining the physical pose of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0142] This application also provides a virtual reality device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method for determining the physical pose in Embodiment 1 above.

[0143] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the method for determining the physical pose in the above embodiments.

[0144] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0145] The aforementioned computer-readable storage medium may be included in a virtual reality device; or it may exist independently and not be assembled into a virtual reality device.

[0146] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0149] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described method for determining the pose of a real object, thereby solving the technical problem of fusion and alignment of the poses of virtual objects and real objects. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for determining the pose of a real object provided in the above embodiments, and will not be repeated here.

[0150] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for determining the pose of a physical object, characterized in that, Applied to a virtual reality device, wherein the virtual reality device is equipped with an image acquisition device, the method includes: The positioning image acquired by the image acquisition device is obtained, and the pixel coordinates of the two-dimensional marker points in the positioning image are determined. Based on the pixel coordinates, the relative pose between the 3D Marker point and the image acquisition device is determined; Based on the relative pose and the calibration data of the image acquisition device, determine the target pose of the entity corresponding to the 3D Marker point in the world coordinate system. Marker points are arranged and placed on the real object. The target pose is used to map the two-dimensional image information acquired by the image acquisition device to the three-dimensional world coordinate system, so as to realize the positioning and attitude determination of the real object. The steps for determining the relative pose between the 3D marker point and the image acquisition device include: The order of selectable spots corresponding to the two-dimensional marker points in the positioning image is determined based on the pixel coordinates. Set the optional spot order with the smallest projection error as the target spot order; Two-dimensional-three-dimensional point pairs are constructed according to the order of the target spots, and the relative pose is determined based on the two-dimensional-three-dimensional point pairs; The step of constructing two-dimensional-three-dimensional point pairs according to the order of the target spots, and determining the relative pose based on the two-dimensional-three-dimensional point pairs includes: Construct the two-dimensional-three-dimensional point pairs according to the order of the target spots; Determine the control points and construct a local coordinate system based on them; A system of equations is constructed based on the two-dimensional-three-dimensional point pairs and the local coordinate system, and the relative pose is solved based on the system of equations. The calibration data includes internal and external parameters.

2. The method as described in claim 1, characterized in that, Before the step of setting the optional spot order with the smallest projection error as the target spot order, the method further includes: Determine the rotation vector and translation vector corresponding to each of the optional spot sequences; The projection error corresponding to the optional spot order is determined based on the rotation vector and the translation vector.

3. The method as described in claim 1, characterized in that, The image acquisition device is a binocular camera, and the step of determining the relative pose between the 3D marker point and the image acquisition device based on the pixel coordinates includes: Obtain the binocular baseline and binocular extrinsic parameters corresponding to the binocular camera, and determine the initial relative pose based on the positioning image acquired by the sub-camera of the binocular camera; Based on the binocular baseline and the binocular extrinsic parameters, establish the relationship equations for the projection positions of the three-dimensional marker points in the binocular camera image; The relative pose is determined based on the relationship equation and the initial relative pose.

4. The method as described in claim 1, characterized in that, The step of determining the target pose of the entity corresponding to the 3D Marker point in the world coordinate system based on the relative pose and the calibration data of the image acquisition device further includes: The device pose of the virtual reality device is determined based on the inertial data collected by the inertial measurement unit. The image acquisition device pose is determined based on the calibration data and the device pose. The target pose is determined based on the pose of the acquisition device and the relative pose.

5. The method as described in claim 1, characterized in that, The method further includes: Obtain the display parameters of the virtual object to be output, and render the virtual object according to the display parameters; The display pose of the virtual object is determined based on the target pose.

6. The method as described in claim 1, characterized in that, The step of acquiring the positioning image captured by the image acquisition device and determining the pixel coordinates of the two-dimensional marker points in the positioning image includes: The two-dimensional marker points of the positioning image are identified according to the image recognition algorithm, and the edge contours of the two-dimensional marker points are determined. Calculate the center point of the edge contour and determine the pixel coordinates of the corresponding two-dimensional marker point.

7. A virtual reality device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for determining the physical pose as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the method for determining the pose of a physical object as described in any one of claims 1 to 6.

Images